Accelerate documentation
Overview
Getting started
Tutorials
OverviewAdd Accelerate to your codeExecution processTPU trainingLaunching distributed codeLaunching distributed training from Jupyter Notebooks
How to guides
Accelerate
Start Here!Model memory estimatorModel quantizationExperiment trackersSave and load training statesTroubleshootExample Zoo
Training
Gradient accumulationLocal SGDLow precision (FP8) trainingDeepSpeedFully Sharded Data ParallelismMegatron-LMAmazon SageMakerApple M1 GPUsIPEX training with CPU
Inference
Concepts and fundamentals
🤗 Accelerate's internal mechanismLoading big models into memoryComparing performance across distributed setupsExecuting and deferring jobsGradient synchronizationHow training in low-precision environments is possible (FP8)TPU best practices
Reference
AcceleratorStateful configuration classesThe Command LineTorch wrapper classesExperiment trackersDistributed launchersDeepSpeed utilitiesLoggingWorking with large modelsDistributed inference with big modelsKwargs handlersUtility functions and classesMegatron-LM UtilitiesFully Sharded Data Parallelism Utilities
You are viewing v0.28.0 version. A newer version v1.13.0 is available.
Overview
Welcome to the 🤗 Accelerate tutorials! These introductory guides will help catch you up to speed on working with 🤗 Accelerate. You’ll learn how to modify your code to have it work with the API seamlessly, how to launch your script properly, and more!
These tutorials assume some basic knowledge of Python and familiarity with the PyTorch framework.
If you have any questions about 🤗 Accelerate, feel free to join and ask the community on our forum.