Accelerate

Stateful Classes

Below are variations of a singleton class in the sense that all instances share the same state, which is initialized on the first instantiation.

These classes are immutable and store information about certain configurations or states.

class accelerate.state.AcceleratorState

< source >

( mixed_precision: str = None cpu: bool = False dynamo_backend = None deepspeed_plugin = None fsdp_plugin = None megatron_lm_plugin = None _from_accelerator: bool = False **kwargs )

Singleton class that has information about the current training environment.

Available attributes:

device (torch.device) — The device to use.
distributed_type (DistributedType) — The type of distributed environment currently in use.
initialized (bool) — Whether or not the AcceleratorState has been initialized from Accelerator.
local_process_index (int) — The index of the current process on the current server.
mixed_precision (str) — Whether or not the current script will use mixed precision, and if so the type of mixed precision being performed.
num_processes (int) — The number of processes currently launched in parallel.
process_index (int) — The index of the current process.

class accelerate.state.GradientState

< source >

( )

Singleton class that has information related to gradient synchronization for gradient accumulation

Available attributes:

end_of_dataloader (bool) — Whether we have reached the end the current dataloader
remainder (int) — The number of extra samples that were added from padding the dataloader
sync_gradients (bool) — Whether the gradients should be synced across all devices