How loss/metric reporting works with deepspeed and transformers.Trainer?

🤗Accelerate
1

I am training my model with deepspeed using hugging face Trainer API. I was wondering how Trainer/Accelerate handles logging of loss / compute_metrices. My observation is that deepspeed creates separate processes per gpu for training and each process has its own Trainer. With this observation I have the following hypothesis:

The acceleare will gather all the losses and avg them in one of the processes before logging them in tensor board.

So my questions are:

  1. Is this hypothesis correct, or do I need additional handling for reporting if I am using deepspeed?
  2. Is this hypothesis valid during evaluation phase as well?
  3. Is this hypothesis valid for compute_metrics function as well? If not, how do I write process-safe compute_metrics function to report global statistics?

Thanks.