Walk me through how gradients are synchronized in a sta...

In a standard data-parallel training run, each GPU computes gradients on its local mini-batch during the backward pass. Following this, the GPUs call AllReduce (typically using a ring algorithm and sum operation) to produce the globally averaged gradient. The result is divided by the world size, or a ReduceOp.AVG is used, ensuring each GPU takes an identical optimizer step. The communication cost for this process is 2P bytes per rank.