Walk me through what happens when your training loop ca...

When dist.all_reduce() is invoked, the Python API triggers the torch.distributed NCCL shim. This shim enqueues a specific CUDA kernel to the GPU. The GPU then performs a ring-allreduce operation using high-speed interconnects like NVLink to synchronize the data across all participating ranks. Once the kernel is launched, the CPU is no longer involved in the data path, and the result is written back in-place on all ranks.