Walk me through how DDP works and what its limitations...

DDP (Distributed Data Parallel) operates by replicating the model across all participating GPUs. Each GPU processes a unique slice of the data batch. After the local forward and backward passes, an AllReduce operation is performed to synchronize gradients across all workers. This operation typically costs 2|θ| bytes of communication. A critical limitation is the memory accounting: each parameter requires 16 bytes (in standard configurations), meaning memory usage is fixed per GPU regardless of the total GPU count, preventing the scaling of model size.