Walk me through how FSDP reduces optimizer-state memory...

Naive DDP replicates the entire model, gradients, and optimizer states on every GPU, which is highly memory-inefficient for large models. FSDP (Fully Sharded Data Parallel) improves this by implementing the ZeRO stages.

By sharding the optimizer states, gradients, and parameters across the available GPUs, FSDP drastically reduces the per-GPU memory footprint. Although Stage 3 increases the communication volume to three times the parameter count, the ability to overlap these communication steps with computation ensures that the training process remains efficient and scalable.