179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· FSDP· ZeRO· Memory Optimization
What is FSDP and when would you use it instead of DDP?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFSDP shards the entire model state (parameters, gradients, and optimizer states) across all GPUs. While it is essential for fitting massive models that exceed HBM, it comes with a cost: network utilization increases to 3x compared to DDP’s 2x, and there is a risk of compute bubbles if AllGather latency is not effectively hidden by prefetching.