179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Parallelism Strategies· Distributed Training· Networking
When would you choose pipeline parallelism over FSDP?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFSDP’s AllGather-per-layer pattern costs 3|θ| per step, which can stall on low-bandwidth links like 25 GB/s InfiniBand. Pipeline parallelism’s activation handoffs are one to two orders of magnitude smaller than FSDP’s weight AllGathers, making it more viable for large models in bandwidth-constrained environments.