179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· ZeRO· Distributed Training· Communication Efficiency
Explain ZeRO Stage 1 and why it doesn’t cost more bandw...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansZeRO Stage 1 optimizes memory by sharding optimizer states. It does not increase bandwidth usage because it decomposes the standard AllReduce into a ReduceScatter (which delivers per-shard reduced gradients) and an AllGather (which redistributes updated parameters). Because the optimizer update occurs locally on each worker after the ReduceScatter, the total communication remains 2|θ|, identical to a ring-AllReduce.