179 questions
No questions match those filters.
← All questions
System DesignMedium· Distributed Training· Scaling Laws· Data Parallelism
Why can’t you just scale data parallelism to 10,000 GPUs?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansScaling data parallelism (DP) indefinitely is limited by memory and efficiency constraints. Without ZeRO-3, DP does not reduce the memory footprint of parameters per GPU. Additionally, there is a critical batch size beyond which adding more replicas does not improve sample efficiency, meaning you run out of a useful batch-size budget before you run out of hardware. At very high DP degrees, the synchronization overhead of ZeRO-3 and topology-dependent latency across spine switches create performance degradation.