179 questions
No questions match those filters.
← All questions
Training & Fine-tuningHard· Communication Bottlenecks· AllReduce· Distributed Training
At what point does the AllReduce become the bottleneck,...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe bottleneck is defined by the compute-to-communication ratio. When the communication overhead of AllReduce dominates the compute time, the system is communication-bound. Remedies include lossy gradient compression, which may hurt convergence; asynchronous SGD, which introduces staleness; or utilizing ZeRO Stage 2, which interleaves communication with computation layer by layer to hide latency.