179 questions
No questions match those filters.
← All questions
Training & Fine-tuningHard· Distributed Training· Performance Profiling· Communication Overhead
At what point does tensor parallelism overhead exceed i...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansCommunication overhead becomes the bottleneck when tensor parallelism crosses NVLink boundaries, where cross-node bandwidth is significantly lower than intra-node bandwidth. Empirically, this is detected by monitoring Model FLOP Utilization (MFU). If MFU drops by more than approximately 5% relative to each doubling of the TP degree, the communication overhead is dominating, and it is preferable to switch to additional pipeline or data parallelism.