179 questions
No questions match those filters.
← All questions
System DesignMedium· Tensor Parallelism· Communication· Performance
What determines the AllReduce size in tensor parallelism?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIn tensor parallelism, the communication overhead is driven by the activation tensors passed between layers. Because the size of these activations is a function of sequence length, batch size, and hidden dimension, the communication traffic scales with these factors rather than the total number of parameters in the model depth.