Why do practitioners limit tensor parallelism to 8 GPUs?

Tensor parallelism (TP) is typically limited to the number of GPUs within a single node (e.g., 8 GPUs in a DGX H100) because of the drastic difference in communication speeds. Within a node, NVSwitch provides a fast all-to-all fabric that makes layer-level allreduce operations efficient. Crossing to a second node requires InfiniBand, which offers 10-20x lower bandwidth than the intra-node interconnect. This bottleneck leads to severe performance degradation, with TP=16 showing a concrete 42% throughput loss compared to intra-node configurations.