Why is tensor parallel degree capped at 8 in most produ...

Once tensor parallelism exceeds the 8-GPU limit of a standard node, AllReduce operations must traverse InfiniBand. For a 70B model, this results in communication times that far exceed compute times (e.g., 75 ms vs 31 ms). Empirically, overhead jumps from 10-12% at degree 8 (intra-node) to 42% at degree 16 and 65% at degree 32, making it highly inefficient.