Why does the standard recipe use tensor parallelism wit...

The standard recipe optimizes for the bandwidth gap between intra-node and inter-node communication. Tensor parallelism requires an allreduce at every layer, which is supported by the 900 GB/s all-to-all bandwidth of NVSwitch within a node. However, InfiniBand between nodes only provides 25-100 GB/s, creating a 10-36x bandwidth gap that makes cross-node tensor parallelism bandwidth-bound. Pipeline parallelism is preferred across nodes because it substitutes point-to-point activation sends for allreduces, significantly lowering the volume of cross-node traffic.