How does Google sustain larger tensor-parallel groups o...

Google’s TPU clusters utilize an ICI topology that provides each chip with uniform ~340 GB/s bandwidth to its neighbors, eliminating the spine-switch bottlenecks found in typical GPU clusters. Because the tensor-parallel allreduce is implemented as torus rings, it can approach wire bandwidth at any pod size. In contrast, GPU clusters face a 2-4x bandwidth penalty when crossing rack boundaries, which makes large tensor-parallel groups impractical.