When does a model actually need tensor/pipeline paralle...

Data parallelism, even with full ZeRO sharding, still requires each GPU to materialize a complete layer’s activations and weights at the moment it’s computing that layer’s forward or backward pass — the sharding saves steady-state memory, not the peak needed during computation. When a single layer is too large for one GPU regardless, that’s the point where you need tensor parallelism: splitting the matrix multiplications within a layer itself across multiple GPUs, so no single device ever holds the whole layer.

Pipeline parallelism solves a different problem: splitting different layers of the model across different GPUs, so each device only ever holds a subset of the model’s depth, at the cost of needing to pipeline micro-batches through the stages to keep every GPU busy rather than idle waiting on the previous stage.

In practice, frontier training runs combine all of these — data parallelism across replicas, tensor parallelism within a node for layers too big for one GPU, and pipeline parallelism across nodes for depth — because each technique alone hits a different wall first.