How do you choose between data parallelism, tensor para...

Data parallelism (specifically ZeRO-3) shards optimizer states, gradients, and parameters across replicas and is the default choice until model size exceeds per-GPU memory. Tensor parallelism partitions weight matrices across devices, requiring two all-reduce calls per layer, and should be restricted to high-bandwidth NVLink domains. Pipeline parallelism partitions layers across nodes and requires careful micro-batching to minimize bubble overhead, where the ideal bubble fraction is approximately 1/m, with m being the number of micro-batches per pipeline flush.