If you had to scale to 1024 GPUs for a single model, ho...

The strategy is to align parallelism with the interconnect hierarchy. Use TP-8 within the node to leverage high-speed NVLink. Use pipeline parallelism across nodes to manage model depth, as it only requires InfiniBand for small activation handoffs. Data parallelism is then used to scale the training throughput once the model fits. The core principle is to avoid extending TP beyond the NVLink boundary to prevent performance-killing InfiniBand stalls.