At 1024 GPUs, would you use FSDP alone or combine it wi...

Scaling to 1024 GPUs requires hierarchical parallelism. FSDP across the entire cluster would result in brutal latency due to inter-node InfiniBand speeds during AllGather operations. By applying tensor parallelism within each 8-GPU node, you leverage high-speed NVLink for the most frequent communication, while using FSDP or pipeline parallelism across nodes for the remaining model state, effectively matching the communication pattern to the available bandwidth.