Design the parallelism strategy for a 100B dense transf...

For a 100B dense transformer, the model size at bf16 is 200GB. By setting Tensor Parallelism (TP) to 8, each GPU handles 25GB, which fits within H100 memory without needing Pipeline Parallelism (PP). With 512 GPUs, this results in 64 TP groups. Setting Data Parallelism (DP) to 64 allows for a global batch size of 2048, with each replica processing 32 sequences. ZeRO-1 is recommended to shard optimizer states across DP ranks, and the strategy relies on overlapping gradient synchronization with computation to maintain efficiency.