You have a 13B parameter model and 8 × A100 80 GB GPUs....

To calculate the memory requirements for a 13B parameter model, we multiply 13 billion parameters by 16 bytes per parameter, resulting in 208 GB. Since each A100 GPU only provides 80 GB of HBM, the model state alone is 2.6 times over the budget, even before accounting for activation memory. Consequently, naive DDP is not feasible, and one would need to pivot to techniques like ZeRO Stage 3 or FSDP to shard the model state.