179 questions
No questions match those filters.
← All questions
Training & Fine-tuningHard· Distributed Training· Performance Estimation
Estimate how long the gradient sync takes for a 70B par...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo estimate the sync time, first calculate the gradient tensor size: 70 billion parameters * 2 bytes (bf16) = 140 GB per rank. The AllReduce communication volume is 2 * (7/8) * 140 GB = 245 GB. Assuming an effective bandwidth of 250 GB/s, the raw transfer time is approximately 1 second. In practice, DDP’s gradient bucketing mechanism hides much of this latency by overlapping the sync with the backward pass computation.