179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· Mixed Precision· Distributed Training· Memory Optimization
Walk me through your mixed-precision training setup for...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe setup uses BF16 for all forward and backward matmuls, while maintaining FP32 master weights, gradients, and Adam optimizer states. FlashAttention is used for numerical stability, and gradient clipping is set at 1.0. FP16 is avoided to bypass the need for dynamic loss scaling. Given the 16-18 bytes per parameter static footprint, ZeRO stage-2 or -3 is required for a 70B model.