Why does training with FP16 sometimes produce NaN value...

FP16 has a 5-bit exponent capping representable values at 65504. Attention logits, activation values, or gradient products can exceed this threshold, overflowing to infinity, which converts to NaN in subsequent operations. BF16’s 8-bit exponent matches FP32’s dynamic range, eliminating this overflow path. The loss in mantissa precision (7 bits vs. 23) is negligible for per-layer computation.