Your training loss suddenly spikes to NaN partway throu...

Softmax and attention-score overflow in low precision is the single most common culprit. Attention scores are large-magnitude dot products before the softmax normalizes them, and in fp16 those pre-softmax logits can overflow before normalization ever happens — the result is NaN propagating through everything downstream in that layer.

My first move is to check the raw attention-score magnitudes right before the spike (not after — by the time loss shows NaN, the actual overflow happened one or more steps earlier). If that’s the pattern, the fix isn’t “lower the learning rate and hope” — it’s computing the softmax itself in higher precision (fp32) even when the rest of the layer runs in fp16/bf16, and/or adding a stabilizing max-subtraction step before exponentiating. Blindly dropping the learning rate can mask the symptom for a while without fixing the actual numerical cause, and the spike usually resurfaces later at a different training step.