You’re seeing attention heads collapse to attending a single position after 20B tokens. What’s happening and what do you do?

Query and key norms have grown unchecked, producing logits large enough for the softmax to concentrate near one-hot. QK-norm prevents this by normalizing Q and K before the dot product, bounding the logit magnitude from the start. Alternatively, soft-capping clips post-dot-product logits via a tanh function. For the current run, monitor per-head logit statistics and consider reducing the learning rate to slow norm growth.