What does QK-norm actually do to the attention distribu...

QK-norm bounds pre-softmax attention logits to O(1/√dk). This prevents the attention mechanism from collapsing into a one-hot distribution over long training durations, which maintains the model’s ability to attend to multiple tokens. The cost of this implementation is two additional RMSNorm calls per attention layer, which is considered negligible compared to the computational cost of the surrounding projections.