You are training a 70B model and gradient norm spikes a...

When encountering gradient norm spikes, the first step is to implement per-layer gradient monitoring to determine if the spikes originate in the attention blocks or the output projection. To address these issues, one might introduce QK-norm for the attention path to bound logits and z-loss for the output path to penalize logit scale. Additionally, it is critical to verify if the current learning rate remains within the stable descent region of the cosine schedule.