179 questions
No questions match those filters.
← All questions
LLMOps & ProductionMedium· Pre-training· Failure Modes· Loss Spikes
You are training a 30B model and the loss spikes mid-ru...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen a loss spike occurs, you must first determine if it is a transient issue or a fundamental model failure. Start by checking the gradient norm trajectory to see if it was climbing before the spike, and inspect the canary batch loss to rule out data-specific artifacts. Identify the specific document at the data loader cursor position to see if the data is problematic. If these checks are inconclusive, wait up to 200 steps for the model to self-recover; if it fails to do so, roll back to the most recent clean checkpoint.