179 questions
No questions match those filters.
← All questions
Training & Fine-tuningHard· Loss Spikes· Numerical Stability· Pre-training
Your 70B pre-training run is showing periodic loss spik...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo investigate, first distinguish a data quality spike (a bad batch artifact) from structural drift. Data quality issues usually recover in 1-2k steps, whereas structural drift recovers slowly and tends to recur. Check gradient norms for correlated spikes. If log Z(x) has been trending upward for tens of thousands of steps before each spike, you are experiencing logit scale drift. The recommended fix is to add z-loss from the start of the next run.