How would you structure the data mix for a 70B model’s...

For a 70B model’s annealing phase, the data mix should consist of approximately 20-30% instruction-adjacent data. This should start with high-quality domain-specific data—such as math, code, and curated web content—and incorporate a smaller fraction of chat SFT pairs. To monitor progress, track validation loss on both the pre-training held-out set and a held-out instruction set; both should decrease simultaneously. Additionally, run intermediate capability evaluations every 5-10 billion decay tokens to detect benchmark regressions early. Finally, apply a KL penalty or a reference-model constraint to bound distribution drift from the stable-phase checkpoint, particularly if the decay phase is extended.