Why would you mix instruction data into pre-training in...

Integrating instruction data into the pre-training phase offers three primary advantages. First, it reduces catastrophic forgetting because the model’s learning rate is already in a decay phase. Second, it improves data efficiency, as the model’s pre-trained representations are already highly developed when the instruction signal is introduced. Third, this approach is more scalable, as it utilizes existing pre-training infrastructure and avoids the overhead of managing a separate training job with a distinct hyperparameter surface.