If you had to cut your corpus from 15 trillion tokens t...

When facing compute constraints, decisions should be driven by capability impact rather than raw volume. Research from studies like Dolma and DCLM demonstrates that code and academic papers provide an outsized impact on benchmark performance per token. Therefore, one should prioritize preserving these domains. Conversely, generic web text is the most substitutable component and should be the first to be reduced. This strategy ensures that the model retains its reasoning and technical capabilities despite a smaller overall token count.