FineWeb outperforms C4 on benchmarks. Is that because o...

Controlled ablations show that while FineWeb uses significantly more tokens than C4, the inclusion of MinHash deduplication is the dominant factor in the performance gap. Deduplication alone accounts for most of the improvement on knowledge-intensive benchmarks. The additional token volume provided by FineWeb is most beneficial for improving few-shot generalization tasks. Practically, this suggests that deduplication offers higher leverage than raw volume once a corpus exceeds a few hundred billion tokens.