You want to add more chemistry data to a pre-training c...

To integrate specialized chemistry data, first define a target distribution (T) using high-quality sources like arXiv chemistry papers and textbook excerpts. Second, train a KenLM model on T to perform a fast initial perplexity-based filter. Third, optionally train a fastText classifier using T as positive data and raw web data as negative data. Fourth, score the full corpus and retain documents exceeding a specific threshold. Finally, apply MinHash LSH to deduplicate the resulting set before merging it into the main pre-training corpus.