Walk me through how Common Crawl becomes training tokens.

Common Crawl provides data in WARC (raw HTTP) and WET (pre-extracted text) formats. A robust pipeline extracts high-quality text from the raw WARC files using specialized tools like Trafilatura, which generally outperforms the WET files. After extraction, the data undergoes language identification, heuristic filtering to remove low-quality content, deduplication, and potentially quality classification to ensure the final training tokens meet the required standards.