The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

DATASET PAPER - relevant for understanding training data quality claims. The paper documents: (1) FineWeb is a 15-trillion token dataset from 96 Common Crawl snapshots. (2) 'LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC.' (3) The paper 'carefully document[s] and ablate[s] all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies.' This paper is from Hugging Face (a neutral party) and provides transparency into dataset curation. It is relevant for skepticism because it shows that dataset quality claims require careful documentation and ablation - many vendor dataset claims lack this rigor.