RT by @huggingface: datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs…

TL;DR - Hugging Face released DataTrove 0.10.0, expanding its data-processing pipelines with managed job execution and native Hugging Face storage integration. The update lowers infrastructure overhead for large-scale dataset preparation.

• JobsPipelineExecutor supports fan-out, multi-stage dependencies, retries, and resumable pipelines on Hugging Face Jobs.
• Pipelines can run without a dedicated Slurm cluster.
• HF storage buckets now work as DataFolder targets via hf://buckets/... for reading, writing, and logging.
• Inference results now preserve reasoning outputs.