ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

TL;DR - ExtractBench is a benchmark for schema-guided document extraction, where an agent fills a user-defined schema from an enterprise document and cites source evidence. It's presented as the first to jointly score value accuracy, large-scale record completeness, grounding, and measured cost — a combination that matters for real document-processing agent deployments.

• Corpus spans 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, tagged by challenge scenario.
• Ground truth is built via a scalable pipeline: independent-system agreement for real documents, known values for synthetic lists, and human verification for forms.
• Metrics are order-insensitive value F1 for accuracy plus word-level and page-level F1 for source traceability/grounding.
• Findings: commercial VLMs do well on short documents but truncate record lists on long ones; coding agents are more accurate but far costlier; LlamaExtract Agentic Plus ranks first on all three metrics at a fraction of coding-agent cost (note: authors are affiliated with LlamaIndex, per the released dataset/code links).