LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

TL;DR - LitTraceQA is a benchmark for literature-grounded scientific QA that requires systems to return canonical paper IDs, supporting evidence locations, and answers together, rather than just fluent text. It matters because it separately scores retrieval, evidence grounding, and answer accuracy, targeting verifiable RAG/research-assistant outputs instead of unsupported summaries.

• Task setup: given a research question plus a metadata pool of papers, a system must produce three connected outputs — paper identifiers, evidence locations, and answers in requested formats (free-form text, multiple choice, structured tables).
• Evidence types reflect real scientific reading: tables, figures, text spans, equations/algorithms, and citation contexts.
• Public dev split has 55 examples (26 hidden-source single-paper, 29 multi-paper) with gold papers, evidence annotations, and answers for local validation.
• A larger final annotation collection is analyzed: 4,978 unique-question records over 4,859 unique gold papers; no model results are reported in the provided abstract.