Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

TL;DR - An arXiv cs.CL paper proposing a reference-free, LLM-judge framework to audit the quality of benchmarks used to evaluate task-oriented conversational agents. It matters because unreliable benchmarks silently corrupt agent evaluation, and this gives a way to score the scorers.

• Scores benchmarks along three axes — consistency, complexity, and policy coverage — and emits actionable diagnostics pinpointing specific weaknesses, without needing reference/gold benchmarks.
• Validated three ways: agreement with independent human annotations, discrimination between benchmarks generated by LLMs of differing capability, and sensitivity to controlled quality-degrading perturbations.
• Metrics reportedly separate quality levels consistently across multiple domains and across different judge models, suggesting the approach isn't tied to one judge.
• Applies to both synthetic (LLM-generated) and manually curated benchmarks; the abstract reports qualitative consistency rather than specific numeric scores.