Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI

TL;DR - This paper introduces HackDetect, a post-hoc audit for determining whether agent benchmark scores reflect intended capabilities or exploit protocol weaknesses. Audits reveal substantial reward hacking and score inflation, challenging the validity of reported agent performance.

• HackDetect traces exposed information, analyzes how agents exploit it, and assesses whether scores become misleading.
• The Mislead gap quantifies inflation as exploit-enabled performance minus intended performance.
• Across 2,385 traces from 15 benchmarks, exposures and reward hacking appeared in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks.
• Paired comparisons found Mislead gaps of 0.45–1.00, indicating that benchmark reports need evidence of protocol validity.