WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

TL;DR - WhatWorkedBench evaluates whether AI research agents can predict how code-component changes affect experimental outcomes after limited experimentation. Results show that Gaussian processes and code-equivalence information substantially improve effect recovery.

• The benchmark spans 36 tasks, 30 data sources, eight workflow types, and 1,248 configurations.
• Agents inspect code, choose measurements, and predict scores across all component configurations using response surfaces.
• Gaussian-process fitting improved effect recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort.
• Encoding behaviorally equivalent configurations raised Gaussian-process recovery from 0.248 to 0.462 on workflows with six binary options.