WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
Ranking
Overall
86
Content
100
Popularity
54
Observed public metrics from 1 member.
Merged summary
TL;DR - WhatWorkedBench evaluates whether AI research agents can predict how code-component changes affect experimental outcomes after limited experimentation. Results show that Gaussian processes and code-equivalence information substantially improve effect recovery.
- The benchmark spans 36 tasks, 30 data sources, eight workflow types, and 1,248 configurations.
- Agents inspect code, choose measurements, and predict scores across all component configurations using response surfaces.
- Gaussian-process fitting improved effect recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort.
- Encoding behaviorally equivalent configurations raised Gaussian-process recovery from 0.248 to 0.462 on workflows with six binary options.
Sources (1)
WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
Public signals
Hugging Face upvotes 8
TL;DR - WhatWorkedBench evaluates whether AI research agents can predict how code-component changes affect experimental outcomes after limited experimentation. Results show that Gaussian processes and code-equivalence information substantially improve effect recovery.
- The benchmark spans 36 tasks, 30 data sources, eight workflow types, and 1,248 configurations.
- Agents inspect code, choose measurements, and predict scores across all component configurations using response surfaces.
- Gaussian-process fitting improved effect recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort.
- Encoding behaviorally equivalent configurations raised Gaussian-process recovery from 0.248 to 0.462 on workflows with six binary options.