TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
Ranking
Overall
79
Content
95
Popularity
42
Observed public metrics from 1 member.
Merged summary
TL;DR - TraceML is a process-level benchmark comparing human and AI-agent trajectories on machine-learning competitions. It shows that agents underperform partly because they repeat narrow optimization loops rather than pivoting, revisiting abandoned ideas, and coordinating diverse development activities like experts.
- The dataset contains 4,465 human Kaggle trajectories across 134 competitions, including 430 human and 207 agent trajectories on seven shared competitions.
- Each code version is annotated with score, timestamp, action, intent, edit size, and score effect, enabling analysis beyond final benchmark outcomes.
- Codex focused heavily on ensemble reweighting and submission tuning, while MLEvolve repeatedly mutated its current model; neither pivoted or reopened prior approaches as often as humans.
- A planning prompt distilled from human behavior improved targeted behaviors and scores, but did not eliminate the agents’ broader planning-pattern gap.
Sources (1)
TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - TraceML is a process-level benchmark comparing human and AI-agent trajectories on machine-learning competitions. It shows that agents underperform partly because they repeat narrow optimization loops rather than pivoting, revisiting abandoned ideas, and coordinating diverse development activities like experts.
- The dataset contains 4,465 human Kaggle trajectories across 134 competitions, including 430 human and 207 agent trajectories on seven shared competitions.
- Each code version is annotated with score, timestamp, action, intent, edit size, and score effect, enabling analysis beyond final benchmark outcomes.
- Codex focused heavily on ensemble reweighting and submission tuning, while MLEvolve repeatedly mutated its current model; neither pivoted or reopened prior approaches as often as humans.
- A planning prompt distilled from human behavior improved targeted behaviors and scores, but did not eliminate the agents’ broader planning-pattern gap.