TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

TL;DR - TraceML is a process-level benchmark comparing human and AI-agent trajectories on machine-learning competitions. It shows that agents underperform partly because they repeat narrow optimization loops rather than pivoting, revisiting abandoned ideas, and coordinating diverse development activities like experts.

• The dataset contains 4,465 human Kaggle trajectories across 134 competitions, including 430 human and 207 agent trajectories on seven shared competitions.
• Each code version is annotated with score, timestamp, action, intent, edit size, and score effect, enabling analysis beyond final benchmark outcomes.
• Codex focused heavily on ensemble reweighting and submission tuning, while MLEvolve repeatedly mutated its current model; neither pivoted or reopened prior approaches as often as humans.
• A planning prompt distilled from human behavior improved targeted behaviors and scores, but did not eliminate the agents’ broader planning-pattern gap.