代码榜逐渐饱和，下一个前沿是AI科研，但大模型卡在了最关键一步

TL;DR — MLS-Bench is a new benchmark from a multi-university team (Berkeley, Princeton, Tsinghua, CMU, and others) that tests whether LLM agents can discover genuinely new, generalizable ML methods rather than just tune existing ones; five frontier models all failed to beat the strongest human methods. It matters because it reframes "AI for research" evaluation away from saturated coding leaderboards toward real methodological discovery.

• Design: 140 real research tasks across 12 ML areas (LM pre/post-training, vision generation, RL, robotics, ML systems, AI4Science, optimization, causal inference, time series, active learning, trustworthy ML). Each task ships a real codebase, a specific component to improve, ≥3 test conditions for transfer, and ≥3 reproduced strong human methods (including the domain SOTA) run in the identical training/scoring pipeline. Scores are anchored per-metric (weakest reproduced baseline = 0, strongest = 50, theoretical bound = 100) and aggregated across metric → condition → task.
• Headline result: Even when given full implementations of the best human methods and multiple rounds of agentic experimentation, all 5 evaluated frontier models failed to surpass the strongest human method overall. Multi-round iteration improved individual submissions but did not change the conclusion.
• Failure mode: Prompting models to "optimize/debug" outperformed prompting them to "discover a new method." Expert code review found models mostly recombine losses, modules, and tricks from the provided baselines. Removing the parameter-count guard let several models beat human methods purely by scaling capacity — showing unisolated variables can masquerade as automated discovery. Evolutionary search and test-time training overfit visible conditions and hurt held-out ones.
• Deeper bottleneck: In a flexible-compute pretraining experiment (two of three 345M runs converted into a free budget), models generally did worse with more freedom — they failed to allocate compute to high-information experiments or revise plans from intermediate results. Web search and supplied papers/derivations helped only marginally. Cost note: full 140 tasks ≈ 700 H100-hours per candidate; a 30-task subset (~100 H100-hours) covers all 12 domains and has been adopted in official frontier-model release evals.