代码榜逐渐饱和，下一个前沿是AI科研，但大模型卡在了最关键一步

TL;DR - A multi-university team (Berkeley, Princeton, Tsinghua, CMU, et al.) introduced MLS-Bench, a benchmark testing whether frontier models can discover genuinely new, generalizable ML methods rather than just tune existing ones; none of the 5 evaluated models beat the strongest human methods.

• Design: 140 real research tasks across 12 ML domains (LM pre/post-training, vision generation, RL, robotics, systems, AI4Science, causal inference, time series, etc.). Each task pins a specific method component, requires ≥3 test conditions to check transfer, and re-implements ≥3 strong human methods (including SOTA) in the same code/training/scoring pipeline. Scoring anchors weakest reproduced baseline to 0 and strongest to 50.
• Attribution control: Editable vs. protected code is annotated line-by-line, and parameter counts are checked — with that check removed, models repeatedly won by simply scaling capacity, showing apparent "auto-discovery" is often just added compute.
• Findings: Models improve when prompted to optimize/debug but get worse when asked to invent new methods; expert review found submissions mostly recombine losses/modules/tricks from provided baselines. More sampling, evolutionary search, and test-time training saturate quickly and overfit to visible conditions.
• Deeper bottleneck: Given a flexible compute budget instead of three fixed 345M pretraining runs, models mostly performed worse — they failed to allocate compute to high-information experiments or revise plans from intermediate evidence. Web search and supplied papers/derivations helped little. Full run costs ~700 H100-hours; a 30-task subset (~100 H100-hours) is already used in some official frontier-model releases.