AI能接管实验室了？中国科大最新研究给出真实物理世界的压力测试
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - USTC built a robotic catalysis lab (45 modular workstations for synthesis, characterization, and performance testing) exposed to LLM agents as machine-readable "skills," then benchmarked whether agent-generated plans actually execute in the physical world. Only 3.3% of 4,608 trials produced workflows runnable without human repair, showing fluent planning ≠ executable science.
- Benchmark scope: 48 configurations (6 agent frameworks × 9 LLMs) across 32 expert-defined research tasks, scored not just on plan generation but on validation, dispatch to robots, and hands-off execution.
- Best results were Claude Code + Claude Opus 4.7 at 28.1% executable and Codex + GPT 5.5 at 19.8%; the overall unaided execution rate was 3.3% (151/4,608).
- In a 5-round closed loop (plan → robot execution → evidence → replan), Codex/GPT 5.5 tuned recipes and conditions but never restructured its workflow skeleton or fixed persistent omissions (missing electrode binder, analyte-specific colorimetric reagent) — parameter tuning is not strategic replanning.
- Long-horizon planning is the bottleneck: only 3 workflows exceeded 30 operation steps (max 44), and the authors frame the robotic lab as both a test bed and a future training ground, logging successes/failures as agent-alignment data.
Sources (1)
AI能接管实验室了？中国科大最新研究给出真实物理世界的压力测试
TL;DR - USTC built a robotic catalysis lab (45 modular workstations for synthesis, characterization, and performance testing) exposed to LLM agents as machine-readable "skills," then benchmarked whether agent-generated plans actually execute in the physical world. Only 3.3% of 4,608 trials produced workflows runnable without human repair, showing fluent planning ≠ executable science.
- Benchmark scope: 48 configurations (6 agent frameworks × 9 LLMs) across 32 expert-defined research tasks, scored not just on plan generation but on validation, dispatch to robots, and hands-off execution.
- Best results were Claude Code + Claude Opus 4.7 at 28.1% executable and Codex + GPT 5.5 at 19.8%; the overall unaided execution rate was 3.3% (151/4,608).
- In a 5-round closed loop (plan → robot execution → evidence → replan), Codex/GPT 5.5 tuned recipes and conditions but never restructured its workflow skeleton or fixed persistent omissions (missing electrode binder, analyte-specific colorimetric reagent) — parameter tuning is not strategic replanning.
- Long-horizon planning is the bottleneck: only 3 workflows exceeded 30 operation steps (max 44), and the authors frame the robotic lab as both a test bed and a future training ground, logging successes/failures as agent-alignment data.