AI让科研变成“科学自动机”，副作用是什么？

TL;DR - HKU professor Zhang Zheng argues that the emerging "science automaton" — closed-loop AI hypothesis generation plus automated verification — will boost scientific throughput but carries two structural side effects: erosion of human scientific taste and a hard ceiling on genuine novelty.

• Closing the loop: generation is already industrialized (Shanghai Analemma's FARS produced 166 papers in 17 days, ~2h and ~$1k each, scoring 5.05 under Stanford's Agentic Reviewer vs. ICLR 2026 human submission mean 4.21); the hard part is verification — OpenAI's Lean 4 certificates for 10 open math problems, Google's Science One Framework evidence chains, and Shanghai AI Lab's Intern·Duanyan dry/wet closed loop with independent review agents all attack the same problem.
• Capability trap: taste and judgment only grow from years of hands-on struggle, especially writing ("writing is half of research" — cleaner designs surface only while drafting). Since institutions reward output, not capability formation, this step is being outsourced first and likely irreversibly; today's senior judgment is a non-renewable stock.
• Ceiling trap: LLMs are super pattern-completers bounded by generalization from existing corpora (author cites his own Comprehension Without Competence, TMLR 2025). Automata excel at combinatorial search in known spaces — materials screening, drug repurposing, weather (1000x faster than numerical methods), chip design (5–10x) — but "fill the floor" rather than raise the ceiling.
• Breakthroughs like negative numbers, complex numbers, non-Euclidean geometry, or AlphaFold's reframing of folding from physics computation to a prediction problem lie outside any existing search space; such outlier "spiky gradients" are exactly what training smooths away.