离开OpenAI和Google后，两位大模型核心负责人决定卷下一代架构
Ranking
Overall
68
Content
75
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - Jerry Tworek (former OpenAI RL lead) and Rohan Anil (former Google Gemini pretraining lead) have founded Core Automation to build a next-generation architecture beyond Transformers, arguing that large-scale RL alone won't reach AGI because today's models cannot keep learning after deployment.
- Core thesis: learning happens almost entirely pre-deployment. In-context learning is flexible but capacity-limited and lost on context compaction; continual fine-tuning risks catastrophic forgetting and needs too much data/compute. They want a third path — meta-learning that bakes "how to learn" into the architecture itself.
- RL is only one form of learning from experience: it relies on massive parallel rollouts with clear actions/rewards, unlike human learning of, say, mathematics via reading, comparison, and internal reasoning. They expect a broader learning algorithm rather than more RL scale.
- Rohan's second critique is compute depth: Transformers have only tens to ~100 layers of fixed forward-pass computation, so chain-of-thought and test-time scaling act as an architectural patch that buys depth one token at a time. He wants deeper computation in fewer steps, with pretraining and RL co-designed alongside kernels and hardware.
- They don't claim Transformers are done — scaling, MoE, sparsity, and speculative decoding still have room — but say scale alone yields a tool dependent on an external human retraining loop. Their AGI bar: a system that improves itself without human involvement. They're building an agent-heavy automated lab to raise architecture-experiment throughput toward tens or hundreds of runs per day.
Sources (1)
离开OpenAI和Google后，两位大模型核心负责人决定卷下一代架构
Public signals
N/A
TL;DR - Jerry Tworek (former OpenAI RL lead) and Rohan Anil (former Google Gemini pretraining lead) have founded Core Automation to build a next-generation architecture beyond Transformers, arguing that large-scale RL alone won't reach AGI because today's models cannot keep learning after deployment.
- Core thesis: learning happens almost entirely pre-deployment. In-context learning is flexible but capacity-limited and lost on context compaction; continual fine-tuning risks catastrophic forgetting and needs too much data/compute. They want a third path — meta-learning that bakes "how to learn" into the architecture itself.
- RL is only one form of learning from experience: it relies on massive parallel rollouts with clear actions/rewards, unlike human learning of, say, mathematics via reading, comparison, and internal reasoning. They expect a broader learning algorithm rather than more RL scale.
- Rohan's second critique is compute depth: Transformers have only tens to ~100 layers of fixed forward-pass computation, so chain-of-thought and test-time scaling act as an architectural patch that buys depth one token at a time. He wants deeper computation in fewer steps, with pretraining and RL co-designed alongside kernels and hardware.
- They don't claim Transformers are done — scaling, MoE, sparsity, and speculative decoding still have room — but say scale alone yields a tool dependent on an external human retraining loop. Their AGI bar: a system that improves itself without human involvement. They're building an agent-heavy automated lab to raise architecture-experiment throughput toward tens or hundreds of runs per day.