DeepMind又改Transformer了：深层激活回流，小模块反超全量微调
Ranking
Overall
82
Content
95
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - DeepMind’s Recirculation method feeds deep-layer Transformer activations back into shallower layers so later tokens can reuse more mature contextual states. A small adaptive controller improved Gemma3 language modeling beyond full fine-tuning while keeping the base model frozen.
- Adaptive Recirculation reduced average perplexity by 23.0% across nine language-modeling datasets, versus 21.6% for full fine-tuning and 8.5% for fixed recirculation.
- The method mixes norm-aligned deep- and shallow-layer states; a token-conditioned MLP dynamically predicts per-dimension mixing coefficients.
- Improvements appeared across several Transformer families, but were strongest on Gemma3 and less consistent on downstream tasks.
- Generation latency was reportedly negligible because computations can run in parallel, while sequential state updates make prefill slower for long contexts.
Sources (1)
DeepMind又改Transformer了：深层激活回流，小模块反超全量微调
Public signals
N/A
TL;DR - DeepMind’s Recirculation method feeds deep-layer Transformer activations back into shallower layers so later tokens can reuse more mature contextual states. A small adaptive controller improved Gemma3 language modeling beyond full fine-tuning while keeping the base model frozen.
- Adaptive Recirculation reduced average perplexity by 23.0% across nine language-modeling datasets, versus 21.6% for full fine-tuning and 8.5% for fixed recirculation.
- The method mixes norm-aligned deep- and shallow-layer states; a token-conditioned MLP dynamically predicts per-dimension mixing coefficients.
- Improvements appeared across several Transformer families, but were strongest on Gemma3 and less consistent on downstream tasks.
- Generation latency was reportedly negligible because computations can run in parallel, while sequential state updates make prefill slower for long contexts.