🛰️ Daily AI Frontier
‹ back to 2026-09-25

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

Research LLMs & Foundation Models

Ranking

Overall 88
Content 95
Popularity 71

Observed public metrics from 1 member.

Representative image for Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

Merged summary

TL;DR - This paper presents evidence that Transformers can linearly superpose distinct text streams, producing a combined next-token distribution resembling the average of their individual distributions. Lightweight fine-tuning and guided decoding restore and exploit this behavior to generate two coherent continuations from one forward pass.

  • The authors call this behavior the Superposition Linearity Hypothesis.
  • Experiments suggest linear superposition is intrinsic to Transformer architecture but weakens during pretraining.
  • Lightweight fine-tuning substantially reduces divergence between superposed predictions and averaged individual predictions.
  • A guided decoding method disentangles the combined output into two simultaneous, coherent generations.

Sources (1)

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

arXiv cs.CL Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Nikita Dragunov, Temurbek Rahmatullaev, Polina Druzhinina, Anton Razzhigaev, Ivan Oseledets, Elena Tutubalina 2026-09-24 arXiv:2609.29845
Public signals Hugging Face upvotes 62
Providers: Hugging Face · Upvotes 62 OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-09-26 14:14:11.074099 UTC

TL;DR - This paper presents evidence that Transformers can linearly superpose distinct text streams, producing a combined next-token distribution resembling the average of their individual distributions. Lightweight fine-tuning and guided decoding restore and exploit this behavior to generate two coherent continuations from one forward pass.

  • The authors call this behavior the Superposition Linearity Hypothesis.
  • Experiments suggest linear superposition is intrinsic to Transformer architecture but weakens during pretraining.
  • Lightweight fine-tuning substantially reduces divergence between superposed predictions and averaged individual predictions.
  • A guided decoding method disentangles the combined output into two simultaneous, coherent generations.
item →