Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Ranking
Overall
78
Content
95
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper proves that frozen transformers can use prompt examples to simulate iterative generative samplers without parameter updates or explicit density estimation. It connects transformer components and layer-wise representation geometry to diffusion and energy-based sampling.
- Softmax attention computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates.
- The constructions realize closed-form and smoothed closed-form diffusion samplers from in-context samples.
- In semantic-topic experiments, normalized hidden states move toward a uniform spherical reference in intermediate layers, then recover topic-dependent structure near the output.
- An interacting-particle energy follows the same U-shaped pattern across layers, which the authors reproduce theoretically with an approximate energy-based sampler.
Sources (1)
Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper proves that frozen transformers can use prompt examples to simulate iterative generative samplers without parameter updates or explicit density estimation. It connects transformer components and layer-wise representation geometry to diffusion and energy-based sampling.
- Softmax attention computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates.
- The constructions realize closed-form and smoothed closed-form diffusion samplers from in-context samples.
- In semantic-topic experiments, normalized hidden states move toward a uniform spherical reference in intermediate layers, then recover topic-dependent structure near the output.
- An interacting-particle energy follows the same U-shaped pattern across layers, which the authors reproduce theoretically with an approximate energy-based sampler.