Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

TL;DR - This paper proves that frozen transformers can use prompt examples to simulate iterative generative samplers without parameter updates or explicit density estimation. It connects transformer components and layer-wise representation geometry to diffusion and energy-based sampling.

• Softmax attention computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates.
• The constructions realize closed-form and smoothed closed-form diffusion samplers from in-context samples.
• In semantic-topic experiments, normalized hidden states move toward a uniform spherical reference in intermediate layers, then recover topic-dependent structure near the output.
• An interacting-particle energy follows the same U-shaped pattern across layers, which the authors reproduce theoretically with an approximate energy-based sampler.