Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

TL;DR - This mechanistic study finds that masked diffusion language models implement a direction-symmetric induction circuit for in-context learning. Their advantage comes from accessing context on both sides of a masked token, not from stronger left-context induction.

• Previous- and next-token attention heads encode local context; later induction heads locate matching contexts and copy answers.
• The circuit works whether the matching source occurs before or after the target.
• With only left context, diffusion models do not outperform matched autoregressive models on induction.
• Diffusion models infer an implicit denoising timestep from the global fraction of masked tokens, without explicit timestep embeddings.