Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable

TL;DR - Möbius RoPE introduces anti-periodic positional encoding that substantially improves the reliability of in-context retrieval without degrading perplexity. It offers a zero-cost architectural change for reducing retrieval failures within the training context window.

• Uses an odd-frequency ladder that couples sequence endpoints through an anti-periodic boundary condition.
• At 160M scale, hybrid Möbius RoPE achieved 90.3% needle retrieval versus 63.3% for standard RoPE, with much lower seed variance.
• Results recurred at 410M scale; matched controls and frozen-weight frequency swaps support long-range geometry as the mechanism.
• Benefits are limited to single-needle retrieval within the training window; NoPE performed better at short context but incurred a 13% perplexity penalty and extrapolated poorly.