Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Ranking
Overall
76
Content
90
Popularity
42
Observed public metrics from 1 member.
Merged summary
TL;DR - Möbius RoPE introduces anti-periodic positional encoding that substantially improves the reliability of in-context retrieval without degrading perplexity. It offers a zero-cost architectural change for reducing retrieval failures within the training context window.
- Uses an odd-frequency ladder that couples sequence endpoints through an anti-periodic boundary condition.
- At 160M scale, hybrid Möbius RoPE achieved 90.3% needle retrieval versus 63.3% for standard RoPE, with much lower seed variance.
- Results recurred at 410M scale; matched controls and frozen-weight frequency swaps support long-range geometry as the mechanism.
- Benefits are limited to single-needle retrieval within the training window; NoPE performed better at short context but incurred a 13% perplexity penalty and extrapolated poorly.
Sources (1)
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - Möbius RoPE introduces anti-periodic positional encoding that substantially improves the reliability of in-context retrieval without degrading perplexity. It offers a zero-cost architectural change for reducing retrieval failures within the training context window.
- Uses an odd-frequency ladder that couples sequence endpoints through an anti-periodic boundary condition.
- At 160M scale, hybrid Möbius RoPE achieved 90.3% needle retrieval versus 63.3% for standard RoPE, with much lower seed variance.
- Results recurred at 410M scale; matched controls and frozen-weight frequency swaps support long-range geometry as the mechanism.
- Benefits are limited to single-needle retrieval within the training window; NoPE performed better at short context but incurred a 13% perplexity penalty and extrapolated poorly.