‹ back to 2026-07-20

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

arXiv cs.CL LLMs & Foundation Models Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu 2026-07-17

TL;DR - Frontier LLMs can fail at exact copying despite sufficient context capacity. The paper proposes 2D-RoPE, a positional encoding that arranges text in a grid and substantially improves copying generalization.

The authors attribute copying failures to standard Transformer positional encodings favoring local-context shortcuts.
2D-RoPE gives each token row and column coordinates, turning copying into retrieval at a fixed column offset.
Shallow models copied sequences hundreds of times longer than their training inputs perfectly in synthetic experiments.
Benefits persisted during DCLM pretraining for models up to 1.4B parameters.

view merged work →