🛰️ Daily AI Frontier
‹ back to 2026-07-20

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Research LLMs & Foundation Models

Ranking

Overall 86
Content 95
Popularity 65

Observed public metrics from 1 member.

Merged summary

TL;DR - Frontier LLMs can fail at exact copying despite sufficient context capacity. The paper proposes 2D-RoPE, a positional encoding that arranges text in a grid and substantially improves copying generalization.

  • The authors attribute copying failures to standard Transformer positional encodings favoring local-context shortcuts.
  • 2D-RoPE gives each token row and column coordinates, turning copying into retrieval at a fixed column offset.
  • Shallow models copied sequences hundreds of times longer than their training inputs perfectly in synthetic experiments.
  • Benefits persisted during DCLM pretraining for models up to 1.4B parameters.

Sources (1)

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

arXiv cs.CL Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu 2026-07-17 arXiv:2607.16072
Public signals Hugging Face upvotes 1
Providers: Hugging Face · Upvotes 1 OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-08-18 14:39:27.857415 UTC

TL;DR - Frontier LLMs can fail at exact copying despite sufficient context capacity. The paper proposes 2D-RoPE, a positional encoding that arranges text in a grid and substantially improves copying generalization.

  • The authors attribute copying failures to standard Transformer positional encodings favoring local-context shortcuts.
  • 2D-RoPE gives each token row and column coordinates, turning copying into retrieval at a fixed column offset.
  • Shallow models copied sequences hundreds of times longer than their training inputs perfectly in synthetic experiments.
  • Benefits persisted during DCLM pretraining for models up to 1.4B parameters.
item →