Graph Machine: Towards Better Pretraining via Edges

TL;DR - Graph Machine is a sparse architecture that maintains an O(n)-sized state while dynamically retrieving only a few entries through differentiably updated, pointer-like edges. Replacing most dense layers in a 0.6B-parameter Transformer preserved—and in one setting marginally improved—pretraining loss despite extremely sparse retrieval.

• GM combines O(n) sparse-layer complexity with access to a state whose size can grow with sequence length.
• Its dynamic routing uses edges updated through a differentiable referral mechanism resembling pointer chasing.
• The authors replaced 75% of Qwen3-0.6B’s dense Transformer layers and pretrained the model from scratch on 15.7B tokens.
• Retrieving 2 of 4,096 tokens per KV head caused only slight loss degradation; retrieving 4 yielded the best model and marginally improved loss.