Graph Machine: Towards Better Pretraining via Edges
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - Graph Machine is a sparse architecture that maintains an O(n)-sized state while dynamically retrieving only a few entries through differentiably updated, pointer-like edges. Replacing most dense layers in a 0.6B-parameter Transformer preserved—and in one setting marginally improved—pretraining loss despite extremely sparse retrieval.
- GM combines O(n) sparse-layer complexity with access to a state whose size can grow with sequence length.
- Its dynamic routing uses edges updated through a differentiable referral mechanism resembling pointer chasing.
- The authors replaced 75% of Qwen3-0.6B’s dense Transformer layers and pretrained the model from scratch on 15.7B tokens.
- Retrieving 2 of 4,096 tokens per KV head caused only slight loss degradation; retrieving 4 yielded the best model and marginally improved loss.
Sources (1)
Graph Machine: Towards Better Pretraining via Edges
TL;DR - Graph Machine is a sparse architecture that maintains an O(n)-sized state while dynamically retrieving only a few entries through differentiably updated, pointer-like edges. Replacing most dense layers in a 0.6B-parameter Transformer preserved—and in one setting marginally improved—pretraining loss despite extremely sparse retrieval.
- GM combines O(n) sparse-layer complexity with access to a state whose size can grow with sequence length.
- Its dynamic routing uses edges updated through a differentiable referral mechanism resembling pointer chasing.
- The authors replaced 75% of Qwen3-0.6B’s dense Transformer layers and pretrained the model from scratch on 15.7B tokens.
- Retrieving 2 of 4,096 tokens per KV head caused only slight loss degradation; retrieving 4 yielded the best model and marginally improved loss.