[Paper] Graph Machine: Towards Better Pretraining via Edges
Overview
Abstract
We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
Figures
Figure 1. Attention and referral viewed as pointer operations.
Figure 2. Different referral approaches with ℓ = 2 (simplified).
Figure 3. GM states.
Figure 4. Adjacency matrices → edge states.
Figure 5. GM layers.
Figure 6. A sparse GM layer.
Paper
Citation
Text
Hou, Lintai. "Graph Machine: Towards Better Pretraining via Edges." arXiv preprint arXiv:2609.02881 (2026). https://arxiv.org/abs/2609.02881.BibTeX
@misc{hou2026gm2,
title = {Graph Machine: Towards Better Pretraining via Edges},
author = {Hou, Lintai},
year = {2026},
eprint = {2609.02881},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2609.02881}
}