[Blog] Graph Machine: Intuitions
Here I share two intuitions for Graph Machine (2), one algorithmic and one biological.
A tale of three temples
Inside a temple sits a scholar, along with their collection of books. The temple is sealed off from the outside world, except that each day one scroll is handed in and one scroll is handed out. The scholar reads the incoming scroll, which contains topics of study, consults the books on the shelves, produces an outgoing scroll, and may also modify or add to the temple’s internal collection. Moving takes no time, and the scholar’s actions are deterministic given what’s inside the temple. Days pass, and everything is measured relative to the number of days that have gone by.
Now, not all temples are created equal.
Temple 1
Temple 1 is the Temple of Recurrence. Each day, the incoming scroll is constant-sized, and inside the temple sits a constant-sized book. The scholar can study the scroll and the book as carefully as they like, but there is only so much to do. They produce a constant-sized output scroll, modify the book, and the day’s work finishes in constant time.
Temple 2
Temple 2 is the Temple of Attention. The incoming scroll is still constant-sized, but now the shelves contain a linear number of constant-sized books recording the scholar’s past daily work. No matter how fast the scholar scans each book, there are now so many books that the day’s work takes linear time. At the end, the scholar hands out a constant-sized output scroll, writes a new constant-sized book, adds it to the shelves, and calls it a day.
Temple 1 only requires constant-time work per day, but has a constant-sized collection. Temple 2 has a linear-sized collection, but requires linear-time work per day. Both exchange only constant-sized scrolls with the outside world. This leads us to wonder if a third temple exists that combines all three properties:
- Exchanges constant-sized scrolls;
- Has a linear-sized book collection;
- Requires constant-time work per day.
Of course, a linear-sized collection may be deceptive if most of its books are not “effectively there.” Therefore, before the arrival of the new scroll, we do not allow all but a constant number of books to be ruled out from the possibility of consuming the scholar’s time (and thus safely stowed away). We strengthen requirement 2:
- Has a non-pre-excludable linear-sized book collection;
A simple (secretly information-theoretic) proof shows that this temple does not exist.
No temple can satisfy these conditions
A constant-sized input scroll admits only a constant number of variations, inducing a constant number of outcomes in the temple. In each outcome, the constant-time restriction allows the scholar to spend time with at most a constant number of books. Therefore, even before the input scroll is revealed, the number of books the scholar could possibly spend time with is constant × constant = constant, breaking requirement 2.
Fortunately, this tells us that we can relax requirement 1 slightly while satisfying the others.
Temple 3
Temple 3 is the Temple of Edges. Here, the incoming scroll also includes a log-sized reference list with addresses for a constant number of books. Logarithmic size is enough because the books can be arranged into shelves, rows, columns, and so on, with their addresses expressed in those coordinates. The scholar follows the addresses and studies the books, producing an output scroll and adding a new book as before.
But if multiple temples are stacked together, the outgoing scroll of one temple becomes the incoming scroll of the next. The scholar also needs to produce a new reference list. Ideally, this new list should not merely copy the old one, but enrich the next scholar’s study. So we modify Temple 3 slightly.
Temple 3+
The books are no longer constant-sized; each now has its own log-sized reference list containing addresses of other books. The scholar reads the incoming addresses, visits the referenced books, and may then follow their references recursively for a constant number of hops. The scholar thereby reads a larger, but still constant-sized, collection of books. From this collection of books, the scholar produces the new reference list. Temple 3+ now not only accepts but also returns logarithmic-sized scrolls.
Of course, this is only a toy model of sequence modeling in ML.
The logarithmic-sized addresses in the model would correspond to things like int32 indices, which have the same bit-width as float32 values.
What matters is not input capacity itself, but the capacity directly usable for addressing.
The model also abstracts away the cost of moving between books.
Moreover, Temple 2 and a slightly modified Temple 3+ admit parallelism, while Temple 1 does not.
Nonetheless, the takeaway for this setting remains: to have a state that grows at least linearly while doing only constant work, one needs addressable inputs.
Gathering fruits → gathering memories
Why did human general intelligence evolve within the primate lineage?
One hypothesis is that complex foraging created an unusual selection pressure on working memory (WM).
To track handholds and fruits while foraging, items in WM must be swapped in and out rapidly and flexibly — a gather-like operation.
Humans may have evolved to increasingly draw on other cognitive machinery, such as long-term memory, for these swaps, rather than on visuospatial perception. This is consistent with the cognitive trade-off hypothesis: humans evolved complex language and abstract thinking partly at the expense of exceptional short-term visual memory abilities still seen in chimpanzees.
We could view a sequential GM as an implementation of such a model of the memory system. Time unfolds along the token axis; thinking unfolds along the depth axis. The latest token represents the current mental state: the targets of its edges can be viewed as the items in WM, while its node features represent other aspects of that state. Across layers, its edges update to point to different tokens via referral or search, progressively developing the contents of WM. The latent graph formed by past tokens and their edges becomes the memory that the model can use in the future through direct search or structural traversal. This further allows forgetting to be modeled via graph-based pruning, offering a plausible foundation for continual learning.