Project 01
Three Retention Networks
Original RetNet paperRetentive Network: A Successor to Transformer for Large Language ModelsYutao Sun et al. · 2023 · arXiv:2307.08621Read the paper ↗ThRetNet is an independent research project and is not affiliated with the authors or institutions behind the original RetNet paper.
Architecture versions
The selected release controls the architecture explanation and reference links on this page.
v0.1 · Previous
ThRetNet v0.1 is organized around three phase-specific RetNet memory banks. One bank belongs to the prompt proper, one to the reasoning trace, and one to the final answer.
Every forward pass reads from all three banks. Their contents are merged through a learned mixing operation, shown in the v0.1 architecture diagram as the “Learned bank combination.” Tokens are written to only one bank: the bank associated with the phase in which each token appears.
Phase switching is controlled by the surrounding harness. The harness recognizes configurable boundary tokens, such as <think> and <EOR>, and uses them to determine the current prompt phase and active write bank.
Class-phase attention assigns tokens an additional phase-dependent position in embedding space. It calculates residual-width class vectors and adds them to the token embeddings.
This addition acts as a geometric sortation. Tokens belonging to the same class retain their relationships to one another, while different phase classes can be moved far apart in embedding space. The model can therefore distinguish between the same token appearing in the prompt, reasoning trace, or answer without discarding the token geometry within each class.
The analogy is to classes in an object-oriented programming language: the underlying token remains recognizable, but its phase class changes how the network interprets and processes it. I call this Object Oriented Inference.
Because class-phase attention gives tokens different phase-dependent meanings, several components of the network also have phase-specific forms. These are called “Quantum” components and are identified by colored dots in the architecture diagram.
A v0.1 Quantum component has three copies in the model weights—one for the prompt, one for the reasoning trace, and one for the answer. The active copy is selected according to the current prompt phase. “Quantum” is a project-specific term for this phase-dependent multiplicity; it does not refer to quantum computing.
During training, gradient contributions from each phase accumulate in their corresponding parameter copies. This produces separate prompt, reasoning, and answer components of what I call the quantum gradient. Keeping those components distinct is intended to reduce destructive interference between prompt phases and stabilize training.
Shock absorbers are small residual-width shims placed around phase-switched components. Each performs a diagonal operation consisting of a vector addition and an element-wise scale.
These shims give the network a lightweight way to adjust for the changes introduced by phase-specific parameters. Their purpose is to absorb abrupt shifts associated with quantum gradients and improve training stability.
Simple Unigram Geometry Initialization, or Simple UGI, is an embedding-table initialization technique developed to stabilize some ThRetNet training runs. It identifies unusually high-frequency tokens as “offenders” and reserves an angular exclusion cone around each offending token’s initial embedding direction, with the size of that region derived from the token’s expected frequency. Other token embeddings are initialized outside those reserved regions.
This is not a packing scheme: the vocabulary is not redistributed across the available embedding space. On tasks such as CopyPrompt, the reserved regions give the model a cleaner initial geometry for distinguishing frequent target tokens, making its initial process of elimination easier.
Release context
v0.1 documents the earlier three-bank architecture, with prompt, reasoning, and answer as separate compute phases. It also preserves the runnable training and atomic-capability tooling that is not included in the narrower v1.3 architecture and correctness package.
Select v1.3 above to see why the prompt bank was bifurcated and how that change allowed the later model to become wider and deeper with fewer total parameters.
Design problems
ThRetNet is designed to be trained on “atomic capabilities”: small synthetic datasets constructed around a single, isolated capability. Each dataset teaches the network how to do one narrowly defined thing.
This controlled setting makes it possible to study which architectural changes enable, improve, damage, or eliminate a capability in a very small network. The longer-term goal is to identify principles that can be tested in larger networks with more complex capabilities.
The original ThRetNet experiments used models with roughly 12 million trainable parameters. The selected v0.1 reference configuration documents 13,415,232 parameters (published package configuration). The newer v1.3 README example documents 12,716,064 parameters while expanding the model’s width and depth. Both remain small enough for architectural changes to be studied directly.
ThRetNet uses an “all-SFT” training approach with the loss masked over the prompt phase. The prompt supplies the problem, but its tokens do not contribute to the training loss. The model is instead supervised as it constructs a reasoning trace and produces its answer.
The architecture is intended to support reasoning traces whose length is not bounded by a growing KV cache. As the model reasons, its RetNet memory is designed to retain the most recent significant details in a fixed-size recurrent state.
This does not imply literally unlimited computation or memory. Rather, the reasoning trace does not require a new cached key and value for every generated token.
ThRetNet includes two mechanisms intended to recover useful forms of positional structure while remaining entirely within a RetNet memory block.
Its global rotary clock is designed to provide RoPE-like positional learning. Class-phase attention provides phase-dependent geometric offsets that reproduce certain useful properties of Absolute Positional Embeddings. Together, they give the network information about token order and prompt phase without introducing a KV cache that grows with sequence length.
This makes ThRetNet a test bed for whether recurrent memory can support structured, extended reasoning while retaining positional properties normally associated with Transformer attention.
RetNet was chosen for the memory banks because its retention mechanism can be parallelized during training, making it a natural fit for GPU hardware. During generation, the same mechanism can operate recurrently, carrying a fixed-size state forward without an ever-growing KV cache. This combination keeps training accelerator-friendly while providing persistent memory for long reasoning traces.
Figure 01 · v0.1
This diagram documents v0.1. The current v1.3 architecture bifurcates the prompt bank; its formal structure is documented in the v1.3 specification.
Current tasks
ThRetNet is currently evaluated on two atomic capabilities. Each task is intentionally narrow so that changes in capability can be attributed more directly to changes in the model or training process.
Capability 01
AtomicOrder teaches the model to traverse a small graph and determine the relative order of two nodes within it. The task isolates the model’s ability to follow relational structure and use the traversed relationships to make a comparison.
Capability 02
CopyPrompt teaches the model to copy the prompt into both its reasoning trace and its final answer. Because the prompt content must be retained and recovered across phase boundaries, the task directly tests ThRetNet’s recall mechanism.