Project 01
Three Retention Networks
Original RetNet paperRetentive Network: A Successor to Transformer for Large Language ModelsYutao Sun et al. · 2023 · arXiv:2307.08621Read the paper ↗ThRetNet is an independent research project and is not affiliated with the authors or institutions behind the original RetNet paper.
Architecture versions
The selected release controls the architecture explanation and reference links on this page.
v1.3 · Latest
ThRetNet retains the project’s original name and three-phase organization—prompt, reasoning, and finalization—but v1.3 uses four physical RetNet memory banks. The original prompt bank is bifurcated into a reasoning-owned prompt bank (P_R) and a finalization-owned prompt bank (P_F), alongside the reasoning (R) and finalization (F) banks.
Prompt tokens are processed through two synchronized branches. One branch writes P_R using reasoning-phase parameters; the other writes P_F using finalization-phase parameters. During generation, reasoning tokens write R and finalization tokens write F. The banks visible to the active phase are merged through learned mixing, while each generated token still writes only to its phase-owned bank.
Phase switching is controlled by the surrounding harness. The harness recognizes configurable boundary tokens, such as <think> and <EOR>, and uses them to select the reasoning or finalization compute phase and its active write bank.
Class-phase attention assigns tokens an additional phase-dependent position in embedding space. It calculates residual-width class vectors and adds them to the token embeddings.
This addition acts as a geometric sortation. Tokens belonging to the same class retain their relationships to one another, while different phase classes can be moved far apart in embedding space. The model can therefore distinguish different uses of a token without discarding the token geometry within each class.
The analogy is to classes in an object-oriented programming language: the underlying token remains recognizable, but its phase class changes how the network interprets and processes it. I call this Object Oriented Inference. In v1.3 the classifier is shared, while the reasoning and finalization phases own distinct class tables; there is no separate prompt-only table.
Because class-phase attention gives tokens different phase-dependent meanings, several components of the network also have phase-specific forms. These are called “Quantum” components and are identified by colored dots in the earlier architecture diagram below.
In v1.3 a Quantum component has two copies in the model weights—one for reasoning and one for finalization. Prompt content is processed once under each downstream phase’s selection, producing the P_R and P_F memories. “Quantum” is a project-specific term for this phase-dependent multiplicity; it does not refer to quantum computing.
During training, gradient contributions accumulate in the copy owned by the phase that receives the loss. Keeping the reasoning and finalization components distinct is intended to reduce destructive interference and stabilize the phase-specific components of what I call the quantum gradient.
Shock absorbers are small residual-width shims placed around phase-switched components. Each performs a diagonal operation consisting of a vector addition and an element-wise scale.
These shims give the network a lightweight way to adjust for the changes introduced by phase-specific parameters. Their purpose is to absorb abrupt shifts associated with quantum gradients and improve training stability.
Simple Unigram Geometry Initialization, or Simple UGI, is an embedding-table initialization technique developed to stabilize some earlier ThRetNet training runs. It identifies unusually high-frequency tokens as “offenders” and reserves an angular exclusion cone around each offending token’s initial embedding direction, with the size of that region derived from the token’s expected frequency. Other token embeddings are initialized outside those reserved regions.
This is not a packing scheme: the vocabulary is not redistributed across the available embedding space. On tasks such as CopyPrompt, the reserved regions give the model a cleaner initial geometry for distinguishing frequent target tokens, making its initial process of elimination easier. The implementation is preserved in the v0.1 reference package; v1.3 is an architecture and correctness reference and does not include the earlier training harness or UGI tooling.
Release context
The central change in v1.3 is the bifurcation of the prompt-phase memory bank. Reasoning and finalization now receive separate prompt memories formed in their own class spaces, so tokens with conflicting phase classes no longer compete for one prompt representation.
The two prompt branches advance in lockstep over the same tokens. This preserves synchronized token history while eliminating prompt-phase read heads and prompt-only class-phase parameter sets: the prompt is represented through the two downstream compute phases instead of acting as a third compute phase.
The prompt-bank split removes prompt-phase heads and parameter sets. Structured butterfly projections separately reduce the memory mixer. Together, the reference configuration reduces each block from 999,120 to 715,520 parameters—about 28%.
That reduction makes room to increase the MLP width from 464 to 512 and the depth from 10 to 13 layers while keeping residual width 256. Under the documented example configurations, the model moves from 13,415,232 parameters in v0.1 to 12,716,064 in v1.3 while becoming wider and deeper.
The reasoning-owned prompt bank learns through reasoning loss; the finalization-owned prompt bank learns through finalization loss. Finalization can also read reasoning memory, but that read is backward-detached so finalization loss does not update the reasoning-owned state through that path.
This gives each branch an explicit learning owner while still allowing the final answer to use the completed reasoning state.
Design problems
ThRetNet is designed to be trained on “atomic capabilities”: small synthetic datasets constructed around a single, isolated capability. Each dataset teaches the network how to do one narrowly defined thing.
This controlled setting makes it possible to study which architectural changes enable, improve, damage, or eliminate a capability in a very small network. The longer-term goal is to identify principles that can be tested in larger networks with more complex capabilities.
The original ThRetNet experiments used models with roughly 12 million trainable parameters. The selected v1.3 reference configuration documents 12,716,064 parameters (readme example: vocabulary 12,098 and 32 classes). The previous v0.1 package documents 13,415,232 parameters. Both remain small enough for architectural changes to be studied directly.
ThRetNet uses an “all-SFT” training approach with the loss masked over the prompt phase. The prompt supplies the problem, but its tokens do not contribute to the training loss. The model is instead supervised as it constructs a reasoning trace and produces its answer.
The architecture is intended to support reasoning traces whose length is not bounded by a growing KV cache. As the model reasons, its RetNet memory is designed to retain the most recent significant details in a fixed-size recurrent state.
This does not imply literally unlimited computation or memory. Rather, the reasoning trace does not require a new cached key and value for every generated token.
ThRetNet includes two mechanisms intended to recover useful forms of positional structure while remaining entirely within a RetNet memory block.
Its global rotary clock is designed to provide RoPE-like positional learning. Class-phase attention provides phase-dependent geometric offsets that reproduce certain useful properties of Absolute Positional Embeddings. Together, they give the network information about token order and prompt phase without introducing a KV cache that grows with sequence length.
This makes ThRetNet a test bed for whether recurrent memory can support structured, extended reasoning while retaining positional properties normally associated with Transformer attention.
RetNet was chosen for the memory banks because its retention mechanism can be parallelized during training, making it a natural fit for GPU hardware. During generation, the same mechanism can operate recurrently, carrying a fixed-size state forward without an ever-growing KV cache. This combination keeps training accelerator-friendly while providing persistent memory for long reasoning traces.
Figure 01 · v1.3
This diagram documents the current v1.3 architecture, including the bifurcated prompt memory and its phase-owned reasoning and finalization paths. The formal structure is described in the v1.3 specification.
Current tasks
ThRetNet is currently evaluated on two atomic capabilities. Each task is intentionally narrow so that changes in capability can be attributed more directly to changes in the model or training process.
Capability 01
AtomicOrder teaches the model to traverse a small graph and determine the relative order of two nodes within it. The task isolates the model’s ability to follow relational structure and use the traversed relationships to make a comparison.
Capability 02
CopyPrompt teaches the model to copy the prompt into both its reasoning trace and its final answer. Because the prompt content must be retained and recovered across phase boundaries, the task directly tests ThRetNet’s recall mechanism.