ThRetNet / Reference / v0.1
Reference releases
Runnable training and atomic-capability reference for the earlier three-bank model.
- Package
- 0.1.0 · Previous
- Runtime
- Python 3.11+
- Parameters
- 13,415,232
- Files
- 21
Archive SHA-2566d4ba3d7b27ad4fd0e01bd41342a3e3f776629bf60db394960141b34be2f7c2c
Available releases
Choose a version
Why v1.3
The prompt memory splits in two
The defining change in v1.3 is the bifurcation of the prompt memory bank into reasoning-owned and finalization-owned banks. During prompt prefill, the same tokens are processed in two synchronized branches. Each subsequent phase therefore reads a prompt representation formed in its own class space, preventing conflicting class interpretations from competing inside a shared prompt bank.
Because the prompt is no longer treated as an independent compute phase, v1.3 eliminates the prompt-phase read heads and prompt-only class-phase parameter sets. Structured butterfly projections separately reduce the weight of the memory mixer. Together, these changes reduce the complete per-layer block from 999,120 to 715,520 parameters—about 28 percent.
That reduction made room to increase the MLP width from 464 to 512 and expand the network from 10 to 13 layers while reducing the representative model from about 13.4 million to 12.7 million parameters. The residual width remains 256.
Release comparison
What changed
v1.3 changes the scope of the reference package as well as the model. The older training harness and capability builders remain available in v0.1.
| Area | v1.3 · Latest | v0.1 · Previous |
|---|---|---|
| Primary purpose | Architecture, correctness, and portability reference | Runnable training and atomic-capability reference |
| Prompt memory | Reasoning-owned and finalization-owned prompt banks; four banks total | One shared prompt bank; three banks total |
| Prompt phase | Two synchronized prompt branches; no prompt-only read heads or class table | Prompt, reasoning, and finalization are independently parameterized phases |
| Geometry | 13 layers · residual width 256 · MLP width 512 | 10 layers · residual width 256 · MLP width 464 |
| Parameters | 12,716,064 in the README example configuration | 13,415,232 in the published package configuration |
| Included material | Architecture specification, checkpoint and parity tests, portable model, isolated DGX Spark artifact | Tokenizer, UGI, CopyPrompt, AtomicOrder, training and generation tools |
Selected release
v0.1: previous reference
This release evaluates the normalized RetNet state update literally, one token at a time. It includes the tokenizer, UGI implementation, CopyPrompt and AtomicOrder builders, and training and generation tools that are outside the narrower v1.3 correctness package.
v0.1 repository snapshot
Files
- thretnet-reference-v0.1
- README.md
7.5 KB · MD