# ThRetNet reference This is a readable, runnable reference implementation of the current ThRetNet Quantum Class-Phase Attention model. It preserves the model and training ideas being tested while deliberately leaving out the Spark-specific execution path, the experiment scheduler, old architecture variants, and diagnostic machinery. The published geometry is full FP32 and has exactly 13,415,232 trainable parameters: ```text vocabulary rows 12,096 (12,095 payload tokens plus ) residual width 256 residual blocks 10 retention heads 8 key width per head 32 value width per head 48 MLP hidden width 464 memory banks 3 CPA classes 32 per phase CPA classifier width 1,024 ``` ## What the model does Each residual block contains one retention mixer and one MLP. The retention mixer maintains three independent recurrent memories: ```text prompt memory updated while prompt tokens are consumed reasoning memory updated while the generated trace is consumed final memory updated while the final answer is consumed ``` At a given phase, the model reads all memories created so far but updates only the current memory. Every source memory has its own query projection. Key, value, and gate projections are shared. The three independently normalized reads are concatenated and passed through one learned output projection. All rotary operations use one globally advancing position. The clock does not restart at phase boundaries. It rotates every source-bank query, the shared key, and the Class-Phase Attention blend. The Class-Phase Attention module contains: ```text one learned phase-direction vector one shared 256 -> 1024 -> 32 classifier one 32 x 256 class-vector table for each phase L2-normalized signed class coefficients one globally rotated class-vector blend added to the token embedding ``` The three class tables begin as exact copies but train independently. This is the "quantum" part of the current name: a token can acquire independently movable prompt, reasoning, and finalization classifications. Phase-specific additive-plus-scale "shock absorbers" surround both sides of: ```text the vocabulary interface Class-Phase Attention every retention mixer every MLP ``` Each shock computes: ```python y = (x + phase_offset) * (1 + phase_scale_delta) ``` Offsets and scale deltas start at zero. The vocabulary ingress shock is selected by the phase of the token being consumed. The vocabulary egress shock is selected by the phase that owns the token being predicted. That distinction is important at boundaries. ## CopyPrompt One training row is: ```text prompt prompt prompt ``` The second region is the reasoning trace and the third is the answer. Prompt loss is masked. Pure cross-entropy is applied to: ```text trace answer ``` The supplied first `` belongs to the prompt memory while its following logit predicts a reasoning token. The reasoning `` belongs to the reasoning memory while its following logit predicts a finalization token. The last `` terminates generation and is not consumed. Training payload tokens are sampled independently and uniformly, with replacement, from IDs 0 through 12,094. The included tokenizer is used for natural-text validation and inspection rather than to generate random training prompts. ## Simple UGI Simple Unigram Geometry Initialization runs after ordinary model initialization. It derives the expected target unigram distribution from the CopyPrompt loss grammar and the planned prompt lengths. Tokens whose probability is strictly greater than the vocabulary mean plus one population standard deviation are "offenders." Each offender keeps its ordinary random embedding direction and receives an angular exclusion cone whose share of hypersphere area is: ```text sqrt(mean token probability * offender probability) ``` Every non-offender keeps its original row unless its direction falls inside an offender cone. Such rows are redrawn from the ordinary N(0, 0.02) initializer until they lie outside all cones. UGI deliberately makes no norm intervention and uses a private random generator, so it does not advance the model RNG. ## Tokenizer `thretnet_reference/assets/tokenizer.json` is the exact tokenizer artifact used by the current CopyPrompt runs: ```text name BPE-12095-FineWeb-plus-EOR style GPT byte-level BPE, no normalization payload IDs 0..12094 ID 12095 SHA-256 8c4ec004fa8bc8e35321da8d590d701c32bbcade4ecf3076c0acbf633a3d508c ``` No FineWeb documents or training shards are bundled. The small validation text is original explanatory prose. Replace it with held-out FineWeb text to mirror the experiment's natural-token validation source. ## Install and run Create an environment with a suitable PyTorch build, then install the package: ```bash python -m pip install -e ".[dev]" ``` Run a small smoke training job: ```bash thretnet-train \ --output-dir reference-run \ --steps 20 \ --warmup-steps 5 \ --validation-rows 4 \ --evaluate-every 10 ``` Run the current fixed-length-four diagnostic shape for 18,000 updates: ```bash thretnet-train \ --output-dir length4-run \ --minimum-prompt-length 4 \ --maximum-prompt-length 4 \ --steps 18000 \ --warmup-steps 500 \ --peak-lr 0.000126148 ``` The peak LR is deliberately exposed because the experiments found a narrow, length-dependent response; the example value is a tested probe coordinate, not a claim of universal optimality. The defaults retain the current AdamW timescale: ```text beta1 0.9 beta2 0.9997996556404107 weight decay 0.000390625 epsilon 1e-8 ``` Use a linear WSD tail by supplying `--decay-steps` and `--minimum-lr`. A zero decay length keeps the learning rate constant after warmup, as in the recent diagnostic sweeps. Generate after training: ```bash thretnet-generate \ --checkpoint length4-run/checkpoint.pt \ --text "A state becomes the next input." \ --length 4 ``` Only the latest `checkpoint.pt` and the current invocation's `run.jsonl` are retained in an output directory. The JSONL log records startup configuration, UGI, training steps, validation, checkpoints, completion, and full exception context. ## Source map ```text config.py fixed current geometry and phase names rotary.py one global rotary clock shocks.py phase-specific add-then-scale boundaries retention.py literal normalized three-bank RetNet recurrence model.py Quantum CPA and ten-block residual model ugi.py Simple UGI hyperspherical initialization tokenizer.py verified tokenizer loader copyprompt.py data construction, phase routing, and masked CE schedule.py warmup/stable/optional-linear-decay schedule train.py plain AdamW training and validation harness generate.py EOR-driven recurrent generation ``` ## Reference versus optimized execution The production training path evaluates retention with parallel matrix operations. This package performs the normalized RetNet state update literally, one token at a time. It exposes the same recurrence, parameters, routing, and global positions, and its state-dict names are kept compatible with the current model. It is intentionally much slower on long sequences and FP32 association order need not be bit-identical to the optimized parallel path. There are no alternate model families or silent fallbacks in this package. To change the architecture, edit the small reference components directly.