Packed AdamW / Reference / v1.0.0
Packed optimizer
Custom packed E4 moment state, fused Triton update, independent scalar authority, finite-tape GPT trainer, tests, and release tooling.
- Package
- 1.0.0 · Latest
- Runtime
- Python 3.11 · NVIDIA CUDA
- Reference model
- 11,637,760 parameters
- Files
- 56
Archive SHA-256343bce5ee32f325503b1ba07c8f57369035489f7d0737bd3e7f2e869475ed59d
Available releases
Choose a version
Release scope
A narrow optimizer-state reference
Version 1.0.0 implements one uint32 state word per FP32 learned parameter: custom signed finite E4M11 for the first moment and custom unsigned finite E4M12 for uncorrected RMS. There are no FP32 shadow moments, block scales, residual state, or stochastic rounding.
The optimizer decodes prior state, reconstructs the prior second moment by squaring RMS, calculates the next moments, and applies the current AdamW update in FP32. Only afterward does it store the next first moment and RMS for the following update.
Ordinary FP32 AdamW and packed FP16-first/FP16-RMS modes are explicit comparison choices in the release. They are not automatic fallbacks, and the packed formats do not convert themselves when a finite bound is reached.
Package contents
Source, tests, and experiment contract
The release includes scalar codecs, an independent scalar AdamW authority, fused Triton update kernels, a reusable dense optimizer, a small learned-position GPT, finite sequential token-tape training, exact-resume checkpoints, format and CUDA tests, documentation, release tooling, the 12,095-entry tokenizer artifact, and corpus provenance without corpus data.
It contains no FineWeb rows, encoded corpus tapes, trained weights, checkpoints, scheduler, monitor, interactive tuning application, or deployment configuration. The synthetic smoke path checks one finite update and is not evidence of learning.
Execution contract
Dense CUDA FP32 parameters
The reference favors an explicit, inspectable contract over a general optimizer API.
| Area | v1.0.0 contract |
|---|---|
| Parameters and gradients | Contiguous CUDA FP32 tensors on one device |
| Gradient availability | Dense gradient for every parameter on every update |
| Optimizer grouping | One parameter group and one global step |
| Kernel structure | One fused launch per parameter tensor; not a persistent whole-model megakernel |
| Model execution | Ordinary PyTorch forward and backward |
v1.0.0 repository snapshot
Files
- packed-adamw-reference-v1.0.0
- README.md
5.4 KB · MD