First moment
Signed, finite, round-to-nearest ties-to-even, with exact zero and finite saturation.
Project 04 · Reference 1.0.0
Packed
AdamW
Two persistent Adam moments stored in one 32-bit word per FP32 learned parameter. The model and the current optimizer update remain FP32.
- Moment state
- 8 → 4 bytes / parameter
- Reference model
- 11,637,760 parameters
- Release
- v1.0.0 · CC0
Concept
Compress the history, not the update
AdamW carries a moving average of gradients and a moving average of squared gradients for every learned parameter. Ordinary FP32 AdamW therefore keeps eight persistent optimizer bytes alongside each four-byte FP32 weight.
Packed AdamW asks whether those histories need the full FP32 fraction between updates. It stores both in custom 16-bit floating-point lanes and expands them only inside the optimizer update tile. Weights, gradients, activations, bias correction, weight decay, and the current parameter update remain FP32.
Persistent state format
One parameter, one word
Custom finite software encodings—not native NVIDIA E4 instruction formats.
High 16 bitsUnsigned finite E4M12
Uncorrected RMS · 4 exponent bits · 12 fraction bits
Low 16 bitsSigned finite E4M11
First moment · sign · 4 exponent bits · 11 fraction bits
Root second moment
Nonnegative RMS with no sign bit, so the lane spends that bit on another fraction position.
Checkpoint form
The packed word remains uint32 in checkpoints; decoded FP32 shadow moments are never serialized.
The square-root technique
Store RMS instead of the raw second moment
AdamW normally carries the uncorrected second moment, v, and later takes its square root for the adaptive denominator. Packed AdamW instead stores r = sqrt(v). On the next update it decodes r and reconstructs the prior estimate as v = r × r.
The transformation is useful for representation, not because it removes Adam's square root. RMS occupies a friendlier numerical range than the raw squared-gradient history. It is also always nonnegative, so its 16-bit lane needs no sign bit. That recovered bit becomes a twelfth fraction bit, giving the RMS lane more local precision while retaining the dynamic range of a four-bit exponent.
Squaring a decoded RMS is not algebraically identical to quantizing the raw second moment. It introduces its own representation error, and finite bounds that worked in the demonstrated run may not fit other tasks. The reference records underflow, saturation, and nonfinite events rather than silently changing formats.
Update ordering
Decode, update, then repack
Decode prior state
Load the FP32 parameter and gradient, then decode the signed first moment and nonnegative RMS from one uint32 word.
Reconstruct the second moment
Square the decoded RMS to recover the prior second-moment estimate, then calculate the next first and second moments in FP32.
Update in FP32
Apply bias correction, decoupled weight decay, and the current adaptive parameter update before either new moment is quantized.
Pack the next history
Take the square root of the new uncorrected second moment and pack it with the new first moment for the following update.
Memory accounting
Half the moments, not half the training footprint
| Persistent or narrow item | FP32 AdamW | Packed AdamW |
|---|---|---|
| FP32 weights | 4P bytes | 4P bytes |
| FP32 gradients | 4P bytes | 4P bytes |
| Persistent moments | 8P bytes | 4P bytes |
| Narrow subtotal | 16P bytes | 12P bytes |
| Ideal optimizer traffic | 28P bytes | 20P bytes |
50%less persistent moment state
25%less in the weights + gradients + moments subtotal
4 GBmoment-state capacity saved per billion parameters
Activations, attention intermediates, logits, temporary buffers, allocator reserves, and framework memory are outside this ledger. One live sample showed about an 18% allocation difference; it was not a proven peak-memory measurement.
Matched experiment
One model, one seed, three state formats
11,637,760 parameters · 203,920 updates · identical FP32 training arithmetic
| Persistent state | Validation BPB | Wall time | Moment bytes / parameter |
|---|---|---|---|
| FP32 first + raw second | 1.38280916 | 3,604.599 s | 8 |
| FP16 first + RMS | 1.40645291 | 3,515.610 s | 4 |
| E4M11 first + E4M12 RMS | 1.38687403 | 3,502.284 s | 4 |
+0.00406487 BPBcustom E4 final gap from FP32 · approximately +0.294%
−2.92% wall timeobserved end-to-end in this small sequential run