# Packed AdamW Reference

This is a standalone reference implementation of AdamW with two persistent
moment values packed into one 32-bit word per learned parameter.

The canonical format is:

- low 16 bits: custom signed finite E4M11 first moment;
- high 16 bits: custom unsigned finite E4M12 uncorrected RMS;
- one `uint32` state value per FP32 learned parameter;
- no FP32 shadow moments, block scales, residual state, or stochastic rounding.

Weights, activations, gradients, bias correction, weight decay, and the current
optimizer update remain FP32. Only the moment state carried between updates is
quantized. The newly calculated moments are used in FP32 for the current
parameter update and are packed afterward for the next update.

The package also contains ordinary FP32 AdamW and packed FP16-first/FP16-RMS
comparison modes. They are explicit test and benchmark choices, not fallbacks.

## What is included

- `src/packed_adamw/formats.py` — readable scalar codecs for all packed lanes.
- `src/packed_adamw/reference.py` — independent scalar AdamW authority.
- `src/packed_adamw/kernels.py` — fused Triton update kernels.
- `src/packed_adamw/optimizer.py` — reusable dense optimizer interface.
- `src/packed_adamw/model.py` — lean learned-position GPT implementation.
- `src/packed_adamw/train.py` — finite, sequential token-tape trainer.
- `configs/reference-12m.json` — the demonstrated V12096 comparison setup.
- `configs/smoke.json` — exactly one synthetic update for installation checking.
- `tokenizer/tokenizer.json` — the exact 12,095-entry BPE tokenizer artifact.
- `corpus-contract.json` — storage, vocabulary, shard, and RDQ provenance without data.
- `tests/` — CPU format tests and CUDA kernel/checkpoint agreement tests.
- `docs/` — concepts, exact formats, update order, accounting, and limitations.
- `scripts/` — deterministic release construction and archive verification.

It deliberately contains no FineWeb rows, encoded corpus tapes, trained weights,
checkpoints, scheduler, monitor, interactive tuning application, deployment
configuration, or unrelated model family.

## The narrow result

In one matched, single-seed, one-RDQ experiment, the custom E4 state finished at
1.386874 validation bits per byte versus 1.382809 for FP32 AdamW, a difference
of +0.004065 BPB or about 0.294%. Persistent Adam state fell from 8 to 4 bytes
per parameter. End-to-end wall time improved by about 2.92% in that small run.

This is evidence for this configuration, not proof of universal equivalence.
See `docs/experiment-results.md` for the complete comparison and limitations.

## Dependency-light inspection

Python 3.11 or newer is sufficient for scalar format inspection. NumPy is only
needed for tape utilities. These commands do not train a model:

```text
python -m pip install -e .
python -m packed_adamw inspect-formats
python examples/inspect_update.py
```

`inspect-formats` intentionally does not import Torch or Triton.

## NVIDIA container path

The supplied container starts from `nvcr.io/nvidia/pytorch:26.05-py3`, which
provides a known recent PyTorch, CUDA, and Triton combination. The host needs a
compatible NVIDIA driver, Docker, and NVIDIA Container Toolkit.

```text
docker build -t packed-adamw-reference .
docker run --rm --gpus all packed-adamw-reference doctor
docker run --rm --gpus all --entrypoint python3 packed-adamw-reference \
  -m pytest -q -m cuda /opt/packed-adamw-reference/tests
```

To test an extracted source tree rather than the copy baked into the image:

```text
docker run --rm --gpus all -v /absolute/project/path:/workspace \
  --entrypoint python3 packed-adamw-reference \
  -m pytest -q /workspace/tests
```

## One-update smoke test

The smoke command creates deterministic random tapes and executes exactly one
optimizer update. It checks installation and numerical sanity only.

```text
docker run --rm --gpus all -v /absolute/project/path:/workspace \
  packed-adamw-reference smoke \
  --config /opt/packed-adamw-reference/configs/smoke.json \
  --output-dir /workspace/outputs/smoke
```

Require finite train and validation loss, zero optimizer non-finite count, a
one-word state ledger, and a successful checkpoint round trip. Never interpret
synthetic loss as a learning benchmark.

## Training on prepared tapes

The trainer consumes a little-endian uint32 token tape and an optional aligned
little-endian uint16 byte-length tape. It advances sequentially and never wraps.

```text
packed-adamw train \
  --config configs/reference-12m.json \
  --train-tokens data/train.tokens.u32 \
  --train-byte-lengths data/train.byte-lengths.u16 \
  --validation-tokens data/validation.tokens.u32 \
  --validation-byte-lengths data/validation.byte-lengths.u16 \
  --output-dir outputs/reference-run
```

The reference configuration fixes batch 1, 1,152 tokens, 203,920 updates,
10,196 warmup updates, historical linear decay, validation every 1,024 updates,
and a latest-only exact-resume checkpoint every 4,096 updates. The original
run's 1-RDQ stopping point depends on the supplied byte-length tape; the trainer
will not silently wrap a short tape.

## Integrity and license

Every release ZIP contains `MANIFEST.sha256`. Verify it before extraction or
execution:

```text
python scripts/verify_release.py packed-adamw-reference-v1.0.0.zip
```

The original reference source and bundled tokenizer artifact are dedicated to
the public domain under CC0 1.0. Dependencies retain their own terms; see
`THIRD_PARTY.md`.