KV cartridges (GPU)

A KV cartridge is one of mnesio’s frontier bets (Phase 12): instead of re-reading a long memory context through the model on every query, you compile the relevant memories’ attention key/value state once into a cartridge, then reuse it. The cartridge is just another materialized view of the log — keyed and rebuildable by replay, like every other index.

Status: shipped (Phase 12 ✅)

The cartridge substrate (keying, versioning, gate-before-activation, crypto-shred-by-recompile) is complete and tested, and the open-weights tensor backend is delivered: real GPT-2 (incl. real q8 quant) and Qwen2.5 in pure Rust and on Metal GPU — all behind the KvBackend trait. A suite-level accuracy-parity eval (mnesio-bench kveval) shows the cartridge answers at least as accurately as per-query text-context retrieval (~167–180× faster), with erasure holding. The default build still ships the dependency-free FakeKvBackend; the real backends are feature-gated and fetch weights on first use (with HF_HUB_OFFLINE-aware errors). The candle backend also accepts Llama-family configs; Qwen2 is the live-verified path.

How it’s keyed

A cartridge is identified by a CartridgeKey of (model_id, quant, rope_config, log_head). Because it’s pinned to a log_head, it’s reconstructible by replaying the log up to that point — satisfying Hard Rule #4 (every view is a function of the log).

The lifecycle

1. Compile. compile() opens key-sealed memories, hands the surviving plaintext to a KvBackend, and produces a cartridge blob.

2. Gate. CartridgeStore::activate only serves a cartridge that passes is_committable() — the same safety gate as everything else (Hard Rule #1). A cartridge that doesn’t hold accuracy is not activated.

3. Version, never mutate. New cartridges supersede old ones; they’re never edited in place (Hard Rule #2).

4. Erase by recompile. Crypto-shred a subject’s key, recompile, and the rebuilt blob can’t reconstruct the erased content — reconciling KV memory with the right to be forgotten on an append-only log.

Measured speedups

These are real numbers from the candle/Metal GPU path, reported with their conditions. See Benchmarks for the full table.

Configuration	Speedup	vs. baseline
GPU KV (general)	107×	text-context retrieval
GPU bf16, 1.5B model	1577×	CPU f32
q8 quantized	4×	f32 cartridge

Why bf16, not f16, for deep models

f16 has a narrow exponent range; deep models overflow it in the residual stream. mnesio’s GPU path uses bf16 for deep models and a mixed-precision residual (f32 accumulation) so accuracy holds. That’s why the headline 1.5B speedup is quoted as bf16-vs-cpu-f32 — it’s the configuration that actually preserves quality.

Selecting a model + precision at runtime

The server exposes the active KV configuration (model + precision) at GET /api/kv/metrics, with a dashboard panel for it. The backend is swappable behind the KvBackend trait — fake, GPT-2 generative, or the candle/Metal GPU path — without touching the cartridge substrate.

Next: the full numbers in Benchmarks.