Benchmarks

This page lists every number mnesio publishes, with the conditions under which it was measured. The harness that produces them lives in mnesio-bench. Anything that comes from a deterministic stand-in is labelled as such and is not a number we headline.

The eval harness is the product

mnesio treats evaluation as a first-class deliverable, not an afterthought. The goal isn’t a leaderboard screenshot — it’s a harness you can rerun to verify every claim on this site.

Retrieval quality

Metric	Value	Conditions
Recall (SQuAD)	98.1%	Hybrid HNSW + BM25 + RRF, fastembed 384-d, top-k retrieval over the SQuAD corpus
Recall@k at scale	100%	105k synthetic memories indexed; recall@k of the planted gold memory

Retrieval is mnesio’s strongest, most-reproducible result: the hybrid pipeline finds the right memory essentially every time. This is the foundation the QA numbers build on — when QA accuracy is lower, the bottleneck is the answerer model, not retrieval.

Scale & latency

Measured at 105,000 memories indexed:

Metric	Value
Append latency p50	~0.0017 ms
Query latency p99	4.11 ms
Recall@k	100%

The append path stays far under the <5ms target (Hard Rule #5) because embedding and evolution are async behind bounded queues — a write never blocks on an LLM call.

Agent task accuracy

A real LLM-driven agent loop over the live mnesio-mcp stdio server (examples/agent_loop_eval.py, Ollama-driven, 6 private-fact questions):

Memory	Score
Off	0 / 6 (0%)
On	5 / 6 (83%)

This isolates the mechanism: the same agent, same questions, memory toggled. Turning mnesio on takes a task it cannot do to one it mostly can. It is a small, reproducible demonstration — not a leaderboard ranking.

QA accuracy (LLM-judged)

QA accuracy needs a model in the loop for both answering and judging. With a local model (Ollama) over a SQuAD-100 slice:

Suite	Model	Accuracy
SQuAD-100	local (Ollama)	66%

On LOCOMO

LOCOMO is a hard long-conversation QA benchmark. We have a real loader and runner for the full splits, but a headline LOCOMO score is gated on a frontier-class answerer model (the local 3B/7B models are the ceiling, not mnesio’s retrieval — which scores 98.1% recall). We will publish a LOCOMO number only when it’s produced by a model we can stand behind, with the standard LLM judge. We do not publish a number we can’t reproduce.

KV cartridge speedups (GPU)

From the candle/Metal GPU path (see KV cartridges):

Configuration	Speedup	Baseline
GPU KV (general)	107×	text-context retrieval
GPU bf16, 1.5B model	1577×	CPU f32
q8 quantized	4×	f32 cartridge

bf16 (not f16) plus a mixed-precision f32-accumulation residual is what keeps deep-model accuracy intact at these speedups.

Cartridge accuracy parity

The Phase-12 “done when” requires the cartridge to answer at least as accurately as text-context retrieval at the speedup. The kveval harness measures this over a suite — compile the corpus into a cartridge once and answer from it, versus retrieve-top-k-and-answer per query:

Suite	Cartridge acc	Text-context acc	Parity Δ	Speedup
LOCOMO-mini	90%	80%	+0.10	~167×
LongMemEval-mini	60%	60%	+0.00	~180×

Cartridge accuracy ≥ retrieval (it attends over the whole corpus, not just the top-k), at ~167–180× lower per-query latency (compile once, replay — the baseline recompiles per query), with erasure-by-recompile holding throughout. These run the deterministic FakeKvBackend so they gate CI; a real generative backend through the same harness yields the publishable number (the GPU path’s token-identical self-consistency makes that parity exact). Reproduce: cargo run -p mnesio-bench -- kveval --suite locomo.

How to reproduce

# retrieval recall + QA accuracy
cargo run --release -p mnesio-bench -- qaeval --dataset squad --rows 100

# scale / latency at 105k
cargo run --release -p mnesio-bench -- scale --n 105000

# agent loop (needs a running Ollama)
python examples/agent_loop_eval.py

Numbers on this site are kept in sync with BENCHMARKS.md in the repo. If a figure here ever disagrees with a fresh run, the fresh run wins — file an issue.