OCRR — the benchmark for online correction recovery.
Static benchmarks tell you whether a model started smart. They don't tell you whether it stays smart when the world drifts. OCRR — Online Correction Recovery Rate — measures how well a system recovers from distribution shift via online correction, at varying memory budgets.
At an equal 1,000-entry memory budget, a bounded reservoir substrate beats A-GEM by 32.6 percentage points on novel-class accuracy (80.7% vs 48.1%) — sacrificing only 4 pp on the original distribution. Retrieval-based online learning is dramatically more sample-efficient than gradient-based at fixed memory.
A protocol you can argue with.
OCRR is permissive about how systems remember — gradient, reservoir, kNN, online linear, river — but strict about what it measures. Each system reports its on-disk footprint alongside its recovery curve, so the storage-vs-recovery Pareto becomes the comparison surface, not a footnote.
12-system sweep
Bounded reservoir substrate, A-GEM, kNN-LM, online linear, FIFO baselines, and seven other treatments — all evaluated head-to-head under the same shift schedule.
Four memory budgets
Each system is evaluated at 100, 500, 1,000, and 5,000 entries. The Pareto closes at 5,000: bounded substrate is within 2 pp of unbounded — quantitatively rebutting the "you stored everything" attack.
Banking77 / oracle split
Three random seeds per cell. Mean ± std reported on every number. The shift schedule and oracle are part of the published artifact, not the proprietary part.
Four falsifiable claims
Pre-registered before the sweep ran. All four passed: bounded reservoir matches A-GEM at small budget; closes the gap by 5 pp at 5,000; FIFO loses >50 pp to forgetting; the substrate's advantage is not a storage artifact.
What's next.
OCRR v1 is the public preprint and the citable reference for the +32.6 pp result. v2 (in progress) extends the protocol to multi-stage cross-modal chains. v3 — co-named AMTB, the Agent Memory Task Bench — is pre-registered for NeurIPS 2026 Datasets & Benchmarks and unifies recall, retention, auditability, cross-modal transfer, scale, and adversarial revision into a single six-axis evaluation.
Adrian Grassi. OCRR: A Benchmark for Online Correction Recovery under Distribution Shift. arXiv preprint arXiv:2605.03153, May 2026. https://arxiv.org/abs/2605.03153
Reviewing the paper, or running the protocol?
If you're reviewing OCRR for a venue, replicating the sweep, or proposing a new system to evaluate, write to the address below — we'll respond with anything we can share without disclosing in-progress methodology.