Which benchmark should I use to compare memory systems?

For recall, LoCoMo and LongMemEval are what vendors report, and the numbers are comparable only when the harness, model and judge are the same, which they rarely are between vendors. For whether a system can update a fact and abstain, LongMemEval's update and abstention splits are the closest public test. For governance, access and deletion, GateMem, ForgetEval and MemSecBench exist as single preprints that nobody has reproduced. There is no benchmark that scores the five obligations together; that is open problem one.

Why does this site not rank products by benchmark score?

Because the scores are mostly self-reported, measured on different harnesses, and have been disputed between vendors; because a recall score says nothing about who may write to the memory, why a fact changed, or whether a deletion reached the summaries; and because the grades on this site come from makers' own documentation, which anyone can check. Benchmark results are recorded in the research library with their limits, never used as a grade.

Has The Continuity Layer run any of these benchmarks?

No. Every figure on this page is as reported by the benchmark's authors or the vendor, and the entry says so. This project recorded one bounded demonstration of its own in July 2026, which is in the Lab with a list of what it does not show.

The Continuity Layer · AI memory and continuity benchmarks

What the benchmarks measure, and what they do not

thecontinuitylayer.com/continuity-benchmarks

Fourteen public benchmarks, oldest first, each with what it scores and which of the five obligations its tests reach. Most measure recall. The ones that measure governance, forgetting and identity are a year old at most, and none has been reproduced by anyone but its authors.

Checked 7 November 2026. Sources linked. Our own framing is labeled as ours.

The table

A filled circle means the benchmark's tests reach that obligation at least in part; an empty one means they do not ask. The rows are what the authors score; the columns are this site's reading.

Benchmark	Date	Measures	Carry	Govern	Evolve	Prove	Forget
Evaluating Very Long-Term Conversational Memory of LLM Agents	February 2024	Multi-session conversational recall, event summarization, multimodal dialogue	●	○	○	○	○
RULER	April 2024	Effective context length against advertised length	●	○	○	○	○
LongMemEval	October 2024	Extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention	●	○	●	○	○
MemoryBench	October 2025	Continual learning from accumulated user feedback across domains and languages	○	○	●	○	○
Beyond a Million Tokens	October 2025	Recall across generated conversations up to 10M tokens, ten abilities	●	○	○	○	○
AMA-Bench	February 2026	Long-horizon memory over agent trajectories (states, actions, tool outputs) with causal dependencies	●	○	●	○	○
ATANT	April 2026	Continuity properties scored directly: persistence, update handling, disambiguation and others	●	○	●	○	○
Control-Plane Placement Shapes Forgetting	June 2026	The operations that change memory (supersede, release, purge), including intent-aware deletion	○	○	●	○	●
GateMem	June 2026	Shared memory across principals: utility, access boundaries, forgetting after deletion requests	○	●	○	○	●
MemSecBench	July 2026	Memory poisoning from persistence to consequence to selective repair	○	●	○	○	●
Identity Is More Than Recall	September 2026	Fidelity to a versioned identity contract: recall, composition, enactment, resistance, lineage, governed updates	●	●	○	○	○
Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents	October 2026	Unlearning across weights, memory and tools together, with provenance-tracked trajectories	○	○	○	●	●

Count the columns. Carry is reached by almost everything; evolve by the update-and-abstention splits and the temporal benchmarks; govern and forget by five preprints from 2026; prove by two. There is no row that fills all five, which is open problem one.

Reading vendor scores

Memory vendors publish LoCoMo and LongMemEval results, and the numbers travel: 92.5 and 94.4 for one vendor in 2026, 75 to 84 for another on the same benchmark depending on who computed it. The spread is the finding. The harness, the backbone model, the judge model and the question subset all move the score, and a 2025 dispute between two vendors over one LoCoMo figure ended with a correction and no agreement. Read any vendor figure as that vendor's figure, on that vendor's harness, on that date; the research library records them that way, with the limits the authors themselves state.

The deeper issue is what recall leaves out. A system can score at the top of LongMemEval and have no rule for who may write to it, no reason attached to any change, no provenance on any fact, and a delete that stops at the store. The grades on this site are read from makers' own pages for exactly that reason: a documented export, scope or delete is a claim anyone can check, and it is labeled for the obligation it reaches and the rung of the erasure ladder it stops at.

The entries

Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo)

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang · UNC Chapel Hill; Snap · 27 February 2024 · arXiv:2402.17753; ACL 2024

Long conversations across up to 35 sessions, with question answering, event summarization and multimodal dialogue tasks.

Contribution: The benchmark most memory vendors report; measures multi-session recall.

Limits: Recall only. Scores vary with the harness and judge; vendor figures on it have been disputed between vendors.

Reaches: Carry · Library entry

RULER: What's the Real Context Size of Your Long-Context Language Models?

Cheng-Ping Hsieh and others · NVIDIA · 9 April 2024 · arXiv:2404.06654; COLM 2024

Synthetic tasks that measure the context length a model actually uses, against the length it advertises.

Contribution: The standard evidence that advertised and effective context differ, which is why a bigger window is not continuity.

Limits: Measures context, not memory; says nothing about anything carried past the call.

Reaches: Carry · Library entry

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu · 14 October 2024 · arXiv:2410.10813; ICLR 2025

500 questions over long chat histories testing extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention.

Contribution: The first widely used benchmark to score knowledge updates and abstention, the two abilities closest to the evolve obligation.

Limits: Does not score who may change a fact, why it changed, or whether it can be deleted.

Reaches: Carry, Evolve · Library entry

MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems

Qingyao Ai, Yichen Tang, Changyue Wang, Jianming Long and two others · Tsinghua University (THUIR) · 20 October 2025 · arXiv:2510.17281; ICML 2026

Simulated, accumulated user feedback across domains and languages, scoring whether a system learns from it.

Contribution: Treats memory as continual learning from feedback rather than recall of a transcript.

Limits: Feedback-driven update, not governance of what was updated.

Reaches: Evolve · Library entry

Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs (BEAM)

Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla and two others · 31 October 2025 · arXiv:2510.27246; ICLR 2026

100 generated conversations up to 10 million tokens, 2,000 validated questions, ten memory and reasoning abilities.

Contribution: Separates long context from memory at a scale no window reaches.

Limits: Synthetic conversations; real dialogue structure may differ.

Reaches: Carry · Library entry

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

Yujie Zhao, Boqin Yuan, Junbo Huang and others · UC San Diego; Recursive · February 2026 · arXiv:2602.22769 (v4, 27 May 2026)

Memory over agent trajectories (states, actions, observations, tool outputs) rather than chat turns; 2,496 expert-annotated questions across six domains plus a scalable synthetic subset.

Contribution: Finds that memory systems often underperform a long-context baseline because compression and similarity retrieval discard causal and state information; GPT-5.2 reaches 72.26%.

Limits: Single-episode memory; cross-task and lifelong learning are future work.

Reaches: Carry, Evolve · Library entry

ATANT: An Evaluation Framework for AI Continuity

Kenotic Labs · 9 April 2026 · arXiv:2604.06710; v1.1 positioning paper arXiv:2604.10981

A framework and corpus for evaluating continuity properties (persistence, update handling, disambiguation and others) directly, rather than recall as a proxy.

Contribution: The first published attempt to score continuity as its own property set; the v1.1 paper positions it against memory, long-context and agentic-memory benchmarks.

Limits: Author-reported results on one narrative corpus; this project has not reproduced them. Kenotic Labs also publishes a thesis paper with the same title as this site; the works are unaffiliated.

Disclosure: Published by Kenotic Labs, whose commercial work this site also profiles.

Reaches: Carry, Evolve · Library entry

Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations

Dongxu Yang · 14 June 2026 · arXiv:2606.15903

Thirteen agent memory configurations on a 385-case adversarial set, varying where the LLM sits relative to recall and to the operations that modify stored facts.

Contribution: Deterministic methods fail canonicalization (5% on obfuscated identifiers, 0% cross-lingual); an LLM at write time fixes that but cannot do intent-aware deletion; a mutation-time LLM hook brings intent-aware deletion to 78–85% and overall accuracy to 91.7–93.2% at about $0.17 a run, without touching the recall path.

Limits: Single-author preprint with author-reported results; not reproduced by this project.

Reaches: Forget, Evolve · Library entry

GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents

Zhe Ren, Yibo Yang, Yimeng Chen, Zijun Zhao, Benshuo Fu, Zhihao Shu and others · 17 June 2026 · arXiv:2606.18829

Shared assistants in hospitals, offices, schools and households, where many people write to and query one memory pool: scores long-running requests that change state, respect for access boundaries, and forgetting after explicit deletion requests, together.

Contribution: No tested method does well on all three; long-context prompting governs best at high token cost, while retrieval and external-memory methods are cheaper and still leak unauthorized or deleted information.

Limits: Results not reproduced by this project; evaluating shared-memory behavior is not a certification of a governance system.

Reaches: Govern, Forget · Library entry

MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair

Xuanze Chen, Xukang Xie, Wentao Fu, Jiajun Zhou, Shanqing Yu, Qi Xuan · Zhejiang University of Technology · July 2026 · arXiv:2607.27080

310 cases that follow a malicious instruction from its write into long-term memory, through whether it later drives a real action, to whether it can be selectively repaired, across two harnesses, four memory backends and three models.

Contribution: Malicious content persisted in 84.2% of cases and the full write-to-execute chain succeeded in 50.3%; selective repair succeeded in 56.1%, with a 41.3-point spread between memory systems.

Limits: Each configuration-case pair run once; an LLM judge at about 91% agreement with human labels; descriptive, not definitive.

Reaches: Govern, Forget · Library entry

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

Zhenyu Zhao, Roy Zhao · University of Washington · 12 September 2026 · arXiv:2609.13637

PAI-Bench scores fidelity to a versioned identity contract across recall, composition, enactment, resistance, persistence, lineage and role-conditioned updates, with the scoring oracle outside the tested agent.

Contribution: Shows that agents recall identity facts (48 of 48 atomic answers) and fail to compose them into a self-portrait (1 of 48), and that evaluator choice moved the headline score by 12.5 points.

Limits: Single samples per condition, synthetic profiles, a judge chosen after inspecting results, no human validation and no runtime migration test; by the authors' account it shows protocol sensitivity, not model rankings.

Reaches: Carry, Govern · Library entry

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

Junkai Lin, Ge Xu, Zheng Xu, Junkai Chen, Xiangyuan Lan · Institute of Automation, Chinese Academy of Sciences; Pengcheng Laboratory · 13 October 2026 · arXiv:2610.17659

AUBench extends the TOFU unlearning setup with Mem0 persistent memory, an offline web snapshot and provenance-tracked trajectories, so forgetting is tested across weights, memory and tools together.

Contribution: Parametric forgetting did not predict agent behavior: adding memory raised one model's forget-set score from 16.8 to 30.1 after unlearning.

Limits: Two 8B models, three methods, one forget split, controlled offline sources; token-level F1 cannot show information is erased.

Reaches: Forget, Prove · Library entry

Questions

Which benchmark should I use to compare memory systems?: For recall, LoCoMo and LongMemEval are what vendors report, and the numbers are comparable only when the harness, model and judge are the same, which they rarely are between vendors. For whether a system can update a fact and abstain, LongMemEval's update and abstention splits are the closest public test. For governance, access and deletion, GateMem, ForgetEval and MemSecBench exist as single preprints that nobody has reproduced. There is no benchmark that scores the five obligations together; that is open problem one.
Why does this site not rank products by benchmark score?: Because the scores are mostly self-reported, measured on different harnesses, and have been disputed between vendors; because a recall score says nothing about who may write to the memory, why a fact changed, or whether a deletion reached the summaries; and because the grades on this site come from makers' own documentation, which anyone can check. Benchmark results are recorded in the research library with their limits, never used as a grade.
Has The Continuity Layer run any of these benchmarks?: No. Every figure on this page is as reported by the benchmark's authors or the vendor, and the entry says so. This project recorded one bounded demonstration of its own in July 2026, which is in the Lab with a list of what it does not show.

The Continuity Layer

The directory shows what exists. This section explains why it matters and what good architecture looks like. The two meet on the graded shelves, where every memory system and assistant is read through the same five obligations.