Benchmark validity
EmergingWhy it matters — A score is useful only when the evaluated task matches the intended use.
A broader task suite is not proof that an earlier benchmark is invalid.
Not everything in this field is settled. These comparisons connect architectural and evaluation questions to specific sources. They distinguish a difference in scope from a documented disagreement, and do not declare a winning system.
How to read this page — reviewed 6 October 2026. On that date the July version’s numerical source grades and unverified percentages were removed (see the changelog). Each entry links its sources and explains its limits. These are source-level comparisons, not independent replications.
Sourced comparisons of architectures or evaluation scopes. Different approaches are not necessarily opposing claims.
Why it matters — A score is useful only when the evaluated task matches the intended use.
A broader task suite is not proof that an earlier benchmark is invalid.
Why it matters — Compression affects storage cost, detail, later reconstruction and auditability.
These design emphases can coexist. Neither source establishes that all summaries destroy useful episodes or that extraction is sufficient for every task.
Why it matters — Choosing what to retrieve affects which causal and temporal relationships remain available.
The sources are not a controlled head-to-head comparison and do not establish a universal winning retrieval method.
Why it matters — A graph introduces relationships and maintenance costs; useful representation depends on the workload.
Comparative cost and quality require matched experiments.
Why it matters — Where memory decisions execute affects permissions, observability and failure recovery.
This is a comparison of architectures, not evidence that all control must be inside or outside the model. CALMem also permits agent-written facts.
Why it matters — Retention, deletion requests and retrieval quality are different requirements.
Utility-driven decay is not equivalent to authorized deletion. This map does not infer a general privacy guarantee from either evaluation.
Open questions and scope comparisons. These are not established two-sided disputes.
Why it matters — What information must survive a one-off event for an agent to use it later?
These sources address different scopes. This map has not established that RULER’s authors oppose episodic memory.
Why it matters — A design must balance context cost, useful information and access boundaries.
Neither establishes that retrieval is obsolete or always superior. Conclusions depend on workload and protocol.
Why it matters — Shared state must still respect who may read, change and delete it.
The first statement is this map’s design question, not an attributed anti-sharing position.
Why it matters — Returning a current answer and invalidating obsolete state are separate tests.
These are complementary evaluation scopes, not proof that LongMemEval’s authors consider retrieval alone sufficient.
Entries are graduated onto this page from an internal review of the field, and this page renders only that published record — so what you read here is deploy-time stable and citable. How entries are chosen, categorized, and re-checked is set out in How the map works. Source quality follows the project’s editorial policy.