Temporal Validity on Real Software Histories - Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixes

Sep 15, 2026

Retrieval-augmented generation lacks a representation of time, so when code facts change within the same development context it retrieves both old and new values, causing stale-fact errors. A deterministic supersession memory (subject, relation, object) was previously shown on synthetic single-value benchmarks to eliminate this failure. This paper validates the same mechanism end-to-end on real GitHub histories by extracting clean atomic state transitions from 707 issues and evaluating marker-free accuracy and latency against RAG and reranking.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Introduction

Contributions

Related Work

Memory for LLM agents and RAG

SWE-bench and code evolution

Temporal knowledge and the synthetic precedent

Method

The memory mechanism (recap)

arXiv : 2608 .20685v 1 [cs .SE] 2 1 Aug 2026 Temporal Validity on Real Software Histories Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixes A deterministic supersession memory, validated end-to-end on SWE-bench buggy→ fixed pairs Neeraj Yadav [MemStrata.dev](MemStrata.dev)—Called It Inc. (Enterprise) [memstrata@gmail. com](memstrata@gmail. com) Draft v1 Companion to “Temporal Validity in Retrieval Memory”(Paper 1), which established the result on synthetic evolving benchmarks. This paper validates it end-to-end on real GitHub histories. All numbers are from the locked, paired run on identical cached scenarios ([REPORT_PAPER2.md](REPORT_PAPER2.md), [REPORT_PAPER2_forced.md](REPORT_PAPER2_forced.md)); local, deterministic (temperature 0, fixed seeds, no network) . For double-blind submission, anonymize the author block and product identifiers. Abstract Retrieval-augmented generation (RAG) has no model of time: when a fact changes across a coding session—a function is renamed, an endpoint moves, a dependency is bumped—RAG retrieves both the old and new value with near-identical similarity and cannot tell which is current, so it serves the superseded value. Paper 1 showed, on synthetic single-value benchmarks, that a deterministic (subject, relation, object) supersession memory eliminates this failure. Here we validate it end-to-end on real software history. From 707 real GitHub issues (SWE-bench Lite + Verified) we extract 130 clean atomic state transitions—a fix that changes one identifiable value from a pre-fix to a post-fix form—and render each marker-free (the stale and current statements differ only in the value) . On this set, MemStrata reaches 0.91 answer accuracy versus RAG’s 0.57–0.59; and, the structural result, when forced to answer RAG serves the superseded value 36–38% of the time (an LLM reranker does not help) while MemStrata drives this to ≈0, at RAG retrieval latency (∼2.1 s vs ∼ 18 s for the reranker) . We are explicit about scope: only ∼ 18% of real fixes are clean atomic transitions; Paper 2 isolates the memory mechanism on that class, and extraction coverage of the remaining fixes is the orthogonal problem we defer to follow-on work. A real product bug surfaced and was fixed during the study (a case/punctuation-insensitive value comparison), with the moat property (deterministic-supersession accuracy on clean code mutations) preserved and verified. 1 Introduction A coding assistant that persists memory across a session accumulates facts about a codebase—the name of a handler, a config value, a pinned version, an endpoint path. The binding difficulty isnot recall but currency: these facts change, often within the same session, and an assistant that confidently reports last week’s function name is worse than useless. Retrieval-augmented generation [Lewis et al., 2020], the dominant memory mechanism, stores statements and retrieves by embedding similarity. It has no representation of time, so when a value changes it keeps both versions—“the login handler is authenticate_user” and “the login handler is login” —which sit close together in any embedding space. Retrieval surfaces both; the model cannot tell which is current; it abstains or serves the stale value. Paper 1 demonstrated, on synthetic single-value benchmarks (code mutation, config migration, dependency bumps, API evolution), that this is a structural failure of similarity-based memory and that a deterministic supersession rule eliminates it. The natural objection is that synthetic benchmarks may flatter the method. This paper answers it: we validate the same mechanism end-to-end on real GitHub buggy→fixed histories drawn from SWE-bench [Jimenez et al. , 2023] . Contributions. 1. A real-data longitudinal benchmark. From 707 real GitHub issues we extract 130 clean atomic state transitions (a verified pre-fix value → post-fix value), rendered marker-free so the only currency signal is order. The ex

FAQs

Why does retrieval-augmented generation fail when code facts change during a session?

RAG has no model of time, so embeddings for old and new values remain near-identical; it retrieves both and the model cannot determine which is current, leading to stale-fact errors.

How does MemStrata address temporal validity in code-assistant memory?

MemStrata uses a deterministic supersession rule over a bi-temporal ledger (subject, relation, object) so that updated facts supersede previous ones, removing the structural failure of similarity-based memory.

What evidence is used to validate MemStrata on real software histories?

The study extracts 130 clean atomic state transitions from 707 real GitHub issues (SWE-bench Lite + Verified) and evaluates answer accuracy and stale-fact error under paired regimes on identical cached scenarios.

Sep 15, 2026

Uploaded by Genevieve