Cost and Accuracy of Long-Term Memory in Distributed Multi-Agent Systems Based on Large Language Models

Jul 16, 2026

Upload by Anda

Long-term memory (LTM) is central to large language model (LLM)-based agents in the internet of agents, yet existing evaluations often emphasize token usage and latency while overlooking system-level cost and deployment constraints in distributed multi-agent systems. The work proposes an independent, reproducible testbed measuring accuracy, latency, CPU time, peak RAM, disk I/O, and network usage in a simulated cloud-edge environment. It compares mem0, Graphiti, and cognee against RAG and full-context baselines on the LoCoMo benchmark. Results show a clear accuracy–cost clustering and identify retrieval incompleteness and compression precision as key drivers under unconstrained and constrained networks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Proposed Evaluation Testbed and Metrics

Compared LTM Frameworks and Baselines

Results and Analysis under Network Constraints

Open-Source Resources

© IEEE 2026. Manuscript accepted at IEEE COMPSAC 2026. Not for redistribution. Published version: [https://doi.org/10.1109/XXXXXX](https://doi.org/10.1109/XXXXXX) Cost and Accuracy of Long-Term Memory in Distributed Multi-Agent Systems Based on Large Language Models Benedict Wolff School of Electrical Engineering and Computer Science KTH Royal Institute of Technology Stockholm, Sweden [bjpwolff@kth.se](bjpwolff@kth.se)[ ](bjpwolff@kth.se)[bjpw@live.de](bjpw@live.de) Jacopo Bennati School of Electrical Engineering and Computer Science KTH Royal Institute of Technology Stockholm, Sweden [jbennati@kth.se](jbennati@kth.se)[ ](jbennati@kth.se)[jacobbista.bennati@gmail.com](jacobbista.bennati@gmail.com) arXiv :2601 .07978v4 [ cs .IR] 13 Jul 2026 Abstract—Long-term memory (LTM) is fundamental to large language model (LLM)-based agents in the emerging internet of agents (IoA), where distributed multi-agent systems (DMASs) span cloud and edge networks. Existing evaluations are typically published by framework providers and focus on token usage and latency, rarely accounting for system-level cost or deployment in DMAS. These gaps are addressed with an independent reproducible testbed that evaluates accuracy, latency, CPU time, peak RAM, disk I/O and network usage in a simulated cloud-edge environment. Three venture capital-funded frameworks spanning vector, graph, and hybrid architectures, namely mem0, Graphiti, and cognee, are compared alongside retrieval-augmented generation (RAG) and full-context baselines on the LoCoMo benchmark under unconstrained and constrained network scenarios. Two clusters emerge: mem0, RAG, and full-context reach 77% to 81% accuracy, while Graphiti and cognee reach only 55% to 56%, a gap driven by retrieval incompleteness rather than reasoning failure. The RAG baseline matches the upper cluster at 8.4 times lower total cost of ownership (TCO) than mem0, and both are the only non-dominated backends on the Pareto frontier. Latency and bandwidth constraints as well as jitter leave retrieval quality unchanged for every backend, while vector-based LTM incurs a modest latency penalty of 4% to 5% under edge-cloud constraints. Compression precision rather than context volume determines LTM accuracy, as full-context forwarding underperforms mem0 despite supplying the entire conversation for each question. Index Terms—long-term memory, multi-agent system, internet of agents, retrieval-augmented generation, knowledge graph Open-source research: the testbed, results, and notebooks for analysis and evaluation are available at: [https://github.com/](https://github.com/)[ ](https://github.com/)[wolffbe/dmas-memory](wolffbe/dmas-memory) I. INTRODUCTION Large language model (LLM)-based agents are increasingly capable of decomposing goals into executable steps using incontext learning, integrating external tools and data sources as well as persisting thoughts using long-term memory (LTM) [1] . LTM refers to the ability of LLM-based agents to store, organize, and retrieve information across interactions, enabling persistent context, behavioral adaptation, and personalization beyond single-turn reasoning [2] . Retrieval-augmented generation (RAG) is foundational to LTM by embedding overlapping chunks of data into high-dimensional vector representations [3] . However, this static approach struggles with dynamic knowledge integration, introduces noise, and overlooks structural relationships. LTM extends this paradigm into dynamic, agent-specific memory that accumulates overtime, organized through textual, vector-based, graph-based and hybrid architectures, and supported by operations for data and knowledge indexing, retrieval, updating, and consolidation. To evaluate these capabilities, LTM benchmarks such as LoCoMo and LongMemEval have been proposed, focusing on tasks including multi-session reasoning, information retrieval, temporal consistency, and knowledge updates [4], [5] . Despite advances in L

FAQs

What problem does the paper address about LTM evaluation in distributed multi-agent systems?

Existing evaluations often focus on token and latency metrics published by framework providers, and rarely measure system-level cost or deployment behavior in cloud-edge distributed settings.

What does the proposed testbed measure besides accuracy?

It evaluates accuracy, latency, CPU time, peak RAM, disk I/O, and network usage using an independent and reproducible setup.

What explains the performance gap between the two accuracy clusters?

The paper attributes the gap mainly to retrieval incompleteness rather than reasoning failure.

Jul 16, 2026

Upload by Anda