The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

Jul 16, 2026

Upload by Quinn

LLM agent benchmarks assess task completion, reliability, and inference cost, but typically omit the persistent disk residue created by agent runs, including logs, context snapshots, checkpoints, and debug traces. These retained bytes affect deployability on desktops, compliance with data-retention rules, and behavior at fleet scale. The paper introduces AgentFootprint, a serialization-aware, cross-framework storage-footprint benchmark covering total retention, channel composition, duplication, growth exponent, compressibility, and reconstructability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Abstract

Introduction

The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation Chenglin Yu 1 , Hongquan Gui 1 , Ying Yu2 , Hongxia Yang3 , Ming Li 1,4∗ 1Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University 2 College of Economics and Management, Zhejiang Normal University 3Department of Computing, The Hong Kong Polytechnic University 4Research Institute for Generative AI, The Hong Kong Polytechnic University [chenglin.yu@polyu.edu.hk](chenglin.yu@polyu.edu.hk), [ming.li@polyu.edu.hk](ming.li@polyu.edu.hk) arXiv :2607 . 1 1 149v 1 [ cs .AI] 13 Jul 2026 Abstract LLM agent benchmarks measure task completion, reliability, and inference cost, but not the persistent data an agent run leaves on disk—logs, context snapshots, checkpoints, debug traces. These bytes are absent from the leaderboards we survey, yet bear on whether an agent system can be deployed on desktops, under data-retention compliance regimes, or at fleet scale. We introduce AgentFootprint, to our knowledge the first systematic cross-framework benchmark of post-run agent storage footprint. Its serialization-aware metric suite—total retention, channel composition, duplication, growth exponent, compressibility, and a conversation-history reconstructability score—addresses a measurement trap: naïve byte-level measurement understates duplication by an order of magnitude, because database paging and JSON escaping obscure repeated content. The footprint has two determinants—the logical volume the agent’s behavior generates, and the amplification the persistence layer adds when storing it—and a fixed-trace control isolates the second: the same trajectory replayed through each persisting framework’s adapter yields retained sizes differing by 6.7× . Under identical models, tools, and tasks, framework configurations at identical 100% accuracy differ by 15.7× in retained bytes; the defaults bundle different recovery and audit capabilities, so the suite reports storage jointly with accuracy and reconstructability. Three full-history configurations grow superlinearly on a repeated-observation stress task, and on a deliberately minimal write task framework residue exceeds the delivered output files by orders of magnitude. Exported trajectories from 108 instance-normalized SWE-bench Verified submissions span three orders of magnitude in perinstance volume with no detectable correlation with resolve rate. A content-addressed store reduces retention 4.8–32.7× while preserving all properties checked by our restoration tests, including every reconstructability score: exact history reconstruction does not require megabytes, and these storage metrics can be reported alongside inference cost. 1 Introduction LLM agents—systems that pursue multi-step tasks by interleaving model calls with tool use—are moving from demos into production, and the community evaluates them along an expanding set of axes: task success on GAIA (Mialon et al. 2024) and SWE-bench (Jimenez et al. 2024), reliability via pass^k on τ-bench (Yao et al. 2025), and, following Kapoor ∗Corresponding author. et al. (2025), inference cost. A recent survey of agent evaluation catalogs more than a dozen such metrics (Mohammadi et al. 2025)—none of them includes the storage footprint: the bytes an agent execution leaves behind on disk. This paper measures that axis. Unlike inference cost, which is consumed when execution ends, retention is a stock that persists and accumulates across runs. This persistence is simultaneously useful and costly: it is the substrate for replay debugging, recovery, and compliance audits, and it enlarges long-term storage, governance, and deletion obligations. The magnitudes involved are easy to underestimate. In a production deployment that we measured directly—a supply-chain quoting system (Supp. App. P)—a single task retained 131 MB, including 79 MB of framework residue, compared with 2.6 MB of delivered artifacts; per-task retentio

FAQs

What problem does the paper address in existing LLM agent benchmarks?

Most benchmarks focus on task success, reliability, and inference cost, but do not measure the persistent storage footprint—what bytes an agent leaves on disk after a run.

What is AgentFootprint and what does it measure?

AgentFootprint is a cross-framework benchmark and measurement harness that quantifies post-run storage footprint using serialization-aware metrics such as total retention, duplication, growth exponent, compressibility, and reconstructability.

Why can naïve byte-level measurement be misleading for storage footprint?

Because serialization details like database paging and JSON escaping can mask duplication, causing byte-level approaches to understate repeated content by large factors.

Jul 16, 2026

Upload by Quinn