SWE-Milestone：评估AI代理在持续软件演进中的表现

Jul 16, 2026

Upload by Finn

Real-world software must adapt to continuously emerging and open-ended requirements, making evolution an incremental, long-horizon process. Existing agent benchmarks test isolated, one-off coding problems and miss temporal dependencies and technical-debt accumulation. DeepCommit introduces an agentic pipeline reconstructing verifiable milestone-level milestone DAGs from noisy commit logs, enabling SWE-Milestone to assess agents on continuous streams. Experiments with 12 frontier models across 4 frameworks show a sharp drop from 80%+ on isolated tasks to at most 38% in continuous settings, highlighting weak long-term maintenance and error propagation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages44

Reading Time1 h 51 min

Introduction

Problem: isolated benchmarks ignore temporal evolution

DeepCommit and milestone DAG reconstruction

SWE-Milestone benchmark design

Experimental results and vulnerability finding

arXiv :2603 . 13428v 3 [ cs . SE] 10 Jul 2026 SWE-Milestone: Evaluating AI Agentson Continuous Software Evolution Gangda Deng1,* , Zhaoling Chen2,* , Zhongming Yu3 , Haoyang Fan1 , Yuhong Liu1 , Yuxin Yang1 , Dhruv Parikh1 , Rajgopal Kannan4 , Le Cong5 , Mengdi Wang6 , Qian Zhang2 , Viktor Prasanna1 , Xiangru Tang7,†, Xingyao Wang8 1USC 2UCR 3UCSD 4Army Research Office 5 Stanford 6Princeton 7 Haven 8 OpenHands * Equal Contribution †Corresponding Author DeepCommit Pipeline: [github.com/DeepCommit-ai/DeepCommit](github.com/DeepCommit-ai/DeepCommit) Benchmark: [github.com/DeepCommit-ai/SWE-Milestone](github.com/DeepCommit-ai/SWE-Milestone) Data: [huggingface.co/datasets/DeepCommit-ai/SWE-Milestone-data](huggingface.co/datasets/DeepCommit-ai/SWE-Milestone-data) Leaderboard: [swe-milestone.com](swe-milestone.com) Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents’ profound struggle with long-term maintenance and error propagation. Figure 1 | Milestone-level task granularity optimally balances functional coherence and evolutionary awareness for benchmarking continuous software evolution. 1. Introduction Software development in the real world is driven by dynamic, open-ended requirements. New requirements continuously emerge, with some building on earlier ones while others can be pursued in parallel. As a result, software evolves through an ongoing, incremental process rather than a one-time effort. Frontier LLM agents (e.g., Claude Code (Anthropic, 2025), Codex (OpenAI, 2025)) are increasingly entrusted to drive this evolution as long-running systems (Nous Research, 2026; OpenClaw, 2026), autonomously developing and refining software within complex environments rather than producing one-off edits. Over time, such an agent’s accumulated efforts naturally trace Contact: Gangda Deng ([gangdade@usc.edu](gangdade@usc.edu)), Zhaoling Chen ([zchen526@ucr.edu](zchen526@ucr.edu)), Xiangru Tang ([xiangru.tang@yale.edu](xiangru.tang@yale.edu)) Table 1 | Representative software engineering benchmarks for LLMs. Unlike other categories of benchmarks that evaluate agents on isolated snapshots or against ground-truth states at each step, Repository Evolution requires agents to continuously build upon their own accumulated development history, exposing them to error propagation across tasks. SWE-Milestone adopts the Milestonelevel granularity, a functionally coherent group of commits that collectively advance a development objective, avoiding the noise of individual commits and the excessive scope of full releases. Dev History denotes the agent’s own development trace accumulated from preceding tasks. | Category | Benchmark | Language | Task Properties | | | Cross-task<br>Dependency | Task<br>Collection | | --- | --- | --- | --- | --- | --- | --- | --- | | | | | Granularity | Avg LoC | Additional Context | | | | Function Completion | HumanEval (Chen et al., 2021) | Python | Function-level | 6.8 | –

FAQs

Why do existing AI-agent benchmarks fail to reflect real software evolution?

They evaluate agents on isolated, one-off coding tasks, neglecting temporal dependencies and the technical-debt accumulation that occurs during continuous development.

What does DeepCommit contribute to the SWE-Milestone benchmark?

DeepCommit reconstructs verifiable milestone DAGs from noisy commit logs, defining milestones as functionally cohesive development goals.

What is the key evaluation finding across models and frameworks?

Overall performance drops significantly in continuous settings, falling from above 80% on isolated tasks to at most 38%, showing major difficulties in long-term maintenance and error propagation.

Jul 16, 2026

Upload by Finn