SWE-MERA：用于动态评估大语言模型的智能体软件工程基准

Jul 16, 2026

Upload by Himbo

SWE-MERA introduces a dynamic, continuously updated benchmark to address critical weaknesses in existing software engineering evaluations, especially the widely used SWEbench. The approach targets data contamination and inadequate test coverage, where prior work shows substantial success via solution leakage and fragile generalization. A reliable seven-stage automated pipeline collects real-world GitHub issues, applies rigorous quality validation, and limits contamination risk. Current resources include about 10,000 candidate tasks and 728 available samples, with Aider-agent evaluation showing strong discrimination across recent LLMs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Abstract

Introduction

Key limitations of SWEbench

How SWE-MERA addresses dynamic evaluation

Related Work

Repository-level benchmarks

Dynamic evaluation frameworks

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks Pavel Adamenko1 , Mikhail Ivanov2 , Aidar Valeev1 , Rodion Levichev1 , Pavel Zadorozhny1 , Ivan Lopatin1 , Dmitrii Babaev1 , Alena Fenogenova1 , Valentin Malykh3,2,4 1 GigaCode, 2ITMO University, 3MWS AI, 4IITU university, [Correspondence:](Correspondence: mera@a-ai.ru)[ mera@a-ai.ru](Correspondence: mera@a-ai.ru) arXiv :2507 . 1 1059v 3 [ cs . SE] 11 Jul 2026 Abstract The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWEbench dataset. Recent studies have uncovered severe data contamination issues, e.g., SWEbench (Jimenez et al., 2023) reports 32.67% of successful patches involve direct solution leakage and 31.08% pass due to inadequate test cases. We introduce SWE-MERA, a dynamic, continuously updated benchmark designed to address these fundamental challenges through an automated collection of real-world GitHub issues and rigorous quality validation. Our approach implements a reliable pipeline that ensures quality while minimizing contamination risks, resulting in approximately 10,000 potential tasks with 728 samples currently available. Evaluation using the Aider coding agent demonstrates strong discriminative power in state-of-the-art models. We report performance across a dozen recent LLMs evaluated on tasks collected between September 2024 and June 2025. 1 Introduction The complexity of real-world software development processes goes beyond merely completing code. It encompasses coding agents and a range of text-to-code tasks. E.g., SWE-bench (Jimenez et al., 2023) was created from a dataset comprising 2,294 GitHub issues and their corresponding pull requests (PRs) . Each task in SWE-bench represents an authentic, real-world problem structured around: 1) The initial commit (code before changes), 2) The fixing commit (solution to the problem), 3) The issue description (what needed to be fixed) . A critical limitation of this benchmark is its static nature—the tasks were collected only once and never updated. This leads to two major issues. Data leakage: As models are repeatedly tested on the same fixed dataset, they may inadvertently memorize solutions or overfit to outdated examples. Benchmark saturation: Over time, the benchmark loses its effectiveness as state-of-the-art models achieve near-perfect scores, making it harder to distinguish meaningful progress. SWE-MERA addresses these shortcomings (typical for many code benchmarks) by introducing dynamic updates to the test cases. Regularly refreshing the dataset with new, unseen issues ensures: 1) real-world relevance—tasks reflect the latest challenges in software development 2) fair evaluation—models are tested on fresh problems, minimizing the risk of data leakage 3) continuous improvement—the benchmark evolves in tandem with advancements in AI and software engineering practices. The contributions of the paper are as follows: 1. The seven-stage pipeline effectively ensures quality and minimizes contamination risks, able to collect approximately 10,000 potential tasks, with 728 samples currently available. 2. An automated scoring system based on Aider coding agent 1 and a dynamic user leaderboard 23. 2 Related Work SWE-bench introduced a semi-automatic pipeline for mining software engineering tasks from popular open-source Python repositories, resulting ina benchmark of 2,294 issues and corresponding pull requests. Although this enabled a large-scale evaluation, the dataset suffered from quality issues, including poorly specified tasks and weak test coverage, which compromised the reliability of model assessment. To improve data quality, 1 [https://aider.chat](https://aider.chat) 2 SWE-MERA leaderboard 3The video screencast of the user’s journey can be accessed through the link provided SWE-bench Verified4 re

FAQs

What problem does SWE-MERA target in existing software engineering benchmarks?

It targets limitations in static benchmarks, especially data contamination from solution leakage and insufficient test coverage that weakens reliable evaluation of LLMs.

How does SWE-MERA create tasks for evaluation?

It builds tasks from real-world GitHub issues by reverting repositories to earlier states, identifying future tests that are introduced later but fail in the current version, and validating task quality through a multi-stage pipeline.

How is model performance evaluated in SWE-MERA?

Performance is assessed using the Aider coding agent, and results are reported with a dynamic user leaderboard to support discriminative evaluation across recent LLMs.

Jul 16, 2026

Upload by Himbo