BACKENDFORGE：使用后端服务的端到端代理式代码生成基准

Jul 16, 2026

Upload by Eden

Large language models (LLMs) increasingly power agentic coding workflows that inspect files, execute commands, run tests, observe failures, and iteratively revise code. The key evaluation challenge is whether an agentic LLM can produce a deployable end-to-end software artifact that is behaviorally correct after execution. Backend services offer a controlled yet realistic substrate via APIs validated deterministically against OpenAPI contracts using black-box HTTP tests, enabling faithful measurement of contract realization.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages15

Reading Time38 min

Abstract

1 Introduction

Evaluation goal: contract realization

Deterministic vs realistic benchmarking

BACKENDFORGE pipeline and scoring

BACKENDFORGE: Benchmarking Agentic End-to-End Code Generation with Backend Services Yuzhe Guo1,2 * Mengzhou Wu1,2 * Yuan Cao1,2 Jialei Wei3 Dezhi Ran1,2† Wei Yang3 Tao Xie1,2,3,4† 1 Key Lab of HCST (PKU), MOE; SCS, Peking University 2Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC) 3Fudan University Institute of Systems for Advanced Computing 4 Shanghai Institute of Systems for Open Computing arXiv :2607 . 1 1042v 1 [ cs . SE] 13 Jul 2026 Abstract Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution? Backend services provide a controlled but realistic substrate for this evaluation. Their APIs expose application-level executable semantics, and deployed behavior can be checked deterministically against an OpenAPI contract through black-box [HTTP](HTTP) interactions. We introduce BACKENDFORGE, a benchmark of 56 contract-defined backend generation tasks rewritten from real open-source applications. Given a visible specification and an OpenAPI contract, an LLM must generate a Dockerized service that is built, deployed, and evaluated only through [HTTP tests. To strengthen evalu](HTTP tests. To strengthen evalu)ation without introducing hidden requirements, BACKENDFORGE uses a test agent and a code agent to co-evolve the test oracle and reference service, where the test agent proposes specification-grounded backend tests and the code agent repairs the reference implementation. Although the best-performing model, GPT-5.5, succeeds on 55.4% of tasks under the base oracle, it succeeds on only 28.6% under the final oracle. This gap suggests that current LLMs can implement many local API behaviors, but still struggle to produce complete backend services. 1 Introduction Large language models (LLMs) are rapidly changing the unit of software generation. Early code generation systems primarily targeted localized assistance, such as completing a function, solving *Equal contribution. †Corresponding authors. a programming problem, or producing a small code snippet (Chen et al., 2021 ; Austin et al., 2021 ; Hendrycks et al., 2021) . Subsequent systems and benchmarks moved toward repositorylevel development, where models modify existing codebases to resolve bugs or implement requested changes (Jimenez et al., 2024) . More recently, LLMs are used in agentic coding settings where they can inspect project files, execute commands, run tests, observe failures, and iteratively revise code within a workspace (Yang et al., 2024 ; Anthropic, 2026a ; OpenCode, 2026) . As LLM coding workflows become more interactive and autonomous, user expectations are shifting from receiving isolated code fragments toward obtaining usable software artifacts (Yao et al., 2023 ; Wu et al., 2024 ; Lu et al., 2021 ; Hong et al., 2024) . For such artifacts, usefulness depends not only on whether code compiles or local tests pass, but on whether the generated system can be built, deployed, and interacted with through its intended interfaces. Evaluating end-to-end software generation requires balancing realism against deterministic evaluation (Zhou et al., 2024 ; Xie et al., 2024) . A fully open-ended benchmark could ask models to infer requirements from underspecified user requests, design an interface, implement the system, and deploy it (Qian et al., 2024 ; Chen et al., 2024) . Such a setting would be realistic, but difficult to evaluate deterministically because failures could arise from requirement interpretation, interface design, implementation, or deployment (Ezzini et al., 2021 ; Kapoor et al., 2024) . In BACKENDFORGE, we instead isolate the contract-realization problem: given an explicit natural-langu

FAQs

BACKENDFORGE基准主要评估什么？

评估代理式LLM能否在给定自然语言规范与OpenAPI合约下，构造可部署的后端服务，并使其对外可观测行为满足合约要求。

为什么在BACKENDFORGE中使用后端服务与OpenAPI合约？

后端服务提供受控但真实的执行底座，其API语义可通过基于HTTP的黑盒交互，并与OpenAPI合约进行确定性对照评测。

BACKENDFORGE如何提升测试与评测的有效性？

通过测试代理与代码代理协同演化：测试代理提出规范约束下的后端测试，代码代理修复参考实现，从而在不引入隐藏需求的前提下加强评测。

Jul 16, 2026

Upload by Eden