An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems

Jul 16, 2026

Upload by Violet

Role-based multi-agent code generation improves LLM effectiveness for repository-scale software engineering beyond small programming exercises. The evaluation covers 12 real-world Java repositories, showing higher similarity between generated and developer-written code than single-LLM baselines. The study also identifies a persistent performance gap versus human implementations. A sequential and reflexive multi-agent pipeline decomposes IEEE-style SRS requirements into tasks, generates Java code, and assembles buildable, compilable projects for assessment.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Multi-Agent Code Generation

Planning

Coding

Assessment

Setup

Theme Article: Special Issue on Engineering Agentic Systems An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems Benedetta Donato, University of Milano-Bicocca, Milano, Italy Noah Hagar-Dent, The University of Auckland, Auckland, New Zealand Aaron Worsnop, The University of Auckland, Auckland, New Zealand Leonardo Mariani, University of Milano-Bicocca, Milano, Italy Valerio Terragni, The University of Auckland, Auckland, New Zealand arXiv :2607 .042 12v 1 [ cs . SE] 5 Jul 2026 Abstract—Role-based multiagent code generation aims to make LLMs more effective on repository-scale problems, moving beyond small programming tasks. We evaluate this approach on 12 Java repositories, finding greater similarity to developer code than single LLMs, but a persistent gap from human implementations. L arge Language Models (LLMs) now support a wide range of software engineering activities, from requirements interpretation to code genera- tion and maintenance [3] . Yet generating entire software systems remains difficult: real-world repositories involve architectural structure, cross-file dependencies, build management, and non-functional constraints that amplify well-known LLM limitations: constrained context, hallucinations, brittle cross-file integration, and toolchain friction [2], [3] . LLMs also lack the high-level reasoning needed for complex requirements and large-scale architectural design [6], typically prioritising immediate functional specifications at the expense of more elaborate overarching requirements [1], constraining the complexity of systems achievable without substantial human guidance [6] . A promising direction is agentic code generation: distributing work across specialised LLM agents that mirror software team roles (e.g. , requirements analyst, architect, developer, tester) [4], [5] . A few recent frameworks have started exploring this direction: MetaGPT [12], AgileCoder [13], and RTADev [14] explore multi-agent collaboration for code generation through structured workflows, Agile role assignments, and intention alignment, respectively. However, these frameworks have been evaluated on function-level benchmarks (e.g., HumanEval, MBPP [15]) or small self-contained applications typically comprising fewer than 10 files and a few hundred lines of code. Successfully coordinating agents at repository XXXX-XXX © 2026 IEEE Digital Object Identifier 10.1109/XXX.0000.0000000 scale remains an active research challenge, and thereis still limited empirical understanding of when (and by how much) agentic approaches outperform standalone LLMs in realistic, repository-scale settings. This paper investigates this tradeoff, comparing standalone LLMs to multi-agent architectures for largescale code generation. To this end, we implemented a sequential and iterative multi-agent pipeline that decomposes requirements into implementable tasks, produces Java code, and integrates outputs into abuildable, compilable project. We then compared this agent team against an individual LLM on 12 real-world Java projects of varying sizes and complexities. The evaluation yielded insights into the current capabilities and limitations of LLM agents for large-scale code generation tasks. Multi-Agent Code Generation Figure 1 illustrates the Agentic code generation process that we implemented. Similarly to other agentic code generation processes [9], [10], it consists of four main phases: Planning, which produces the plan that has tobe completed to implement the application described in the input Software Requirements Specification (SRS); Coding, which runs LLM agents according to the plan to obtain the implementation; Assessment, which assesses the generated code, also against the input SRS; and Setup, which generates the files necessary to build and run the generated code. We consider two variants of our flow, one Sequential, where the assessment is skipped, and another Reflexive, where the

FAQs

What problem does role-based multi-agent code generation target compared with standalone LLMs?

It targets the limitations of generating code for repository-scale projects, where real repositories include architecture, cross-file dependencies, build management, and non-functional constraints that are harder than small tasks.

How is the evaluated multi-agent approach implemented in the study?

The approach uses a sequential and iterative multi-agent pipeline that decomposes IEEE-style SRS requirements into implementable tasks, generates Java code for those tasks, and integrates outputs into a buildable, compilable project.

What were the main evaluation findings across the 12 Java repositories?

The multi-agent architecture produced code that was more similar to developer implementations than single-LLM generation, yet it still showed a persistent gap from human-written code.

Jul 16, 2026

Upload by Violet