3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Jul 16, 2026

Coding agents now generate full pull requests, but practitioners disagree on how this reshapes code review—whether review becomes the delivery bottleneck, whether human review remains necessary, and whether the practice’s educational value erodes. Repository-mining studies describe unstable surface trends without explaining mechanisms, and even their directions can flip under reasonable analysis choices. This work synthesizes 38,709 grey-literature sources into an explanatory causal theory via an LLM-assisted workflow, yielding 26 constructs and 67 relationships, while framing competing claims as falsifiable propositions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages143

Reading Time6 h 0 min

Introduction

Practitioner disagreements and proposed consequences

Limitations of repository-mining studies

Need for causal mechanisms and theory

Grey-literature synthesis with LLM-assisted methods

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse Shyam Agarwal, Courtney Miller, Christian Kstner, and Bogdan Vasilescu Carnegie Mellon University [shyamaga@andrew.cmu.edu](shyamaga@andrew.cmu.edu), [courtneymiller@cmu.edu](courtneymiller@cmu.edu), [vasilescu@cmu.edu](vasilescu@cmu.edu) arXiv :2607 .07980v 1 [ cs . SE] 8 Jul 2026 Abstract—Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built. Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. A motivating observational analysis of public GITHUB activity finds that agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones, yet the direction of these trends flips under different but equally defensible analysis choices, so the traces establish what is changing without explaining why. To recover the mechanisms, we synthesize practitioner discourse at scale into an explanatory theory: we collect 38,709 grey-literature documents (engineering blogs and Reddit threads), filter to those substantively about code review, and code a stratified random sample of 3,100 with an LLM-assisted pipeline, from which we build a causal model of 26 constructsand 67 relationships (64 directed, 3 contested). Its organizing claim is that review is the control point through which a coding agent’s effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. The theory makes the competing positions explicit and turns“AI is changing code review” into falsifiable propositions with named constructs and moderators. As a secondary contribution, we offer the underlying LLM-assisted, grey-literature theorybuilding method as a scalable template for software-engineering research, with a public implementation. I. INTRODUCTION Code review is under pressure from AI, but there is little agreement on how exactly it is changing or what the consequences will be. As coding agents can now author entire pull requests, practitioners have staked out sharply different positions on how this does and should affect review [1]–[3] . A common refrain is that agents now produce code faster than humans can review it, so that review becomes the delivery bottleneck [4], [5] . Some worry about the human cost, that the work shifts people away from an activity many enjoy, coding, toward one many do not, reviewing, with disengagement and burnout risks [6]–[8] . In open source, there is concern about large amounts of AI-generated low-quality contributions flooding projects, overwhelming maintainers and externalizing the cost of review onto them [9]–[11] . Others argue the opposite, that human review may no longer be necessary at all and can be delegated to AI reviewers or folded into the agent’s own loop as it checks its own work [12]–[14] . Still others warn of subtler and possibly delayed harms, a creeping cognitive debt or cognitive surrender as developers stop building the understanding that review once demanded [15]–[17] . These effects are not well understood, and they are difficult to study. Repository-mining work has been measuring how AI is changing development activity [18]–[20], including code review, but the results so far are fragmentary and at times conflicting, reporting opposite signs for basic quantities such as how large agent-authored changes are and whether they are merged more or less readily than human ones [21]–[26] . In our own analysis of code review of AI-authored pull requests on GitHub (Section II-B), we find trends, but they are not always stable under different but equally

FAQs

What disagreements do practitioners have about AI-authored pull requests and code review?

Practitioners differ on whether agents make review a bottleneck due to speed, whether humans can be removed or replaced by AI reviewers, and whether subtler harms like reduced developer understanding will emerge.

Why do repository-mining studies fail to fully answer how code review changes?

They often report unstable surface trends and do not explain underlying mechanisms, and the sign of measured effects can reverse under different but defensible operational choices.

How does the paper build a causal theory of code review in an AI world?

It collects 38,709 grey-literature documents, filters for content about code review, codes a stratified random sample of 3,100 using an LLM-assisted pipeline, and derives a causal model with 26 constructs and 67 relationships.

Jul 16, 2026

Upload by Genevieve