Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft’s Early 2026 Rollout of Claude Code and GitHub Copilot CLI

Jul 16, 2026

Organizations introducing agentic command line tools such as Claude Code and Copilot CLI need clarity on adoption drivers, retention patterns, and whether generated work justifies cost. At Microsoft during an early-2026 rollout, analysis of tens of thousands of engineers shows that initial use spreads mainly via social networks. Retention correlates more with ongoing coding activity than demographics. Adopters merge about 24% more pull requests than they otherwise would, using merged PRs as an output proxy, and the lift persists across a four-month window.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Introduction

Adoption and retention framework

Related Work

Adoption of developer AI assistance

arXiv :2607 .0 14 18v 1 [ cs . SE] 1 Jul 2026 Adoption and Impact of Command-Line AI Coding Agents A Study of Microsoft’s Early 2026 Rollout of Claude Code and GitHub Copilot CLI Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva Microsoft [emerson. rex@microsoft. com](emerson. rex@microsoft. com) [jennbu@microsoft. com](jennbu@microsoft. com) [alexandra. savelieva@microsoft. com](alexandra. savelieva@microsoft. com) Abstract Organizations rolling out agentic command line tools like Anthropic’s Claude Code and GitHub’s Copilot CLI need to know who will try them, who will keep using them, and whether the tools produce enough output to justify their cost. At organizational scale, token spend can run into millions of dollars annually, so misreading adoption, retention, or impact can make a rollout expensive without changing engineering velocity. Studying tens of thousands of engineers at Microsoft over its early-2026 rollout, we find that first use spread primarily through social networks, retention was associated more with engineers’ coding activity than with demographics, and adopters merged roughly 24% more pull requests than they would have otherwise. We use merged pull requests as our proxy for output — acknowledging that a merged PR is not the same as the value it delivers—and the lift persists across our four-month window. These results suggest that CLI coding agents are neither uniformly adopted nor mere novelty effects and that organizations should treat visible peer use as central to rollout strategy. 1 Introduction Agentic command line tools like Anthropic’s Claude Code, Google’s Gemini CLI, and GitHub’s Copilot CLI are increasing in popularity among software developers. Such tools harness agents that call large language models, where the agents execute semi-autonomous commands on behalf of the user from the command line. In early 2026, the Pragmatic Engineer’s survey indicated that Claude Code was the most popular AI-based developer tool among respondents [26] . At the same time, organizations considering whether to purchase such tools do not yet know which of their engineers are likely to adopt and use them. At the end of 2025, StackOverflow’s survey [35] of more than 49,000 respondents indicated that developers’ trust in AI is falling and that “developers remain willing but reluctant to use AI”. Even if organizations choose to use AI, the tokens needed to execute these tools can be expensive. At the extreme high end, Fortune reports on Meta employees’ usage of AI [14]: In a 30-day period, total employee usage on the dashboard exceeded 60 trillion tokens, and the highest-ranked individual user averaged 281 billion tokens. Using the least expensive version of Claude Opus 4 . 6, which costs $5 for every million tokens, that one user alone could have cost Meta more than $1 .4 million. Even at more modest levels of usage, organizations may wonder what return on investment they should expect. Three concerns thus follow any rollout: which engineers will adopt, whether they will keep using the tool, and whether the tool produces enough additional output — which we operationalize as merged pull requests — to justify its cost. To examine them, we analyze our experience in early 2026 with Microsoft offering its engineers two agentic command line tools — Claude Code and Copilot CLI—over a roughly four-month window of usage and pull request (PR) activity. This paper contributes the first field study to use developer-level telemetry to analyze both the adoption of agentic command line tools and their effect on pull-request output. Prior developer AI adoption work typically relies on surveys and interviews, and prior impact work largely infers AI use from public-repository signals (Section 2); our enterprise setting instead observes every engineer who could adopt alongside direct usage. Within this setting we separate initial use from retention, trace merged-PR output against how intensive

FAQs

How is the tool’s impact measured in the study?

Impact is operationalized as merged pull requests. The study uses merged PRs as a proxy for output, noting that a merged PR is not identical to the value it delivers, and finds a sustained lift across the four-month window.

Jul 16, 2026

Upload by Lucas Martin