An Exploratory Study on LLM-Generated Code and Comments in Code Repositories

Sep 21, 2026

Uploaded by Aurora

Exploratory research examines code and comments in repositories that are likely generated by large language models (LLMs), focusing on their characteristics, differences between company- and community-maintained repositories, and associations with bugs. Extensive experiments run on active company and community repositories from 2021 to 2025 using multiple LLM-generated content detectors and proxy analysis. Results indicate decreasing detected LLM-generated code over time, frequent appearances in test cases, stable detected LLM-generated comments, substantial intra-repository code clones, and low proportions of grammatically correct comment sentences, with limited coverage of human-labeled bugs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Abstract

1. Introduction

An Exploratory Study on LLM-Generated Code and Comments in Code Repositories Yongyi Ji, Jiaji Wang, Yi Zhou, Fuxiang Chen∗, Hongji Yang School of Computing and Mathematical Sciences, University of Leicester, University Road, Leicester, LE1 7RH, United Kingdom Abstract The use of LLMs in software development has become increasingly widespread on tasks such as code generation and summarization. Reports from large technology companies showed that around 20% to 30% of their code are generated by LLMs. However, there remains skepticism about the practical usage of LLM-generated code and comments, such as concerns on more time for debugging the generated code and the unnaturalness of the generated comments. In this paper, we study the code and comments detected as likely to be generated by LLMs and their characteristics, the differences between company-and community-maintained repositories, and how likely bugs are associated with LLM-generated code.We conduct extensive experiments on active companyand community-maintained repositories from 2021 to 2025 using various tools and techniques that detect code and comments generated by LLMs. Based on our detector-based proxy analysis, the results suggest that code detected as likely to be generated by LLMs decreased over time and appeared frequently in test cases, while that of comments remains relatively stable. Proxy results further suggest that code detected as likely to be generated by LLMs shows substantial intra-repository code clones, whereas comments exhibit a relatively low proportion of grammatically correct sentences. In addition, the company-maintained repositories show a higher percentage of code and comments detected as likely to be generated by LLMs, and only a small percentage of the human-labelled bugs are detected as being likely associated with LLM-generated code. Keywords: LLM-generated detection on Code and Comments, Empirical Study, Bug analysis, Code Clone Detection 1. Introduction The rapid advancement of Large Language Models (LLMs) has resulted in multiple adaptation of LLMs in various downstream tasks [1] . In Software Engineering (SE), it was reported that some developers are already using LLMs in code generation and summarization, the two most important activities in software development and comprehension [2, 3, 4, 5, 6, 7, 8] . Moreover, the JetBrains Developer Ecosystem Survey [9] found that developers use LLMs to generate code and comments frequently. There have also been reports suggesting that the use of LLMs is gaining popularity among software companies. For example, in 2024, Google stated that 25% of the company’s new code is generated by LLMs [10] . Similarly, in 2025, Mi- by Github Copilot was vulnerable. In the paper, the Github Copilot refers to the Copilot AI service hosted by Github. This is not to be mistaken for the Copilot model developed by Microsoft [15] . Liang et al. [16] surveyed 410 developers and found that the primary reason for not using LLMs was that the generated code did not meet their requirements. For comments, Sergeyuk et al. [17] collected opinions from 481 developers and found that one of the top reasons for not using comments generated by LLMs is that the generated comments are often unnatural and that they do not match the required tone and clarity of a human developer. Moreover, the code and comments that were generated by LLMs raised the uncertainty on whether a generated software artifact is created by an individual or a machine [18, 19] . arXiv :2607 .0 1867v 1 [ cs . SE] 2 Jul 2026 crosoft stated that 20% to 30% of the code within the company’s repositories is written by LLMs [11] . In addition, the 2025 Google’s DORA report [12], which surveyed nearly 5,000 professional developers from around the world, found that 90% of the surveyed developers reportedly use some elements of LLMs at work, and over 80% of them believed this had improved their productivity. On the other hand, there is reported skept

FAQs

What questions does the study aim to answer about LLM-generated code and comments?

It investigates the proportion of code and comments likely generated by LLMs, their characteristics, differences between company- and community-maintained repositories, and whether bugs are associated with LLM-generated code.

How are the experiments conducted and what time range is covered?

The study conducts extensive experiments on active company and community repositories from 2021 to 2025, using multiple tools and techniques that detect LLM-generated code and comments.

What main proxy-based findings are reported for code and for comments?

Detected LLM-generated code decreases over time and appears frequently in test cases, while detected LLM-generated comments remain relatively stable. Additionally, code shows substantial intra-repository clones, whereas comments have a relatively low proportion of grammatically correct sentences.

Sep 21, 2026

Uploaded by Aurora