Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications

Jul 16, 2026

Upload by Riley

AI code-generation tools embedded in programming environments shift students from writing code to specifying intended behavior in natural language. However, the specifications students provide via comments remain understudied. This study analyzes a four-year dataset of undergraduate submissions and reflections from tasks where students used comments to guide GitHub Copilot, iteratively refining solutions with test-case feedback. A taxonomy spans comment type, code expression level, and targeted code construct, enabling automated analysis of changes across attempts.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Introduction

Related Work

AI Prompting and Code Generation

Study Overview and Research Questions

Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications Nasser Giacaman University of Auckland Auckland, New Zealand [n.giacaman@auckland.ac.nz](n.giacaman@auckland.ac.nz) Valerio Terragni University of Auckland Auckland, New Zealand [v.terragni@auckland.ac.nz](v.terragni@auckland.ac.nz) Paul Denny University of Auckland Auckland, New Zealand [paul@cs.auckland.ac.nz](paul@cs.auckland.ac.nz) Viraj Kumar University of New South Wales Bengaluru, India [viraj.kumar1@unsw.edu.au](viraj.kumar1@unsw.edu.au) arXiv :2607 . 10674v 1 [ cs . SE] 12 Jul 2026 Abstract As AI code tools become integrated into programming environments, students increasingly describe intended behavior in natural language and rely on these tools to generate code, shifting emphasis from code writing to specification. Yet little is known about the comments students write as specifications in AI-assisted programming tasks. We analyze a four-year dataset of undergraduate programming submissions and reflections from tasks in which students wrote comments to guide code generation and refined solutions using test-case feedback. We introduce a taxonomy spanning three dimensions: comment type, code expression level, and code construct. Using automated classification, we examine how these dimensions vary across attempts and how students describe the process in their reflections. Our findings show that students mostly wrote naturallanguage What comments, shifted toward How comments for more procedural constructs, and focused more on verifying generated code than on repeatedly rewriting comments. CCS Concepts • Social and professional topics → Computing education. Keywords AI-assisted programming, Code comprehension, Code generation, GitHub Copilot, Student comments ACM Reference Format: Nasser Giacaman, Valerio Terragni, Paul Denny, and Viraj Kumar. 2026. Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications. In Proceedings of the 2nd ACM Virtual Global Computing Education Conference V.1 (SIGCSE Virtual 2026), November 12–15, 2026, Virtual Event, USA. ACM, New York, NY, USA, 7 pages. [https:](https:)//[doi.org/10.1145/3795867.3830978](doi.org/10.1145/3795867.3830978) 1 Introduction AI code-generation tools are increasingly integrated into programming environments [33], changing the skills students need when This work is licensed under a Creative Commons Attribution 4 .0 International License. SIGCSE Virtual 2026, Virtual Event, USA © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2506-7/2026/11 [https://doi.org/10.1145/3795867.3830978](https://doi.org/10.1145/3795867.3830978) learning to program [25] . Rather than writing every line, students can describe intended behavior in natural language and rely on AI tools to generate candidate implementations [7]. This has motivated new tasks in computing education. Prompt Problems ask students to solve tasks by writing prompts that elicit correct solutions from a code-generating model [8], while dialogue-based prompt programming environments support iterative natural-language interaction with AI models [27] . Other work asks students to clarify ambiguous specifications before prompting AI to generate solutions [24] . However, this skill is not straightforward for novices. Students can struggle to describe their intent, evaluate generated code, and revise prompts when the output is incorrect [13, 20] . Different patterns ofAI use may have different implications for learning, with hybrid approaches involving human judgment and verification appearing more promising than simply asking an AI tool for complete solutions [16] . There are also concerns that over-reliance on AI might harm core skills such as code reading and comprehension. In this paper, we analyze student-written comments from four years of an AI-assisted programming activity in an undergraduate object-oriented p

FAQs

What is the main focus of the study on Copilot-assisted programming tasks?

The study focuses on the comments students write as natural-language specifications to guide AI code generation, and how those comments change across attempts and code constructs.

What data source does the paper analyze?

It analyzes a four-year dataset consisting of undergraduate programming submissions and student reflections from an AI-assisted object-oriented programming activity using GitHub Copilot.

What taxonomy dimensions does the paper introduce for categorizing student comments?

The taxonomy characterizes each comment by its purpose, the level of code-like expression, and the targeted code construct.

Jul 16, 2026

Upload by Riley