Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions

Q: How accurately can programmers judge the correctness and completeness of LLM-generated assertions?

Programmers show relatively good accuracy for correct assertions (74%), but accuracy drops substantially for incorrect assertions (49%). They also consider both correctness and completeness when evaluating generated assertions.

Q: What effect do natural-language explanations have on programmers’ judgment of assertions?

Natural-language explanations provide no overall benefit. Low-quality explanations can impair specification assessment accuracy while simultaneously increasing developer confidence.

Q: Why does the study focus on postcondition assertions?

Postconditions have well-defined formal semantics, enabling precise measurement of correctness and completeness. The setup also reflects a realistic human-in-the-loop reliability workflow where LLMs can generate useful assertions but still require human review.

Jul 16, 2026

Upload by Miles

Code comprehension and code review remain critical software engineering activities, and AI code generation increases their importance. Generative AI can augment code with assertions and natural-language explanations, yet effectiveness for evaluating such support is unclear. A controlled experiment with 86 Python programmers and a follow-up think-aloud study examine accuracy in judging correctness and completeness of generated assertions of varying quality and assess how explanations affect those judgments. Participants judge correct assertions reasonably well (74%) but incorrect ones poorly (49%) with statistically significant accuracy differences. Natural-language explanations show no overall benefit: low-quality explanations reduce assessment accuracy while increasing confidence, suggesting reliability artifacts may require careful evaluation rather than blind adoption.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

1 Introduction

Background: AI in software engineering

Reliability artifacts and key assumption

Research questions and study focus

Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions Zhanna Kaufman [zhannakaufma@umass.edu](zhannakaufma@umass.edu)[ ](zhannakaufma@umass.edu)University of Massachusetts Amherst Amherst, MA, USA Yuriy Brun [brun@cs.umass.edu](brun@cs.umass.edu)[ ](brun@cs.umass.edu)University of Massachusetts Amherst Amherst, MA, USA Adithya Murali [adithyamurali@cs.wisc.edu](adithyamurali@cs.wisc.edu)[ ](adithyamurali@cs.wisc.edu)University of Wisconsin–Madison Madison, WI, USA Madeline Endres [mendres@umass.edu](mendres@umass.edu)[ ](mendres@umass.edu)University of Massachusetts Amherst Amherst, MA, USA arXiv :2607 .08885v 1 [ cs . SE] 9 Jul 2026 Abstract Code comprehension and code review are already critically important software engineering tasks, and the rising use of AI code generation tools is only increasing that importance. Generative AI has the possibility of supporting these activities, for example by augmenting code with assertions and natural-language explanations describing code behavior. However, little is known about how effective such support may be. We conduct a controlled experiment with 86 Python programmers and a follow-up think-aloud study to examine developers’ ability to assess the correctness and completeness of generated assertions of varying quality, and to investigate how natural-language explanations influence these assessments. While programmers can somewhat accurately judge correct assertions (74% accuracy), they perform poorly when shown incorrect assertions (49% accuracy), despite reporting similar levels of confidence in both judgments. This difference in judgment accuracy is statistically significant (􀀿 < 0. 001): the odds of a developer accurately judging a correct assertion was nearly three times higher than the odds of accurately judging an incorrect assertion (OR = 2.94) . Surprisingly, natural-language explanations of assertions provided no overall benefit. Furthermore, low-quality explanations could impair specification assessment accuracy (􀀿 = 0.037, 􀀤􀀧 = 0. 58) while simultaneously increasing developer confidence (􀀿 = 0.005, 3. 99/5 vs. 4. 25/5) . Our findings suggest that, contrary to common assumptions, AI assistance may not improve the reliability of code comprehension and review. More broadly, our findings highlight the importance of helping developers evaluate machine-generated reliability artifacts, in addition to generating them. 1 Introduction Generative AI tools have transformed software engineering, and are now a routine part of virtually every aspect of development including code generation [63], architectural design [34], debugging [63], and verification [20] . This makes code comprehension and code review, already important parts of the software engineering lifecycle, even more important, as humans are asked to be part of the loop, reviewing, adapting, and adopting automatically generated code [55, 78]. However, reviewing code is difficult, time-consuming, and cognitively demanding [5, 6] . Programmers often struggle to understand non-self-authored code [74], including LLM-generated code [69], as such code may be incorrect or misaligned with user intent [17, 77], which can be difficult to fix in practice [80] . To reduce this burden, researchers have proposed shifting developer review toward smaller reliability-oriented artifacts such as tests [17, 41], specifications [40, 81], or assertions [16] . For example, Endres et al. propose using LLMs to generate executable assertions that allow developers to review intended program behavior rather than the full implementation, reducing the amount of code that must be inspected while enabling run-time validation of generated code. Similarly, Lahiri envisions a future in which LLMs help formalize user intent through behavioral artifacts such as tests, assertions, and formal specifications [40] . By formalizing intended program behavior, such artifacts have the potential to help developers determine

FAQs

How accurately can programmers judge the correctness and completeness of LLM-generated assertions?

Programmers show relatively good accuracy for correct assertions (74%), but accuracy drops substantially for incorrect assertions (49%). They also consider both correctness and completeness when evaluating generated assertions.

What effect do natural-language explanations have on programmers’ judgment of assertions?

Natural-language explanations provide no overall benefit. Low-quality explanations can impair specification assessment accuracy while simultaneously increasing developer confidence.

Why does the study focus on postcondition assertions?

Postconditions have well-defined formal semantics, enabling precise measurement of correctness and completeness. The setup also reflects a realistic human-in-the-loop reliability workflow where LLMs can generate useful assertions but still require human review.

Jul 16, 2026

Upload by Miles