The Cognitive Cost of Poor Item Design - How Item-Writing Flaws Shape Response Time and Difficulty Across Assessment Domains

Oct 15, 2026

Item-writing flaws (IWFs) violate multiple-choice question guidelines and undermine assessment quality, yet prior studies report inconsistent effects on item difficulty. This work tests a measurement account: different IWF types impose distinct cognitive demands that cancel when aggregated. Using cognitive load theory, 19 automatically detected IWFs are classified into cuegiving, confusion-inducing, and structural complexity. Across medical licensure, K-12 mathematics, and university computer science, confusion-inducing flaws increase response time by 14–29% and indirectly raise difficulty via longer processing time.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

ABSTRACT

1. INTRODUCTION

Multiple-choice questions and the role of item quality

Cognitive load mechanisms of different flaw types

The Cognitive Cost of Poor Item Design: How Item-Writing Flaws Shape Response Time and Difficulty Across Assessment Domains Steven Moore Information Sciences and Technology George Mason University [StevenJamesMoore@gmail.com](StevenJamesMoore@gmail.com) ABSTRACT Item-writing flaws (IWFs), violations of established guidelines for constructing multiple-choice questions, are known to compromise assessment quality. Their effects on item difficulty have proved inconsistent, with flawed items sometimes appearing easier, sometimes harder, and often no different from well-constructed items. We propose that this reflects a measurement problem, as different IWFs impose distinct cognitive demands that cancel or dilute when aggregated. Drawing on cognitive load theory, we classify 19 automatically detected IWFs into three categories of cuegiving, confusion-inducing, and structural complexity based on their expected cognitive function. Analyzing 1,230 items across three domains (medical licensure, K-12 mathematics, and university computer science), we find that each domain exhibits a different mix of IWFs rather than simply more or fewer flaws. Flaws categorized as confusion-inducing consistently increase response time by 14-29% across all three datasets, while cue-giving and structural flaws show no significant effects. This pattern holds across all three datasets, with no significant flaw-by-dataset interactions, suggesting the effect is robust to the differences in subject matter. Bootstrap mediation analyses reveal that confusion-inducing flaws increase processing time, which in turn associates with greater difficulty. This indirect effect is 2.5 times the magnitude of the aggregate indirect effect. These findings suggest that item quality improvement efforts should prioritize confusion-inducing flaws, which are the primary source of unnecessary cognitive burden on examinees. Keywords Assessment Quality, Response Time, Question Difficulty 1. INTRODUCTION Multiple-choice questions (MCQs) remain the most widely used assessment format in education, certification, and professional licensure, where their quality directly affects what Steven Moore. The Cognitive Cost of Poor Item Design: How ItemWriting Flaws Shape Response Time and Difficulty Across Assessment Domains. In Anthony Botelho, Maria Mercedes T. Rodrigo, Adish Singla, Hiroaki Ogata, Hyojeong So, and Young Hoan Cho (eds.) Proceedings of the 19th International Conference on Educational Data Mining, Seoul, Republic of Korea, June, 2026, pp. 293– 305. International Educational Data Mining Society (2026) . © 2026 Copyright is held by the author(s) . This work is distributed under the Creative Commons Attribution NonCommercial NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. [https://doi.org/10.5281/zenodo.21040093](https://doi.org/10.5281/zenodo.21040093) we can validly infer about the knowledge and skills of students [58] . With generative AI now accelerating item writing, quality is both more necessary and more possible [4] . Clear standards govern how to create high quality questions, but violations of those standards, item-writing flaws (IWFs), continue to appear even in commercial e-learning instruction [9, 36] . This compromises construct validity, fairness, and the interpretability of learning analytics derived from these assessments [12, 56] . IWFs impose unnecessary cognitive load as students interpret unclear wording rather than demonstrate knowledge, artificially inflating time-ontask through extraneous processing [44, 26, 53] . Not all flaws are expected to impose the same kind of cognitive load, some may even reduce it by cueing the answer, while others introduce confusion that slows examinees down. When researchers have tested whether IWFs actually affect item performance, the results have been equivocal. Studies examining whether IWFs predict item difficulty or discrimination have reported weak, inconsistent, or null associations

FAQs

Why do item-writing flaws lead to inconsistent difficulty findings in prior research?

Different IWF types can impose opposite cognitive demands. When all flaws are aggregated into one count, opposing effects cancel, masking clear relationships with difficulty.

How are the 19 item-writing flaws categorized in this study?

The study classifies them into three cognitive-function groups: cue-giving, confusion-inducing, and structural complexity, based on expected effects on the examinee’s cognitive processing.

What is the main empirical effect of confusion-inducing flaws across domains?

Confusion-inducing flaws consistently increase response time by 14–29% across the medical licensure, K-12 mathematics, and university computer science datasets.

Oct 15, 2026

Uploaded by Aria Callaghan