Exploring AI-Enabled Test Practice, Affect, and Test Outcomes in Language Assessment - arXiv

Aug 29, 2026

Uploaded by Eden

Practice tests for high-stakes assessment are designed to increase test familiarity and reduce construct-irrelevant variance that can distort score interpretation. This work studies how generative AI-driven automated item generation (AIG) enables large-scale practice through repeated test opportunities, and how that exposure relates to official results. A large observational study (N=25,969) with the Duolingo English Test (DET) examines links among repeated practice, official DET scores, test-taker affect such as confidence, and score sharing for university admissions. Results show benefits for 1–3 practice tests, while more than 3 relates to lower performance, suggesting washback beyond familiarity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Introduction

Practice tests and construct-irrelevant variance

Generative AI automated item generation (AIG) and repeated practice

Duolingo English Test (DET) and free practice test

Background

Test preparation, affect, and performance relationships

Exploring AI-Enabled Test Practice, Affect, and Test Outcomes in Language Assessment Jill Burstein* Ramsey Cardwell Ping-Lin Chuang Allison Michalowski Steven Nydick Duolingo {jill, ramsey, pinglin, allison.michalowski, [steven.nydick}@duolingo.com](steven.nydick}@duolingo.com) arXiv :2508 . 17108v1 [ cs .CY] 23 Aug 2025 Abstract Practice tests for high-stakes assessment are intended to build test familiarity, and reduce construct-irrelevant variance which can interfere with valid score interpretation. Generative AI-driven, automated item generation (AIG) scales the creation of large item banks and multiple practice tests, enabling repeated practice opportunities. We conducted a large-scale observational study (N = 25,969) using the Duolingo English Test (DET)—a digital, highstakes, computer-adaptive English language proficiency test to examine how increased access to repeated test practice relates to official DETscores, test-taker affect (e.g., confidence), and score-sharing for university admissions. To our knowledge, this is the first large-scale study exploring the use of AIG-enabled practice tests in high-stakes language assessment. Results showed that taking 1-3 practice tests was associated with better performance (scores), positive affect (e.g., confidence) toward the official DET, and increased likelihood of sharing scores for university admissions for those who also expressed positive affect. Taking more than 3 practice tests was related to lower performance, potentially reflecting washback – i.e., using the practice test for purposes other than test familiarity, such as language learning or developing test-taking strategies. Findings can inform best practices regarding AI-supported test readiness. Study findings also raise new questions about test-taker preparation behaviors and relationships to test-taker performance, affect, and behaviorial outcomes. 1 Introduction For millions of international test takers, scores on high-stakes English language proficiency (ELP) assessments can profoundly impact their educational and professional goals. As a result, they engage in various test preparation strategies. For example, *Authors are listed alphabetically to reflect equal contributions. practice tests aim to build familiarity for a specific test; reading books and articles can improve English language reading skills; and, deliberate engagement in conversations with peers and instructors can strengthen English language speaking and listening skills. This paper focuses on practice tests. Practice tests aim to build test familiarity to reduce testdesign-related construct-irrelevant variance (CIV) . CIV is associated with the introduction of factors unrelated to the skills a test is intended to measure (the target construct) (Messick, 1982 ; Powers, 1985) . For instance, CIV can stem from unfamiliar technical features (e.g., drag-and-drop), lack of familiarity with the device required for taking a test (e.g., test requirements to use a laptop for test takers who have limited laptop experience (Koné et al. (2024)), or anxiety triggered by an unfamiliar format (Winke and Lim, 2017) . Conventional practice tests, often developed by testing organizations, aim to reduce CIV. However, they typically contain a limited number of fixed forms, restricting opportunities for repeated test practice. Modern generative AI-powered automated item generation (henceforth, AIG) alleviates this constraint by enabling the creation of large item pools for digital practice tests. As a result, practice test generation can be scaled to support repeated practice test opportunities for test takers. The Duolingo English Test (DET)is a digital, AI-driven, high-stakes, computer-adaptive ELP assessment used for international student university admissions. The DET is taken by hundreds of thousands of test takers each year. To help test takers become familiar with the test, the DET offers afree practice test that simulat

FAQs

What problem do practice tests aim to solve in high-stakes language assessment?

Practice tests are intended to build familiarity and reduce construct-irrelevant variance that can interfere with valid score interpretation.

How does generative AI-driven automated item generation (AIG) change practice test design?

AIG scales the creation of large item banks, enabling dynamic generation of multiple practice-test versions and repeated practice opportunities.

What did the study find about the relationship between number of practice tests and official DET outcomes?

Taking 1–3 practice tests was associated with better performance and more positive affect (e.g., confidence). Taking more than 3 practice tests was linked to lower performance, potentially reflecting washback.

Aug 29, 2026

Uploaded by Eden