Machine Learning–Driven Language Assessment - Method for Rapidly Creating Valid, Reliable, and Secure Proficiency Tests

Sep 30, 2026

We describe a method for rapidly creating language proficiency assessments, with experimental evidence that resulting tests are valid, reliable, and secure. The approach is the first to use machine learning and natural language processing to induce proficiency scales from a given standard, then use linguistic models to estimate item difficulty for computer-adaptive testing. This reduces expensive human pilot testing and enables a large item bank, demonstrated with the Duolingo English Test.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

1 Introduction

2 Background

2.1 Item Response Theory (IRT)

Machine Learning–Driven Language Assessment Burr Settlesand Geoffrey T. LaFlair Duolingo Pittsburgh, PA USA {burr,[geoff](geoff}@duolingo.com)[}](geoff}@duolingo.com)[@duolingo.com](geoff}@duolingo.com) Masato Hagiwara∗ Octanove Labs Seattle, WA USA [masato@octanove.com](masato@octanove.com) Abstract We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales basedona given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviatesthe need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with otherhigh-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements. 1 Introduction Language proficiency testing is an increasingly important part of global society. The need to demonstrate language skills—often through standardized testing—is now required in many situations for access to higher education, immigration, and employment opportunities. However, standardized tests are cumbersome to create and maintain. Lane et al. (2016) and the Standards for Educational and Psychological Testing (AERA et al., 2014) describe many of the proceduresand requirements for planning, creating, revising, administering, analyzing, and reporting on high-stakes tests and their development. In practice, test items are often first written by subject matter experts, and then ‘‘pilot tested’’with a large number of human subjects for psy- ∗ Research conducted at Duolingo. chometric analysis. This labor-intensive process often restricts the number ofitems that can feasibly be created, which in turn poses a threat to security: Items may be copied and leaked, or simply used too often (Cau, 2015; Dudley et al., 2016) . Security can be enhanced through computeradaptive testing (CAT), by which a subset of items are administered in a personalized way (based on examinees’ performance on previous items). Because the item sequences are essentially unique for each session, there is no single test form to obtain and circulate (Wainer, 2000), but these security benefits only hold if the item bank is large enough to reduce item exposure (Way, 1998) . This further increases the burden on item writers, and also requires significantly more item pilot testing. For the case of language assessment, we tackle both of these development bottlenecks using machine learning (ML) and natural language processing (NLP) . In particular, we propose the use of test item formats that can be automatically created, graded, and psychometrically analyzed using ML/NLP techniques. This solves the ‘‘cold start’’ problem in language test development, by relaxing manual item creation requirements and alleviating the need for human pilot testing altogether. In the pages that follow, we first summarize the important concepts from language testing and psychometrics (§2), and then describe our ML/NLP methods to learn proficiency scales for both words (§3) and long-form passages (§4) . We then present evidence for the validity, reliability, and security of our approach using results from the Duolingo English Test, an online, operational English proficiency assessment developed using these methods (§5) . After summarizing other related work (§6), we conclude with a discussion of limitations and future directions (§7) . 247 Transactions of the Association for Computational Linguistics, vol. 8, pp. 247–263, 2020. [https://doi.org/10.1162/tacl](https://doi.org/10.1162/tacl a 00310)[ a ](https://doi.org/10.1162/tacl a 00310)[00310](https://doi.org/10.

FAQs

How does the method reduce the need for expensive human pilot testing?

It automatically induces proficiency scales using machine learning and natural language processing and uses linguistic models to estimate item difficulty for computer-adaptive testing, reducing reliance on human-graded pilot studies.

What psychometric framework and model are used for item scoring?

The work uses a logistic item response function, specifically the Rasch model, to express the probability of a correct response as a function of item difficulty and examinee ability.

What exam is developed using these methods?

The approach is used to develop the Duolingo English Test, and the paper reports that its scores align significantly with other high-stakes English assessments while remaining reliable and secure.

Sep 30, 2026

Upload by kopisore