The Duolingo English Test: Psychometric considerations - Measurement Model and Scalable Real-Time Rating System

Aug 29, 2026

The Duolingo English Test is a computer-adaptive assessment of English proficiency. The study proposes a measurement model for the test and derives its formal characteristics, then introduces a scalable real-time rating approach with favorable statistical properties. The method addresses differential model fit across item and person characteristics by applying the proposed framework to Duolingo English Test data. Results show that the signed residual time model fits well and reveals no statistical bias across different examinee groups.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

A Measurement Model for the Duolingo English Test

The Duolingo English Test: Psychometric considerations Duolingo Research Report DRR-20-02 February 27, 2020 (11 pages) [www.duolingo.com/research](www.duolingo.com/research) Gunter Maris􀀃 Abstract The Duolingo English Test is a computer adaptive test of English language proficiency. In this paper, a measurement model for the Duolingo English Test is proposed, and its formal characteristics are derived. A scalable real-time rating system, with favorable statistical properties, is proposed and applied to data from the Duolingo English Test to address issues related to differential model fit with respect to item and person characteristics. The results indicate that the signed residual time model fit the test data well, and that there is not evidence of statistical bias toward different groups of test takers This commissioned study is part of the Duolingo English Test external research collection. Introduction The Duolingo English Test is a large-scale, high-stakes, online test of English language proficiency that can be delivered anywhere and at any time. This approach to (language) testing inherently comes with a cold start problem: one needs a large item bank calibrated to an appropriate item response theory (IRT) model before one can start administering tests. The traditional solution to this problem, large scale pretesting, isnot only expensive but suffers from a number of drawbacks. One needs large numbers of test takers to respond to items under realistic testing conditions. Both ensuring realistic testing conditions and getting test takers from the appropriate populations are challenging. To support anywhere anytime testing, the Duolingo English Test is assembled uniquely for every test taker, in the form of a computer adaptive test (CAT) . For the Duolingo English Test an alternative solution to the cold start problem is pursued. Items are automatically generated, using machine learning and natural language processing algorithms, to match the Common European Framework of Reference for languages (CEFR, Council of Europe, 2001), with item difficulty parameters that are predicted using machine learning. Item responses are automatically scored on a continuous zero to one scale. The development and scoring of the Duolingo English Test are documented in Settles, LaFlair, and Hagiwara (in press) and LaFlair and Settles (2019) . The Duolingo English Test contains 10 item types, five of which are delivered in the CAT administration. These five item types include a ctest, audio yes/no vocabulary, text yes/no vocabulary, dictation, and elicited speech. The other five item types are open-ended speaking and writing tasks. The speaking tasks include a picture description prompt, a text-based prompt, and an audio prompt. The writing tasks include a picture description task and a text-based prompt. Duolingo English Test test takers receive a minimum of three and a maximum of seven of each of the CAT items; they receive four each of the open-ended writing and speaking tasks. Once real tests are being administered using the pre-calibrated item bank, actual response data under high stakes testing conditions from actual test takers become available. This response data can be used to a) update where needed the predetermined item difficulties, b) evaluate the fit of the IRT model, and c) to further train the machine learning and natural language processing algorithms to obtain ever better pre-calibrated item difficulties. A CAT administration benefits both test security and measurement accuracy. At the same time, however, it complicates statistical analyses of item response data. Below we introduce an appropriate measurement model for the Duolingo English Test, and an appropriate scalable algorithm for statistical inference on the parameters of the measurement model. A Measurement Model for the Duolingo English Test The basic observations collected with the Duolingo English Test are continuous item responses betw

FAQs

What is the Duolingo English Test and how is it delivered?

It is a large-scale, high-stakes online assessment that can be taken anywhere and at any time. The test is administered as a computer adaptive test using dynamically assembled items for each examinee.

What measurement model is proposed in the report?

The report proposes a measurement model and derives its formal characteristics for the Duolingo English Test. It emphasizes the signed residual time (SRT) model for continuous item responses and evaluates model fit.

How does the scalable real-time rating system help with test scoring and fairness?

It provides real-time statistical inference for the parameters of the measurement model and manages complications caused by CAT administration. The results indicate good model fit and no evidence of statistical bias toward different groups of test takers.

Aug 29, 2026

Uploaded by Clementine