Analysis of the Scoring and Reliability for the Duolingo English Test - Procedures and Reliability/Precision

Aug 30, 2026

Uploaded by Connor

Review of research supporting Duolingo English Test (DET) scoring methods and the reliability/precision of resulting scores. The document applies the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014) together with DET documentation and analyzed questions on established scoring procedures, item generation and difficulty estimation, item selection for administration, and how item responses produce total scores and subscores. It further examines evidence for valid, appropriate score interpretations, including total-score and subscore reliability/precision, standard error of measurement, and inter-rater reliability for writing and speaking responses.

LanguageEnglish

CategoryExam

FormatPDF

Pages27

Reading Time1 h 8 min

AccessFree to download

Section 1 Purpose of this Document

Section 2 Sources Reviewed

Section 3 DET Scoring Procedures

Overview of Scoring Procedures

Computer-Adaptive Testing (CAT)

Analysis of the Scoring and Reliability for the Duolingo English Test Table of Contents Section 1 Purpose of this Document 2 Sources Reviewed 3 DET Scoring Procedures 4 Score Reliability/ Precision – Overview 5 Procedures for Item Generation and Estimation of Item-Level Difﬁculty 6 Procedures for Scoring Item Responses 7 Procedures for Deriving Total Scores and Subscores 8 Procedures for Supporting Valid and Appropriate Score Interpretations 9 DET Total Score Reliability/ Precision 10 DET Subscore Reliability/ Precision 11 Standard Error of Measurement 12 Inter-Rater Reliability of Writing and Speaking Responses References Section 1: Purpose of this Document The purpose of this document is to report our review of the research supporting the Duolingo English Test (DET) scoring and score reliability/precision. We used the Standards for Educational and Psychological Testing (AERA , APA , & NCME, 2014) and DET’s documentation to analyze the topics identiﬁed below. As the Standards address scoring and reliability issues across various chapters, this document integrates sections from chapters 2, 5, and 6. Key considerations related to scoring and reliability associated with the chapters include: Standard 2.0: Appropriate evidence of reliability/precision should be provided for the interpretation for each intended score use (AERA , APA , & NCME, 2014, p. 42) . Standard 5.0: Test scores should be derived in a way that supports the interpretations of test scores for the proposed uses of tests. Test developers and users should document evidence of fairness, reliability, and validity of test scores for their proposed use (AERA , APA , & NCME, 2014, p. 102) . Standard 6.0: To support useful interpretations of score results, assessment instruments should have established procedures for test administration, scoring, reporting, and interpretation. Those responsible for administering, scoring, reporting, and interpreting should have sufﬁcient training and supports to help them follow the established procedures. Adherence to the established procedures should be monitored, and any material errors should be documented and, if possible, corrected (AERA , APA , & NCME, 2014, p. 114) . Questions Analyzed We analyzed the following questions related to the scoring of the DET: 1. What are the established scoring procedures for the DET? ● What are DET ’s procedures for generating items and estimating item-level difﬁculty? ● What are DET ’s procedures for selecting items for administration? ● What are DET ’s procedures for scoring item responses? ● What are DET ’s procedures for deriving total scores and subscores? ● What are DET ’s procedures for supporting valid and appropriate score interpretations? 2. What is the level of reliability/precision of the DET scores? ● What is the reliability/precision of the DET total scores? ● What is the reliability/precision of the DET subscores? ● What is the amount of error contained within the DET scores? In what follows, we summarize our analysis and use examples of evidence collected from DET ’s documentation to address the questions. Back to Table of Contents Section 2: Sources Reviewed To address the aforementioned goals and questions, we reviewed publicly-available DET documentation (e.g. , articles and DET websites for test takers and institutions) . We also reviewed a few independently completed evaluations of the DET and held semi-structured interviews with DET staff. The staff included the chief of assessment and assessment scientists. The published resources used in the development of this report are cited in the reference section at the end of this document. Back to Table of Contents Section 3: DET Scoring Procedures Standard 5.16: When test scores are based on model-based psychometric procedures, such as those used in computerized adaptive or multistage testing, documentation should be provided to indicate that scores have co

FAQs

What is the purpose of this document on the DET?

It reports the review of research supporting DET scoring and the reliability/precision of scores. It evaluates scoring-related procedures and evidence for score interpretation.

How does the DET scoring process estimate item difficulty?

DET uses machine learning and natural language processing models to create proficiency scales and to estimate item difficulty for computer-adaptive test (CAT) algorithms.

Which reliability and precision aspects does the document analyze?

It covers DET total-score reliability/precision, subscore reliability/precision, the standard error of measurement, and inter-rater reliability for writing and speaking responses.

Aug 30, 2026

Uploaded by Connor