Validity Verification of the New TOEFL Writing Task Based on Classical Test Theory

Sep 30, 2026

Upload by Blitz

The TOEFL iBT Academic Discussion Task (ADT) was introduced to assess candidates’ ability to engage in academic discourse, yet evidence on its validity and fairness for culturally and linguistically diverse groups remains limited. This study applies Classical Test Theory to evaluate the ADT’s psychometric properties among Chinese university students. Results show a robust correlation between ADT and CET-6 writing and translation sub-scores, with strong expert agreement supporting construct validity and scoring rubric suitability. Gender differences are minimal, indicating fairness. Recommendations include refining cultural sensitivity in the rubric and retaining CET-6 writing sub-scores for prediction to better serve diverse test-taker needs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

AccessFree to download

Abstract

TOEFL iBT writing task background

Validity verification framework (CTT)

Reliability and validity concepts in CTT

Criterion validity and correlation interpretation

Content validity via expert judgment

Item- and scale-level content validity indices

Inter-rater reliability (ICC)

1 Validity Verification of the New TOEFL Writing Task Based on Classical Test Theory Zhang Yinyu School of Foreign Language, Zhongnan University of Economics and Law August 25, 2025 2 Abstract The TOEFL iBT has introduced the Academic Discussion Task (ADT) to assess test-takers’ability to engage in academic discourse, reflecting the growing emphasis on interactive communication skills in higher education. However, research on the ADT ’s validity and fairness particularly for culturally and linguistically diverse groups, such as Chinese students, remains limited. This study addresses this gap by employing Classical Test Theory (CTT) to evaluate the psychometric properties of the ADT among Chinese university students. This study finds a robust correlation between the ADT and the CET-6 writing and translation subscores. In addition, there is a high level of expert agreement regarding the construct validity evidence and the appropriateness of the scoring rubric. Furthermore, the results indicate that gender differences in validity indices are minimal. Taken together, these results suggest that the ADT is a valid measure for Chinese test-takers without gender discrimination. However, it is recommended that the cultural sensitivity of the scoring rubric be further refined and that the CET-6 subscores for writing be retained for predictive purposes, in order to better accommodate the needs of diverse test-taker populations. By addressing these issues, this study contributes to the broader discourse on fairness and validity in high-stakes language assessments. Keywords: Classical Test Theory, Validity Verification, TOEFL Writing, Academic Discussion Task 3 Validity Verification of the New TOEFL Writing Task Based on Classical Test Theory The TOEFL iBT (Internet-Based Test) is a comprehensive English language proficiency test that evaluates non-native speakers’ ability to communicate effectively in academic settings. Since July 2023, the writing section has been revised to include two tasks: the Integrated Writing Task and the new Academic Discussion Task (ADT) . The ADT simulates an online classroom discussion, requiring test-takers to present their views and provide arguments on a given topic within 10 minutes. However, few studies have examined its validity and fairness. Therefore, validation is essential to ensure that the ADT measures the intended construct and functions fairly across genders. Validity verification Classical Test Theory (CTT), a foundational theory in psychological and educational measurement, is widely used for analyzing the reliability and validity of standardized tests due to its operational simplicity and intuitive interpretation (Bachman, 1990; Swaminathan, n.d.) . The core assumption of CTT is that each observed score (X) consists of a true score (T) and an error component (E), expressed as X=T+E (Lord & Novick, 1968) . Key propositions of CTT include reliability, validity, and item analysis. Reliability reflects the stability and consistency of measurement results, with commonly used indices such as Cronbach’s alpha and split-half reliability. Validity concerns whether a test measures the intended psychological or ability constructs, with common types including content validity, construct validity, and criterion validity (Messick, 1989) . In addition, CTT examines item difficulty 4 and discrimination, which help evaluate the quality of test items and the overall measurement effectiveness (AERA/APA/NCME, 2014) . Within CTT, criterion validity is typically evaluated by correlating test scores with external criteria, such as standardized proficiency tests or academic outcomes. The Pearson correlation coefficient (r) quantifies the strength and direction of the linear relationship. Rather than invoking a universal cut-off, researchers interpret r with respect to decisionstakes and domain norms. In high-stakes language testing, coefficients ≥ 0.60 are typical

FAQs

What is the main purpose of validating the TOEFL iBT Academic Discussion Task (ADT)?

To determine whether the ADT measures the intended construct and operates fairly, especially for culturally and linguistically diverse test-taker groups.

How does the study evaluate validity in relation to Classical Test Theory (CTT)?

It uses CTT to examine psychometric properties, including reliability aspects and validity evidence such as construct and scoring-rubric appropriateness.

What do the results indicate about gender differences in the ADT’s validity indices?

Gender differences in validity indices are minimal, suggesting the ADT is a valid measure for Chinese test-takers without gender discrimination.

Sep 30, 2026

Upload by Blitz