L2-Bench An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education

Jul 16, 2026

Upload by Miles

Despite fast adoption of AI in education, rigorous evaluation for AI-powered educational systems remains limited, especially for second language (L2) learning. L2-Bench introduces an open-source benchmark with 1,000+ task-response pairs to evaluate LLM performativity through learning-experience design principles, not only knowledge of them. It provides a validated taxonomy of 12 competencies and 31 subcompetencies, a rubric-based evaluation method, and a dataset producing reliable signals on strengths, weaknesses, and robustness across L2 scenarios, enabling more informed AIED use and governance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages49

Reading Time2 h 3 min

Abstract

Introduction

Evaluation gap in AI for education

Motivation for L2-Bench

Benchmark contributions

L2-Bench: An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education James Edgell* 1 , Wm. Matthew Kennedy*2, 3 , Ben Knight 1 , Danielle Carvalho 1 , Martin Ku 1 , Isaac Pattis 1 1 Oxford University Press 2 Oxford Internet Institute, University of Oxford 3 King’s College London [elt-bench@oup.com](elt-bench@oup.com) arXiv :2607 .08842v 1 [ cs .CY] 9 Jul 2026 Abstract Despite rapid AI adoption in education, rigorous evaluation of AI-powered educational (AIED) systems remains critically underdeveloped, particularly in second language (L2) education, one of the most common yet least evaluated AI applications. We introduce L2-Bench, an open-source benchmark of 1,000+ task-response pairs to aid the pedagogy-led evaluation of LLM capabilities relating to language learning and assessment. Crucially, L2-Bench measures model performativity on the application of learning experience design principles rather than mere knowledge of those principles or broad learning outcomes. Our contributions include: (1) a validated taxonomy of 12 competencies and 31 subcompetencies validated by 200+ expert practitioners (task authenticity = 4.42/5.00, criteria adequacy = 4.18/5.00); (2) a rubric-based evaluation methodology that we believe can, if adapted, generalize to similar (open-ended, qualitative) disciplines; (3) an evaluation dataset that produces reliable signal about model strengths, weaknesses, and contextual robustness across diverse L2 education scenarios. We find that, among large models, Claude Opus 4.7 performs best overall (85.5%), though is marginally outperformed on several constituent tasks. We also find that performance drops notably on harder tasks ∈(69.9%–73.4%) . L2-Bench provides education stakeholders better methods to make more informed decisions about realworld AIED adoption, use, and governance, while advancing the maturing science of AI evaluations for education. 1 Introduction Despite rapid adoption of AI systems in educational spaces (Digital Education Council 2024), very few evaluations for AI in educational (AIED) exist. Those that purport to cover this space often suffer from poor construct validity, are underpowered, or focus more on broad effects rather than instance-level performativity. The result is little short of a“wild, wild west”: a situation in which deployment is far outpacing evidence even of basic validation of AIED systems (Hudig, Kallina, and Singh 2026) . This AIED evaluations gap reflects the ongoing crisis in AI evaluations in general. However, it is more acute: although prior validation and efficacy studies of conventional *These authors contributed equally. Copyright © 2026, Association for the Advancement of Artificial Intelligence ([www.aaai.org](www.aaai.org)). All rights reserved. education technologies suggest that these predecessor systems have not meaningfully improved learning outcomes (Cuban 2001; Zawacki-Richter et al. 2019; Selwyn 2012, 2019), many AIED systems replicate or extend the same conventional system design patterns (Jurenka et al. 2024) . Such a headlong rush on the part of AIED developers and adopters without parallel development of novel evaluation methodologies creates the conditions for myriad interaction-, learning-group-, systemic-, and compounding harms (Bastani et al. 2024; Holmes and Miao 2023; Kasneci et al. 2023; Wachter, Mittelstadt, and Russell 2024; Holmes 2024; Kennedy and Campos 2025) . In the meantime, a generation of learners must endure laboratory conditions (Alcaras and Ricci 2025) in their pursuit of educational attainment. Drawing inspiration from allied efforts to improve the state of the evaluations ecosystem (Reuel et al. 2024; Eriksson et al. 2025; Biderman et al. 2024; Weidinger et al. 2025; Schwartz et al. 2025; Bean et al. 2025; Paskov, Soder, and Smith 2025; Reuel et al. 2025), our work hastens progress in one particularly challenging subdomain of AI for education: second language (L2) educat

FAQs

What problem does L2-Bench address in second language education and AI evaluation?

L2-Bench targets the lack of rigorous, construct-valid evaluations for AI-powered educational (AIED) systems in second language (L2) education, where many existing evaluations are underpowered or focus on broad effects rather than task-level performativity.

How does L2-Bench evaluate LLMs differently from knowledge-only or outcome-only assessments?

L2-Bench measures model performativity by checking how well models apply learning experience design principles for language learning and assessment, rather than merely testing knowledge of those principles or broad learning outcomes.

What are L2-Bench’s main contributions and what does the benchmark contain?

L2-Bench contributes a validated taxonomy of 12 competencies and 31 subcompetencies, a rubric-based evaluation methodology, and a benchmark dataset of 1,000+ structured task-response pairs to support reliable comparisons across L2 education scenarios.

Jul 16, 2026

Upload by Miles