Question Type, Cognitive Load, and CEFR Alignment Evaluating LLM-Generated EFL Grammar Drill Exercises

Jul 16, 2026

This study evaluates the pedagogical viability of LLM-generated English as a Foreign Language (EFL) learning content. Using log data from Japanese junior high school students practicing on an LLM-driven grammar drilling application, the research analyzes how question modalities affect performance and whether theoretical localised CEFR difficulty tiers match empirical task difficulty. Results show a clear cognitive-load hierarchy: multiple-choice is lowest, cloze tasks hinder active recall most, and drag-and-drop causes the largest time penalties.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Related Work

Question Type, Cognitive Load, and CEFR Alignment: Evaluating LLM-Generated EFL Grammar Drill Exercises Steve WOOLLASTONa*, Brendan FLANAGANb, Yuko TOYOKAWAa, & Hiroaki OGATAa aKyoto University, Japan bRitsumeikan University, Japan *[s.m.woollaston@gmail.com](s.m.woollaston@gmail.com) Abstract: This study evaluates the pedagogical viability of LLM-generated English asa Foreign Language ( EFL) learning content. Utilising log data from Japanese junior high school students practicing on a grammar drilling application, we analysed how different question modalities impact student performance and whether theoretical localised CEFR difficulty tiers accurately predict empirical task difficulty. Results reveal a clear performance hierarchy: multiple-choice questions carried the lowest cognitive load, cloze tasks posed the greatest barrier to active recall, and drag-and-drop exercises incurred the heaviest time penalties. Furthermore, learner data validated the CEFR-J grammar framework, showing a steady decline inaccuracy and increased response times as proficiency levels advanced. These findings demonstrate that LLMs can successfully generate learning content, while highlighting the need for developers to strategically sequence question modalities to transition learners from passive recognition to active linguistic production. Keywords: Grammar drill, EFL, log data, second language learning, CALL CEFR-J 1. Introduction Second Language Acquisition (SLA) is being transformed by Computer-Assisted Language Learning (CALL) systems, which offer learners the benefit of unlimited, self-directed, and self-paced practice sessions anywhere at any time ( Levy & Stockwell, 2013) . However, developing these digital learning systems so they are effective and aligned to established frameworks can be difficult for educators and developers. Handcrafting learning content demands significant pedagogical expertise, making content creation both time-consuming and expensive. Consequently, traditional CALL software often relies on static, limited item banks; when students exhaust these resources, they face repetitive exercises that lead to boredom, diminished learning outcomes and motivation. To overcome these scalability limitations, Large Language Models ( LLMs) are increasingly used to automate content generation ( Park & Derakhshan, 2026) . LLMs can instantly generate vast, linguistically diverse, personalised, and contextually rich content. However, the rapid deployment of AI-generated content raises pedagogical concerns regarding proficiency alignment (Zhang & Huang, 2024) . It remains unclear whether LLM-generated material reliably matches theoretical proficiency frameworks, or if it inadvertently introduces unintended friction for language learners. Furthermore, additional empirical research is required to evaluate how this generated content performs within localised English as a Foreign Language ( EFL) contexts such as Japanese secondary education. Popular CALL apps (e.g. , Duolingo) rely heavily on closed-ended question formats (e.g. , multiple-choice ( MCQ) , drag-and-drop, and cloze tasks) to provide immediate feedback, yet few studies have compared the direct performative impacts of these distinct question modalities on actual students (Syahid, 2018) . To address these gaps, this study utilises Grammar Gretel, an EFL grammar drilling application driven by a dataset of over 55 ,000 LLM generated English-Japanese sentence pairs mapped to the CEFR-J framework. By analysing the log data of Japanese junior highschool students, we evaluate the real-world viability of LLM-generated grammar drill practice. Specifically, this study investigates the following research questions: 1. How do different question modalities ( MCQ, drag-and-drop, cloze) impact student accuracy and response times? 2. Do generated CEFR sentence levels accurately predict empirical task difficulty for learners? 2. Related Work Within SLA research, the deba

FAQs

How do different question modalities affect student performance in the study?

Multiple-choice questions produce the lowest cognitive load, cloze tasks create the greatest barrier to active recall, and drag-and-drop exercises lead to the heaviest time penalties.

Does the CEFR-J framework align with the real task difficulty observed in learner data?

Yes. Learner data supports the CEFR-J grammar framework, showing decreasing inaccuracy and increasing response times as proficiency levels advance.

What guidance does the study provide for using LLMs to generate grammar drill exercises?

LLMs can generate viable learning content, but developers should sequence question modalities strategically to move learners from passive recognition toward active linguistic production.

Jul 16, 2026

Upload by Emma Wilson