Improving the Production Efficiency and Well-formedness of Automatically-Generated Multiple-Choice Cloze Vocabulary Questions

Sep 29, 2026

Multiple-choice cloze (fill-in-the-blank) questions are widely used to test vocabulary knowledge and can be produced automatically for large-scale settings. This paper evaluates new features in Word Quiz Constructor (WQC), a Java application that generates items from the Academic Word List and online/offline resources. A production test and a well-formedness survey using Amazon Mechanical Turk show that readability and Google Books NGrams frequency features significantly improve item quality over prior WQC configurations. Items are produced faster and have shorter stem sentences without clear loss of well-formedness, with nearly 90% judged well-formed.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Introduction

Background

Automatic production of language testing questions

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020) , pages 7094–7101 Marseille, 11–16 May 2020 􀀍c European Language Resources Association (ELRA), licensed under CC-BY-NC Improving the Production Efficiency and Well-formedness of AutomaticallyGenerated Multiple-Choice Cloze Vocabulary Questions Ralph L. Rose Waseda University Faculty of Science and Engineering Tokyo, Japan [rose@waseda.jp](rose@waseda.jp) Abstract Multiple-choice cloze (fill-in-the-blank) questions are widely used in knowledge testing and are commonly used for testing vocabulary knowledge. Word Quiz Constructor (WQC) is a Java application that is designed to produce such test items automatically from the Academic Word List (Coxhead, 2000) and using various online and offline resources. The present work evaluates recently added features of WQC to see whether they improve the production quality and well-formedness of vocabulary quiz items over previously implemented features in WQC. Results of a production test and a well-formedness survey using Amazon Mechanical Turk show that newly-introduced features (Linsear Write readability formula and Google Books NGrams frequency list) significantly improve the production quality of items over previous features (Automated Readability Index and frequency list derived from the British Academic Written English corpus) . Items are produced faster and stem sentences are shorter in length without clear degradation in their wellformedness. Nearly 90% of such items are judged well-formed, surpassing the rate of manually-produced items. Keywords: multiple-choice cloze, vocabulary testing, automated test creation, readability formula, frequency list 1. Introduction A standard part of any language teaching curriculum is a focus on vocabulary study. Any program of vocabulary study consists of several components: some specification of the target lexical items, the order of their presentation to learners, presentation and study methods, and means of evaluation to gauge learners’ mastery of the vocabulary. Although these components are interdependent, the present paper is focused on the last of these: means of evaluation. There are many ways of evaluating learners’mastery of vocabulary items. Lexical variation in a writing sample could be measured. Alternatively, fill-inthe-blank (hereafter, cloze) items may be used to evaluate learners’ productive knowledge of best-fitting words. The most reliable approach with cloze items is likely freeresponse in which learners must think of the most suitable word(s) from their own lexical knowledge. But in largescale testing situations, this may be impractical because checking answers may be time-consuming as the checker must make decisions about unexpected responses that might in fact be suitable completions. To avoid these problems a common approach is to check vocabulary knowledge with multiple-choice cloze items. These can be checked very quickly and reliably by hand, or nearly immediately if the evaluation is performed through an online interface such as a web browser. But the burden with multiple-choice cloze questions is the time it takes to prepare them in advance. Fortunately, a number of automated applications exist that may produce such vocabulary questions. The present paper is a report on recent developments in one of these, Word Quiz Constructor (WQC: Rose, 2014a, 2014b, 2016) . Earlier work demonstrated that WQC can produce multiplechoice cloze vocabulary quiz items that are comparable in well-formedness to manually-produced items and at a modestly faster rate. The present report outlines improvements made to WQC to yield items that are produced faster and are more often well-formed and the optimal configuration of resources and settings to produce items. The remainder of the paper is organized as follows. The following section reviews earlier work on multiple-choice questions for vocabulary knowledge evaluation and then descr

FAQs

What is the main goal of this study?

To evaluate recently added features of Word Quiz Constructor (WQC) and determine whether they improve production quality and well-formedness of multiple-choice cloze vocabulary quiz items.

How does WQC generate multiple-choice cloze vocabulary items?

It uses a Java pipeline to automatically produce test items from the Academic Word List (Coxhead, 2000) and various online and offline resources, then constructs cloze stems with options including a key and distractors.

What features were newly introduced in WQC and what was their impact?

The newly introduced Linsear Write readability formula and Google Books NGrams frequency list significantly improve production quality over earlier features, and items are produced faster with shorter stem sentences without clear degradation in well-formedness.

Sep 29, 2026

Upload by anakgang17