Optimal Testing With Easy or Difficult Items in Computerized Adaptive Testing - 2004-2

Sep 14, 2026

Computerized adaptive tests (CATs) use individualized item sequences that are measurement-optimal for each examinee, subject to practical constraints. This study shows that selecting maximum-information items from an item bank calibrated with the one- or two-parameter logistic model leads to an individual answering about half of the items correctly. Two selection procedures are evaluated to generate easier or more difficult tests for students. Results indicate probability-point selection works well only with the 1PL model, while a maximum-information-at-shifted-ability approach performs satisfactorily with both models.

LanguageEnglish

CategoryExam

FormatPDF

Pages32

Reading Time1 h 21 min

AccessFree to download

Abstract

1. Introduction

Computerized adaptive testing (CAT)

Practical conditions and content/exposure control

Psychometric optimality and perceived difficulty

Self adaptive testing (SAT) and related trade-offs

Measurement and Research Department Reports Optimal Testing With Easy or Difficult Items in Computerized Adaptive Testing Theo J.H.M. Eggen Angela J. Verschoor 2004-2 Measurement and Research Department Reports 2004-2 Optimal Testing with Easy or Difficult Items in Computerized Adaptive Testing Theo J.H.M . Eggen Angela J. Verschoor I \ Citogroep Arnhem, 2004 Ci􀀋 ::> !:F _;i􀀌p Po::,tbu5 1034 6801 MG 􀀚.r􀀛_hen1 \r'..c•nr 1 , :l ri+rnm ,. 8501 005 6159 . IIIIIII Ill IIIIII Ill II Ill II Ill111111111111111 This manuscript has been submitted for publication. No part of this manuscript may be copied or reproduced without permission. Abstract Computerized adaptivetests (CATs) are individualized tests which, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study it is shown that maximum information itemselection in CATsusing an item bank which is calibrated with the one-orthe two-parameter logisticmodel,results in each individual answering about 50% of the items correctly. Two item selection procedures giving easier (or more difficult) tests for students are presented and evaluated. Item selection on probability points of items yields good results only with the 1 pl model and not with the 2pl model. An alternative selection procedure, based on maximum information ata shifted ability level, gives satisfactory results with both models. 4 1. Introduction Computerized adaptive tests (CATs) are individualized tests that are administered in an automated environment. CATs are used for estimating the ability of a student or for making a decision on, for instance, the most appropriate training program for that student. It has been shown that, compared to traditional linear tests, CATs yield a considerable gain in efficiency. In the literature (see, e.g., Wainer, 2000 and Eggen & Straetmans, 2000), it has been reported that halving the average number of items needed is feasible, while at the same time the accuracy ofthe ability estimates or the decisions taken is maintained. CATs make use ofitem banks which are calibrated using item response theory (IRT) (Hambleton & Swaminathan, 1985) . The gain in CATs is realised by selecting, on the basis of the results on previously administered items, the most informative item from an available item bank. During testing, the optimal item is chosen after every item for every student and thus the optimal test is assembled and administered. CAT-tailored testing has a number of :frequently mentioned advantages: the gain in measurement efficiency goes hand in hand with the fact that each student is challenged at his or her own level because items which are too difficult or too easy fora given student will never be administered. Initially, the intended optimality, and, consequently, the item selection method, was based solely on a measurement theoretic or psychometric criterion. The criterion ofmaximum item information atthe current ability estimate isin common use(Van der Linden & Pashley, 2000). The increasing number of CAT applications has resulted in more consideration being given to content-based and practical requirements or conditions in itemselection algorithms. Applying content control (Kingsbury & Zara, 1991)and exposure control (Eggen, 2001) is routinely possible. In modem CATs, items that are psychometrically optimal are selected from an item bank which, to the degree possible, meets these practical conditions. The aim ofthe present study was to determine whether it is possible to take consideration of testees even more by not only considering the practical conditions, but also by relaxing the psychometrically optimal selection. Psychometrically optimal selection of items means that items will always be chosen for an individual student, which he or she, at his or her thus far known ability level, has a 50% probability of answering correctly. Thus, as a rule, students taking a CATwill

FAQs

What is the main goal of computerized adaptive testing (CAT) in this study?

To administer individualized tests in an automated way by selecting the most informative items based on previously answered responses, aiming to estimate ability efficiently and accurately.

How does maximum-information item selection perform in terms of expected correct answers?

With an item bank calibrated using the one- or two-parameter logistic model, each individual is expected to answer about 50% of the items correctly.

Why might making tests easier or harder be important for examinees?

CATs can be perceived as very difficult, potentially increasing test anxiety and reducing performance, especially in primary and secondary education where traditional tests target higher correct-answer probabilities.

Sep 14, 2026

Uploaded by Bill Black