Anchor-Based Methods for Judgmentally Estimating Item Difficulty Parameters - LSAC Research Report Series 98-05 - October 2003

Sep 2, 2026

Upload by Anda

This study developed and field-tested anchor-based judgmental methods for enabling test specialists to estimate item difficulty statistics for Law School Admission Test (LSAT) items. Three related field tests involved six LSAT test specialists and one or more LSAT subtests. Findings focused on the item-estimation process, the time required for training and initial ratings, and the effect of structured item discussions, which often improved difficulty estimates.

LanguageEnglish

CategoryExam

FormatPDF

Pages49

Reading Time2 h 3 min

AccessFree to download

Executive Summary

Introduction

Overview of the Research Findings

Field Test One

Design

Results

Conclusions

Field Test Two

Design

Results

Conclusions

Field Test Three

Design

Results

Conclusions

References

Appendix A—Descriptions of the Item Judgmental Tasks in Field Test One

Appendix B—Field Test Two Evaluation Form

Appendix C—Field Test Three Materials

Appendix D—Sets of Item Characteristic Curves for the Three LSAT Subtests

ED 482926 AUTHOR TITLE INSTITUTION REPORT NO PUB DATE NOTE PUB TYPE EDRS PRICE DESCRIPTORS IDENTIFIERS ABSTRACT # DOCUMENT RESUME Hambleton,Ronald K.;Sireci,Stephen G.;Swaminathan,H.;Xing,Dehui;Rizavi,Saba Anchor-Based Methods for Judgmentally Estimating Item Difficulty Parameters.LSAC Research Report Series. Law School Admission Council,Newtown,PA. LSAC-CTR-98-05 2003-10-00 48p. Reports -Research(143) EDRS Price MF01/PC02 Plus Postage. *Adaptive Testing;College Entrance Examinations;*Computer Assisted Testing;*Difficulty Level;Estimation (Mathematics);Field Tests *Anchor Tests;*Item Parameters;Law School Admission Test The purposes of this research study were to develop and fieldtest anchor-based judgmental methods for enabling test specialists toestimate item difficulty statistics.The study consisted of three relatedfield tests.In each,researchers worked with six Law School Admission Test(LSAT)test specialists and one or more of the LSAT subtests.The three fieldtests produced a number of conclusions.A considerable amount was learnedabout the process of extracting test specialists'estimates of itemdifficulty.The ratings took considerably longer to obtain than had beenexpected.Training,initial ratings,and discussion took a considerableamount of time.Test specialists felt they could be trained to estimate itemdifficulty accurately and,to some extent,they demonstrated this.Averageerror in the estimates of item difficulty varied from about 11号to 138.Alsothe discussions were popular with the panelists,and almost always resultedin improved item difficulty estimates.By the end of the study,the twoexpected frameworks that developers thought they might provide test specialists,had merged to one.Test specialists seemed to benefit from thedescriptions of items located at three levels of difficulty and from information about the item statistics of many items.Four appendixes describetasks and contain the field test materials.(Contains 8 tables and 18references.)(SLD) | Reproductions supplied by EDRS are the best that can be made
from the original document. | | --- | ED 482926 PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL HASBEEN GRANTED BY J.VASELECK TO THE EDUCATIONAL RESOURCESINFORMATION CENTER(ERIC) U.S.DEPARTMENT OF EDUCATIONOflice of ECucational Research and improvement EDUCATIONAL RESOURCES INFORMATION CENTER(ERIC) received from the person oi orgenizationoriginating it 口Minor changes have been made toimprove reproduction quality. ·Points of view or opinions stated in thisdocument do not necessarily ropresentofficial OERI position or policy. TM035401 LSAC RESEARCH REPORT SERIES # ■Anchor-Based Methods for JudgmentallyEstimating Item Difficulty Parameters Ronald K.HambletonStephen G.Sireci H.SwaminathanDehui XingSaba Rizavi University of Massachusetts at Amherst ■Law School Admission CouncilComputerized Testing Report 98-05October 2003 LSAC RESEARCH REPORT SERIES ■Anchor-Based Methods for JudgmentallyEstimating Item Difficulty Parameters Ronald K.HambletonStephen G.Sireci H.SwaminathanDehui XingSaba Rizavi University of Massachusetts at Amherst ■Law School Admission CouncilComputerized Testing Report 98-05October 2003 ⑧ The Law School Admission Council is a nonprofit corporation that provides services to the legal educationcommunity.Its members are 201 law schools in the United States and Canada. CopyrightC 2003 by Law School Admission Council,Inc. All rights reserved.No part of this report may be reproduced or transmitted in any part or by any means,electronic or mechanical,including photocopying,recording,or by any information storage and retrievalsystem,without permission of the publisher.For information,write:Communications,Law SchoolAdmission Council,661 Penn Street,Box 40,Newtown,PA18940-0040. LSAT®and LSAC are registered marks of the Law School Admission Council,Inc. This study is published and distrib

FAQs

What was the purpose of the research study?

The study aimed to develop and field-test anchor-based judgmental methods that allow test specialists to estimate item difficulty statistics for LSAT items.

How were the field tests conducted?

Three related field tests were run, each involving researchers working with six LSAT test specialists and one or more LSAT subtests.

What did the researchers find about the estimation process?

Training, initial ratings, and discussion took considerable time, but specialists could be trained to estimate item difficulty accurately, and discussions typically improved the resulting estimates.

Sep 2, 2026

Upload by Anda