A Simulation Study to Explore Configuring the New SAT Critical Reading Section Without Analogy Items - Research Report No. 2004-2

Sep 14, 2026

Uploaded by Bintang

A simulation study examines how to configure the new SAT Critical Reading section without analogy items. The research develops and compares multiple prototypes across three phases, using prototype and item-pool construction, then evaluating item statistics and test statistics against psychometric and nonpsychometric criteria. The analysis reports item- and test-level performance indicators including equated deltas and r-biserial, along with CSEMs, SEMs, and reliability estimates to support decisions for the revised assessment design.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Abstract

Introduction

Phase 1

Method

Design of Prototypes

Item Pool Construction

Assembly of Simulated Forms

Analyses of Prototypes: Item Statistics

Analyses of Prototypes: Test Statistics

Establishing Psychometric Criteria

Establishing Nonpsychometric Criteria

Results

Discussion

Phase 2

Method

Results

Revised Delta Distribution

Item Statistics

Test Statistics

Discussion

Phase 3

Method

Estimation of the Length and Delta Distribution of Three Hypothetical Tests

Computation of CSEM for Three Hypothetical Tests

Results

General Discussion

References

Research Report No . 2004-2 A Simulation Study to Explore Conﬁguring the New SAT ® Critical Reading Section Without Analogy Items Jinghua Liu Miriam Feigenbaum Linda Cook College Board Research Report No . 2004-2 ETS RR-04-01 A Simulation Study to Explore Configuring the New SAT ® Critical Reading Section Without Analogy Items Jinghua Liu, Miriam Feigenbaum, and Linda Cook College Entrance Examination Board, New York, 2004 Jinghua Liu is measurement statistician at Educational Testing Service. Miriam Feigenbaum is principal statistical associate level II at Educational Testing Service. Linda Cook is principal research scientist level II at Educational Testing Service. Researchers are encouraged to freely express their professional judgment. Therefore, points of view or opinions stated in College Board Reports do not necessarily represent official College Board position or policy. The College Board: Expanding College Opportunity The College Board is a not-for-profit membership association whose mission is to connect students to college success and opportunity. Founded in 1900, the association is composed of more than 4,500 schools, colleges, universities, and other educational organizations. Each year, the College Board serves over three million students and their parents, 23,000 high schools, and 3,500 colleges through major programs and services in college admissions, guidance, assessment, financial aid, enrollment, and teaching and learning. Among its best-known programs are the SAT®, the PSAT/NMSQT®, and the Advanced Placement Program® (AP® ) . The College Board is committed to the principles of excellence and equity, and that commitment is embodied in all of its programs, services, activities, and concerns. For further information, [visit www.collegeboard.com](visit www.collegeboard.com). Additional copies of this report (item \#030481025) maybe obtained from College Board Publications, Box 886, New York, NY 10101-0886, 800 323-7155. The price is $15. Please include $4 for postage and handling. Copyright © 2004 by College Entrance Examination Board. All rights reserved. College Board, SAT, and the acorn logo are registered trademarks of the College Entrance Examination Board. SAT Reasoning Test is a trademark owned by the college Entrance Examination Board. PSAT/NMSQT is a registered trademark of the College Entrance Examination Board and National Merit Scholarship Corporation. Other products and services may be trademarks of their respective owners. [Visit College Board on the Web: www.collegeboard.com](Visit College Board on the Web: www.collegeboard.com). Printed in the United States of America. Contents Abstract...............................................................1 Introduction ........................................................1 Phase 1 ................................................................ 2 Method ........................................................2 Design of Prototypes ...............................2 Item Pool Construction ...........................2 Assembly of Simulated Forms..................2 Analyses of Prototypes: Item Statistics ....4 Analyses of Prototypes: Test Statistics .....4 Establishing Psychometric Criteria ..........5 Establishing Nonpsychometric Criteria ...5 Results .........................................................5 Item Statistics—Equated Deltasand r-biserial........................................5 Test Statistics—CSEMs, SEMs, and Reliabilities ..........................................5 Discussion .................................................10 Phase 2 .............................................................. 10 Method ......................................................10 Results .......................................................11 Revised Delta Distribution ....................11 Item Statistics ........................................11 Test Statistics .....

FAQs

What is the main goal of the simulation study?

To explore how configuring the new SAT Critical Reading section can be done without analogy items while maintaining appropriate performance characteristics.

How many phases does the research include and what do they cover?

The study is organized into three phases, each following a method and analyzing results using item statistics and test statistics, with discussion of findings.

Which types of metrics are used to evaluate prototypes and revised designs?

The report uses item statistics (including equated deltas and r-biserial), test statistics (including CSEMs, SEMs, and reliabilities), and applies psychometric and nonpsychometric criteria.

Sep 14, 2026

Uploaded by Bintang