ACT Working Paper 2015-05 - Spring 2014 ACT Test Mode Comparability Study

Oct 9, 2026

ACT working paper 2015-05 reports research leading to score comparability between online and paper administrations of the ACT. The executive summary covers a fall 2013 timing study and a spring 2014 mode comparability study, emphasizing how results guide operational decisions. Analyses use descriptive and inferential methods on student item and test scores, omit rates, latency, and survey responses. Findings indicate that additional time improves comparability for Reading and Science online, followed by random-groups equating to adjust for any mode-related differences.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages41

Reading Time1 h 43 min

AccessFree to download

Executive Summary

Fall 2013 Timing Study

Data and Design

Results

Spring 2014 Mode Comparability Study

Design and Data

Results

ACT WORKING PAPER 2015-05 Spring 2014 ACT® Test Mode Comparability Study Dongmei Li, PhD Qing Yi, PhD Deborah Harris, PhD April, 2015 ACT Working Paper Series ACT working papers document preliminary research. The papers are intended to promote discussion and feedback before formal publication. The research does not necessarily reflect the views of ACT. | Dongmei Li, is a Principle Psychometrician at ACT providing psychometric support for the ACT® test with specialties in educational measurement theories and practices, test equating and scaling, and growth modeling.
Qing Yi, is a Senior Psychometrician at ACT specializing in computerized adaptive testing, test equating, and educational measurement theories.
Deborah Harris, is a Chief Research Scientist at ACT, specializing in equating, linking, and the comparability of reported scores. | | --- | | This report is a summary of work from many. The authors are in debt to David Woodruff, Tony Thompson, Yu Fang, Yang Lu, Wei Tao, Zhongmin Cui, Andrew Mroch, J.P. Kim, Tianli Li, Lu Wang, Qing Xie, and many others in the Measurement Research Department who participated in various components of the study. Their unique contributions and diligent work made this study and report possible. The authors would also like to thank all other ACT departments that were involved in the study and Creative Services for their editorial work on the report. | Executive Summary In preparation for online administration of the ACT®, ACT has conducted studies to ensure the comparability of scores between online and paper administrations, including a timing study in fall 2013 and a mode comparability study in spring 2014. This report presents major findings from these two studies, focusing on the spring 2014 mode comparability study. Fall 2013 Timing Study Standard paper administration of the ACT allows 45, 60, 35, and 35 minutes for the English, Mathematics, Reading, and Science Tests, respectively. The purpose of the timing study was to evaluate whether online administration of the ACT would require different time limits than the paper administration. Data and Design The four tests were administered online to approximately 3,000 examinees, with each examinee responding to one test. Students taking each subject were randomly assigned to take the test under one of the three timing conditions: the current paper time limit, the current time limit plus five minutes, and the current time limit plus ten minutes. At the end of the test, the students were also given a survey with questions regarding their testing experience, including whether or not they felt they had enough time to finish the test. The three testing time limits and the four tests produced 12 different combinations with about 250 examinees in each condition. Results Student item and test level scores, item omit rates, item and test latency information, and student survey results were analyzed using a variety of methods, both descriptive and inferential. Results suggested that online scores on Reading and Science would be more likely to be comparable to paper administration scores with an increase in testing time. Given the potential confounding of motivation and familiarity with the online testing format in the timing study, the final decision was to tentatively increase testing time for the Reading and Science tests by five minutes and continue to evaluate the timing issue in the spring 2014 mode comparability study. Spring 2014 Mode Comparability Study To increase student motivation and to learn about administration issues, ACT conducted the mode comparability study in an operational testing environment where participating students received college-reportable scores. Therefore, it was imperative that scores reported across modes be comparable. To ensure this was the case, even if differences were found across mode in terms of item or raw score differences, a random groups design was implemented, allowing equa

FAQs

What was the purpose of the fall 2013 timing study?

Evaluate whether online administration required different time limits than paper, by testing three timing conditions across four subjects and collecting survey feedback about time sufficiency.

How did the study ensure comparability of scores across modes in spring 2014?

It used a random equivalent groups design and equating methodologies to adjust for differences, even when item or raw score differences were observed between online and paper formats.

What did the results suggest about online testing time for Reading and Science?

Online scores for Reading and Science were more likely to be comparable to paper scores when testing time was increased, prompting a tentative five-minute increase and continued evaluation in the spring 2014 study.

Oct 9, 2026

Upload by Mary Man