Communicating to the Public About Machine Scoring - What Works, What Doesn’t

Oct 9, 2026

Upload by Blitz

This paper documents six case studies examining how testing entities communicate the transition to machine scoring and where such communication fails. Evidence is drawn from four U.S. state programs, one Canadian province, and one country’s testing programs. Analysis across the six cases supports several tentative, practical recommendations for large-scale implementation, focusing on overcoming objections raised by earlier critiques of machine scoring.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Abstract

Communicating to the Public About Machine Scoring

Objections to machine scoring

Recent history and Hewlett Trials

West Virginia case study and implementation approach

| | | | | --- | --- | --- | | Communicating to the Public About Machine Scoring: What Works, What Doesn’t
Mark D. Shermis
University of Houston—Clear Lake
Susan Lottridge
AIR
Paper presented at the annual meetings ofthe National Council of Measurement in Education
Toronto, Canada, April 7, 2019 | | | | Abstract
This paper documents six case studies about how to, and how not to, communicate a testing entity’s transition to machine scoring. Data are drawn from four U.S. state-, one Canadian Province-, and one Country’s testing programs. Based on the analysis of the six cases, several tentative recommendations can be made.
Keywords: machine scoring, automated essay scoring, state testing programs | | | | Communicating to the Public About Machine Scoring: What Works, What Doesn’t
In his last major contribution to automated essay scoring, Page (2003) discussed three objections to machine scoring which he labeled as humanistic, defensive, and construct. The humanistic objection stipulates that writing is a unique human skill and cannot be evaluated by machine scoring algorithms. Defensive objections deal with concerns about “bad faith” or off-topic essays and scoring algorithm vulnerabilities to them. The construct argument suggests that what the human rater is evaluating is substantially different than what machine scoring algorithms used to predict scores for the text. Therefore, it may be possible for humans and machines to come to similar score assignments, but evaluate different things.
All of these objections, and some others, have made testing entities reluctant to engage in a wide-spread implementation of machine scoring for either short-form constructed or essay-length artifacts. This paper uses six case studies, three in which the implementation of machine scoring went well, and three where the implementation was blocked or substantially hindered. The purpose of the paper is to help identify steps that can be taken to address objections to implementing machine scoring on a large-scale basis.
Recent History
Perhaps more than anything else, the socalled Hewlett Trials (Shermis & Hamner, 2013) stimulated interest in the application of machine scoring for high-stakes assessment. In the essay scoring trial, eight commercial vendors and one university laboratory participated ina blind study which evaluated eight different essay prompts across multiple genre. For the first part of the trial, the competitors were given data sets containing essay text and at least two human ratings. In the second part of the trial, the competitors were simply given the text and asked to predict human-rater scores. The results | showed that, for the most part, machine score predictions were as reliable, and sometimes even more reliable, than human ratings. On that basis, several large testing entities began to investigate machine scoring for high-stakes essay writing. The notable exception was the state of West Virginia which began a version of machine scoring in 2005.
West Virginia
Rich, Schneider, & D’Brot (2013) documented one of the first positive experiences with machine scoring done on a statewide basis. In 2005 West Virginia was one the first states to implement automated essay scoring using software technology that combined both formative and summative assessment. From 2005-2014, WVDE used automated essay scoring as part of its formative and summative assessment programs. From 2015-2017, the state used hand-scoring only. In 2018 WVDE reinstated the use of automated essay scoring with a new vendor (American Institutes for Research, or AIR) . The state has been known for writing innovations since 1984 when it established a stand-alone assessment of writing.
West Virginia and the vendor at the
time, CTB, took a three-pronged approach in implementing the testing program. First, it hada large-scale state summative assessment that was scored using CTB’s Bookette scoring engine in grades 3 and 11

FAQs

What is the paper’s main focus regarding machine scoring?

The paper studies how to, and how not to, communicate a testing entity’s transition to machine scoring, using six case studies.

Where does the paper’s evidence come from?

Data come from testing programs in four U.S. states, one Canadian province, and one country.

What outcome does the paper report for the Hewlett Trials?

Machine score predictions were generally as reliable as human ratings and sometimes more reliable, encouraging further high-stakes adoption.

Oct 9, 2026

Upload by Blitz