Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn

Oct 15, 2026

This study investigated WriteToLearn’s application to Chinese undergraduate English majors’ essays, focusing on scoring consistency and the accuracy of error feedback. It involved 163 second-year English majors from Sichuan and 326 essays from two prompts, rated by four human raters and WriteToLearn. Many-facet Rasch measurement calibrated WriteToLearn’s scoring against human raters, and feedback accuracy was checked on 60 randomly selected essays. Results showed higher consistency but strict scoring, failures on seven essays, and error-feedback precision 49% and recall 18.7%, below a 90% precision reliability threshold.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Abstract

Introduction

AWE systems’ scoring ability

Table 1: Correspondence Indices between Human Rater Scores and Automated Scores

calico journal (online) issn 2056–9017 Article Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn Sha Liu and Antony John Kunnan Abstract This study investigated the application of WriteToLearn on Chinese undergraduate English majors’ essays in terms of its scoring ability and the accuracy of its error feedback. Participants were 163 second-year English majors from a university located in Sichuan province who wrote 326 essays from two writing prompts. Each paper was marked by four human raters as well as WriteToLearn. Many-facetRasch measurement (MFRM) was conducted to calibrate WriteToLearn’s rating performance in scoring the whole set of essays against those of four trained human raters. In addition, the accuracy of WriteToLearn’s feedback on 60 randomly selected essays was compared with the feedback provided by human raters. The two main findings related to scoring were that: (1) WriteToLearn was more consistent but highly stringent when compared to the four trained human raters in scoring essays; and (2) WriteToLearnfailed to score seven essays. In terms of error feedback, WriteToLearn had an overall precision and recall of 49% and 18.7% respectively. These figures did not meet the minimum threshold of 90% precision (set by Burstein, Chodorow, and Leacock, 2003) for it to be considered a reliable error detecting tool. Furthermore, it had difficulty in identifying errors made by Chinese undergraduate English majorsin the use of articles, prepositions, word choice and expression. Keywords: Accuracy of error feedback; automated writing evaluation; Chinese undergraduate English majors; scoring ability; WriteToLearn Affiliation School of Foreign Languages, China West Normal University, Nanchong City, Sichuan Province, 637000, China. email: [liusha.sally@yahoo.com](liusha.sally@yahoo.com) (corresponding author) calico journal vol 33.1 2016 71–91 doi : 10.1558/cj.v33i1.26380 ©2016, equinox publishing 72 Investigating the application of WriteToLearn Introduction The last few decades have seen the development of Automated Essay Soring (AES) systems for instructional applications including e-rater by Educational Testing Service (ETS), Intelligent Essay Assessor (IEA) by Pearson Education Inc., Intellimetric by Vantage Learning, Bookette by CTB/McGraw-Hill, and LightSIDE by Turnitin LLC. with corresponding instructional systems: Criterion, WriteToLearn, My Access, Writing Roadmap, and Turnitin. In addition to the primary function of essay scoring, these systems are trained to provide automated feedback on errors in students’ essays (i.e., error feedback) and offer suggestions on discourse elements such as organization and development of ideas (i.e., discourse feedback) . With this combination of automated scoring and feedback, these instructional systems are now referred to as automated writing evaluation (AWE) systems (McNamara, Crossley, Roscoe, Allen, & Dai, 2015). Particularly in the last decade, these AWE systems have been gradually marketed in classroom settings with English language learners (ELLs) although they were initially designed for native speaker-writers of English in the US (Warschauer & Ware, 2006). As there is insufficient research evidence to support the instructional application in the ELL setting, questions have been raised regarding AWE systems’ ability to score and detect errors in essays written by ELLs (Weigle, 2013a) . This study investigated the application of WriteToLearn, the AWE system that has now been marketed to schools and colleges in China. The aim of the study was to systematically evaluate WriteToLearn’s performance in scoring essays written by Chinese undergraduate English majors and its accuracy in providing error feedback when compared to trained human raters. AWE systems’ scoring ability An extensive review showed that the vast majority of studies of automated scoring have been con

FAQs

What does the study evaluate about WriteToLearn?

It evaluates WriteToLearn’s scoring ability and the accuracy of its error feedback for essays written by Chinese undergraduate English majors.

How were essays scored in the study?

Each essay was marked by four human raters and also by WriteToLearn, and Many-facet Rasch measurement was used to calibrate WriteToLearn against the human raters.

How accurate was WriteToLearn’s error feedback?

WriteToLearn’s overall precision and recall were 49% and 18.7%, respectively, which did not meet the 90% precision threshold for reliability in detecting errors.

Oct 15, 2026

Uploaded by Well Done