Connecting Criterion Scores and Classroom Grading Contexts - A Systemic Functional Linguistic Model for Teaching and Assessing Causal Language

Oct 15, 2026

This study uses Mohan, Slater, Luo, and Jaipal’s developmental path of cause to analyze how AWE (Automatic Writing Evaluation) scores function in classroom contexts. The path supports validity arguments by organizing causal linguistic features hierarchically. AWE scores are compared with teacher-intuition ratings and assignment-rubric ratings, using both qualitative focus groups and quantitative analyses. Results indicate stronger alignment between path-trained raters’ grades and Criterion scores, suggesting teaching with the path can support AWE implementation and help students write cause-effect essays with greater appropriateness and sophistication.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

Development of academic language and causal explanations

Automated Writing Evaluation (AWE) systems and classroom concerns

Validating AWE scores and washback risks

Previous research on AWE score use

Psychometric approach

Naturalistic classroom-based approach

calico journal (online) issn 2056–9017 Article Connecting Criterion scores and Classroom Grading Contexts: A Systemic Functional Linguistic Model for Teaching and Assessing Causal Language Hong Ma and Tammy Slater Abstract This study utilized a theory proposed by Mohan, Slater, Luo, and Jaipal (2002) regarding the Developmental Path of Cause to investigate AWE score use in classroom contexts. This ‘path’ has the potential to support validity arguments because it suggests how causal linguistic features can be organized in hierarchical order. Utilization of this path enabled this study to investigate AWE scores by comparing them to ratings based on teachers’ intuitions as well as to scores based on assignment rubrics. Qualitative focus group data suggested that the path can help teachers articulate their intuitions. Quantitative results showed that the grades provided by raters trained to use the path tended to support AWE scores from Criterion, a webbased AWE system, more strongly than did rubric-generated grades. The findings from this study suggest that Criterion scores not only closely correlated with teachers’intuitions and with raters trained to use the path, but that the use of the path for teaching may support the implementation of AWE systems in classroom contexts, and would help students focus on the core of a cause-effect essay: appropriateness and sophistication of causal language. Keywords: AWE scores; causal discourse; systemic functional linguistics; the developmental path of cause Affiliation Iowa State University, Ames IA 50011, USA. [email: mahong84@hotmail.com](email: mahong84@hotmail.com) (corresponding author) calico journal vol 33.1 2016 1–18 doi : 10.1558/cj.v33i1.26562 ©2016, equinox publishing 2 Connecting Criterion Scores and Classroom Grading Contexts Introduction The development of academic language and the mastering of diverse subject matter are indispensable for academic and occupational success (Grimes & Warschauer, 2010) . In academic writing, causal explanations play a critical role since they occur across subject areas, from explaining causes and effects in science, to documenting historical conflicts, to arguing motives in literary characters. Even though various automated evaluation systems have been developed since the 1960s to compensate for the shortage of teacher time available for the individual responses needed to hone students’ writing skills (Burstein, Chodorow, & Leacock, 2003a; Dikli, 2006), few studies have investigated the extent to which AWE scores can accurately reflect the quality of causal explanations and whether automated evaluation systems can scaffold students’ development of causal explanations. Different from earlier counterparts such as Project Essay Grade, created in the 1960s exclusively for automated scoring, the most up-to-date Automatic Writing Evaluation (AWE) systems, Criterion by Educational Testing Service and My Access!, can provide both immediate scoring and formative feedback (Chen & Cheng, 2008), thus holding potential for classroom implementation. However, some writing instructors are dissuaded from using AWE systems because AWE scores have not consistently correlated well with instructor grades (Ebyary & Windeatt, 2010; Li, Link, Ma, Yang, & Hegelheimer, 2014; Wang & Brown, 2007). Re-examining this psychometric method of validating AWE scores, we argue that using rubric-based instructor grades as the benchmark may be inappropriate, since these rubrics can at times subvert instructors’ intuitive judgments by forcing them to focus on elements peripheral to the writing task (Slater, 1998; Mohan & Slater, 2004) . Slater (1998; see also Mohan & Slater, 2004), examining connections between a functional perspective on the assessment of causal discourse and raters’ intuitions of the discourse, suggested that the scores assigned can be valid and reliable if what is being assessed matches raters’ intuitions. Another reason to hesitat

FAQs

What theoretical model guides the study’s analysis of causal language?

The study uses Mohan, Slater, Luo, and Jaipal’s developmental path of cause to investigate AWE score use in classroom contexts.

How do the study’s comparisons evaluate AWE scores?

AWE scores are compared to ratings based on teachers’ intuitions and to scores generated from assignment rubrics, supported by focus group and quantitative results.

What do the quantitative results suggest about Criterion scores in classrooms?

Grades from raters trained to use the developmental path tended to align more strongly with AWE scores from Criterion than rubric-generated grades did.

Oct 15, 2026

Upload by Well Done