ArguLens - An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation

Sep 15, 2026

Upload by Hazel

Automated essay scoring systems typically deliver a single holistic grade without interpretable evidence and often depend on closed APIs that create privacy, cost, and latency barriers. ArguLens provides an opensource, locally deployable framework that splits AES into a discourse-move classifier, a grade-independent LightGBM scorer built on 31 linguistic/discourse features, and a label-aware feedback generator served via vLLM. Experiments on PERSUADE 2.0 report 82.6% classifier accuracy and QWK improvements with gold discourse annotations; the feedback component uses a structured workflow while human evaluation is reserved for future work.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Related Work

AES Paradigms

Argument Structure and Feedback

Feedback Evaluation and Fairness

ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation Weiran Wang* Hongxiang Shi Huitao Tang Wenjuan Qin Fudan University [wrwang25@m.fudan.edu.cn](wrwang25@m.fudan.edu.cn) * Corresponding author arXiv : 2608 . 17356v 1 [cs .CL] 18 Aug 2026 Abstract Most automated essay scoring (AES) systems output a single holistic score without interpretable evidence and rely on closed APIs that introduce data privacy and cost barriers. We present ArguLens, an opensource, locally deployable system that decomposes AES into three decoupled components: a discourse-move classifier (Qwen2 .5- 7B-Instruct fine-tuned with LoRA on PERSUADE 2.0), a grade-independent LightGBM scorer over 31 linguistic and discourse features, and a label-aware feedback generator served through vLLM with a Qwen2.5-14BInstruct backbone. A Gradio web UI exposes pluggable inference backends and supports single-essay and batch scoring with downloadable per-essay breakdowns. On an essaydisjoint PERSUADE 2.0 test split, the logitprobe classifier achieves 82.6% accuracy and 0.727 macro-F1; under prompt-grouped 5-fold cross-validation the scorer reaches a mean QWK of 0 . 813 under an oracle discoursefeature protocol, and an ablation shows that adding gold discourse annotations yields an increment of +0 .055 QWK over the lexical+syntactic configuration (paired t-test, p = 0.010) . This is a component-level diagnostic rather than an end-to-end classifier-to-scorer result. The feedback generator ships with a structured evaluation protocol; its human-rater study is left to future work. The system is released under Apache 2.0 at [https://gith](https://gith)[ub.com/wwrwbs/AI_AWE](ub.com/wwrwbs/AI_AWE). 1 Introduction Automated essay scoring (AES) has evolved from linear regression on surface features (Page, 1966) to deep neural architectures (Taghipour and Ng, 2016 ; Mayfield and Black, 2020) and, most recently, to LLM-based holistic scoring (Beigman Klebanov and Madnani, 2020 ; Tate et al., 2024) . Despite this progress, a fundamental tension persists between interpretability and performance. Classic feature-based methods offer transparent scoring but cap ceiling accuracy; neural models improve accuracy at the cost of opacity, and LLM prompting requires closed APIs that restrict local deployment and data privacy. Three specific gaps remain for research and assistive deployment in educational contexts. First, end-to-end models output a single score without interpretable linguistic or rhetorical evidence, making it difficult for instructors to understand how a grade was reached. Second, reliance on commercial APIs introduces cost, latency, and data-privacy barriers that limit adoption in resource-constrained settings. Third, most AES systems stop at a holistic score; fine-grained, label-aware revision suggestions that could help learners improve their writing are rare. We address these gaps with ArguLens, a locally deployable, Apache 2.0-licensed system whose design and evaluation are presented in this paper. The system: • decomposes scoring into discourse-move classification, feature extraction, LightGBM scoring, and label-aware feedback; • supports local deployment via vLLM with tensor parallelism and a 4-bit HuggingFace fallback, while retaining an optional API backend; • ships reproducible artifacts including the scorer model, scaler, LoRA adapter configuration in the repository, and weights as a versioned Release asset with checksums; • exposes a config-driven stack with zero hardcoded paths, environment-variable precedence, and an offline test suite. We evaluate the classifier and scorer on the PERSUADE 2.0 corpus (Crossley et al., 2024) of US middle-school argumentative essays graded 1 to 6 . The discourse-move classifier is assessed through sentence-level accuracy and macro-F1; the scorer is evaluated through quadratic weighted kappa (QWK), accuracy, and macro-F1 under prompt-grouped cross-validati

FAQs

What problem does ArguLens target in automated essay scoring?

It targets the lack of interpretability in holistic AES scores, the deployment limits caused by closed commercial APIs, and the rarity of fine-grained, label-aware revision feedback for learners.

How does ArguLens decompose automated essay scoring?

It decomposes AES into three decoupled components: a discourse-move classifier, a grade-independent LightGBM scorer over 31 linguistic/discourse features, and a label-aware feedback generator served through vLLM.

What evidence is provided for ArguLens performance on PERSUADE 2.0?

On a PERSUADE 2.0 essay-disjoint test split, the logit-probe classifier achieves 82.6% accuracy and 0.727 macro-F1, and the scorer reaches a mean QWK around 0.813 under prompt-grouped cross-validation; an ablation shows gold discourse annotations increase QWK by +0.055.

Sep 15, 2026

Upload by Hazel