Classification of reflective writing - A comparative analysis with shallow machine learning and pre-trained language models

Aug 3, 2026

Upload by Levi

Reflective practice is crucial in higher education and teacher education, but developing students’ reflective skills remains difficult. Building on sentence-level classification, this study targets document-level understanding of reflective writing by applying shallow machine learning and pre-trained language models, including BERT, RoBERTa, BigBird, and Longformer. A dataset of 1,043 reflective writings from a German teacher education program was used. Results show BigBird and Longformer outperform BERT and RoBERTa, and shallow models remain below 60% accuracy. Findings support improved automated feedback for teacher education.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

AccessFree to download

Abstract

Introduction

Motivation for AI-assisted feedback

Challenges in dataset quality and sentence-level modeling

Study aims and document-level methodology

Models and data overview

Classification of reflective writing: A comparative analysis with shallow machine learning and pre-trained language models Chengming Zhang1 · Florian Hofmann1 · Lea Plößl1 · Michaela Gläser‑Zikuda1 Received: 15 November 2023 / Accepted: 14 April 2024 / Published online: 2 May 2024 © The Author(s) 2024 Abstract Reflective practice holds critical importance, for example, in higher education and teacher education, yet promoting students’ reflective skills has been a persistent challenge. The emergence of revolutionary artificial intelligence technologies, notably in machine learning and large language models, heralds potential breakthroughsin this domain. The current research on analyzing reflective writing hinges on sentence-level classification. Such an approach, however, may fall short of providing a holistic grasp of written reflection. Therefore, this study employs shallow machine learning algorithms and pre-trained language models, namely BERT, RoBERTa, BigBird, and Longformer, with the intention of enhancing the document-level classification accuracy of reflective writings. A dataset of 1,043 reflective writings was collected in a teacher education program at a German university (M = 251.38 words, SD = 143.08 words) . Our findings indicated that BigBird and Longformer models significantly outperformed BERT and RoBERTa, achieving classification accuracies of 76.26% and 77.22%, respectively, with less than 60% accuracy observed in shallow machine learning models. The outcomes of this study contribute to refining document-level classification of reflective writings and have implications for augmenting automated feedback mechanisms in teacher education. Keywords Reflective writing · Pre-trained language model · Shallow machine learning · AI feedback · Teacher education 1 Introduction The dawn of the Artificial Intelligence (AI) era has brought about transformative educational changes. From profiling and prediction to automated assessment and personalized learning, the increasing use of AI applications is evidence of its burgeoning influence (Zawacki-Richter et al., 2019 ; Zhai et al., 2021) . One particularly Extended author information available on the last page of the article impactful instance of AI’s utility is the deployment of Large Language Models (LLMs) like ChatGPT. This powerful tool can support students in problem-solving, personalized guidance, feedback provision, and other areas. Interestingly, the benefits of AI feedback have been empirically quantified. For instance, a meta-analysis by Cai et al. (2023) revealed a moderate positive impact of feedback on academic achievement in technology-rich learning environments compared to traditional feedback-absent environments. These findings hint at AI’s potential to address longstanding educational challenges, especially those related to providing timely, personalized feedback (Russell & Korthagen, 2013) . In light of these developments, the fusion of machine learning (ML) and natural language processing (NLP) technologies may present an efficient way to meet students’ diverse learning needs. In the rapidly evolving educational technology landscape, a significant breakthrough has occurred in using AI for assessment, particularly concerning reflective writing. Reflection forms an essential bridge between theoretical knowledge and practical application (Korthagen & Vasalos, 2005) . Despite the deployment of various approaches aiming to bolster reflective practice, the endeavor has encountered constraints, primarily due to the multifaceted nature of reflective writing, which poses a challenge for evaluation (Körkkö et al., 2016 ; Poldner et al., 2014 ; Ullmann, 2019) . Recent researches indicate that shallow ML and pre-trained language models are particularly effective in assessing reflective writing, marking a significant advancement in this field (Nehyba & Štefánik, 2023 ; Solopova et al., 2023 ; Wulff et al., 2023) . Moreover, AI’s role in assessing s

FAQs

Why is reflective writing difficult to assess automatically?

Reflective writing contains intrinsic narrative coherence, so sentence-level classification can miss context and important information in the broader text.

What models are used to classify reflective writing in this study?

The study applies shallow machine learning algorithms and pre-trained language models including BERT, RoBERTa, BigBird, and Longformer, using a document-level classification approach.

Which models performed best and what accuracy ranges were observed?

BigBird and Longformer significantly outperformed BERT and RoBERTa, achieving about 76.26% and 77.22% accuracy respectively, while shallow machine learning models stayed below 60% accuracy.

Aug 3, 2026

Upload by Levi