A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol

Jul 16, 2026

Upload by Ophelia

The study evaluates whether a previously validated protocol for classifying open-ended teaching-evaluation comments remains effective as representations evolve and whether it transfers across languages. It extends an earlier Spanish setup using documented annotation guidance, intra-annotator reliability, stratified cross-validation, and a held-out evaluation. Re-executions across three representation generations include frozen transformer embeddings and prompted large language models, with sentiment transferred to English using a 45,000-comment balanced corpus.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Background and prior protocol

Motivation: two objections

Paper approach and evaluation setup

A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol Esteban U. Vega Barajas Universidad de Guadalajara arXiv :2607 . 1 1873v 1 [ cs .CL] 13 Jul 2026 Abstract Institutions collect far more open-ended teaching-evaluation feedback than they read. A prior study introduced a validated protocol for classifying such comments by thematic category and sentiment, built from a documented annotation guide, an intra-annotator reliability measurement, stratified cross-validation, and a held-out evaluation on a Spanish institutional corpus with a frozen-encoder design. Two questions limit its reuse: whether a protocol fixed to 2019-era frozen embeddings stays competitive as representation methods advance, and whether it transfers to a second language. Were-run it on the original Spanish data across three representation generations, sparse lexical features, frozen transformer embeddings, and prompted large language models, and transfer its sentiment task to English with a balanced 45,000-comment corpus checked against an aspect-labeled education dataset. Treating paired comparisons as descriptive, we find the protocol durable: a 2026 frontier model posts the highest thematic F1 on the hardest Spanish task, yet shows no sentiment advantage over a cheap model and no descriptive separation from it on English, so model choice is a deployment decision, not a property of the method. Resumen Las instituciones recogen mucho más texto deevaluación docente del que llegan a leer. Unestudio previo presentó un protocolo validado para clasificar esos comentarios por categoríatemática y sentimiento, construido a partir de una guía de anotación documentada, una medición de fiabilidad intra-anotador, validacióncruzada estratificada y una evaluación sobre un conjunto reservado en un corpus institucional en español. Dos preguntas limitan su reutilización: si un protocolo fijado a representaciones congeladas de la generación de 2019 sigue siendo competitivo a medida que avanzan los métodos de representación, y si se transfiere a un segundo idioma. Lo reejecutamos sobre los datos originales en español a través de tres generaciones de representación, rasgos léxicos dispersos, representaciones congeladas de transformadores y modelos de lenguaje grandes, y transferimossu tarea de sentimiento al inglés con un corpus balanceado de 45,000 comentarios contrastado con un conjunto educativo etiquetado por aspectos. Tratando las comparaciones pareadas como descriptivas, encontramos que el protocolo es duradero: un modelo de frontera de 2026 obtiene el F1 temático más alto en la tareamás difícil en español, pero no muestra ventajaen sentimiento sobre un modelo económico ni separación descriptiva frente a él en inglés, demodo que la elección del modelo es una decisión de despliegue y no una propiedad del método. Keywords: teaching-evaluation feedback; text classification; sentiment analysis; cross-lingual transfer; large language models; educational NLP; reproducible benchmark Palabras clave: evaluación docente; clasificaciónde texto; análisis de sentimiento; transferencia translingüe; modelos de lenguaje grandes; PLNeducativo; benchmark reproducible 1 Introduction Open-ended comments on teaching-evaluation surveys are among the most informative and least-used data that educational institutions hold. A single mid-sized faculty generates tens of thousands of free-text comments per academic cycle; the volume routinely exceeds what staff can read, so institutions fall back on the accompanying numeric ratings and the qualitative signal is archived unread. Natural language processing (NLP) can recover that signal, classifying each comment by what it is about and by its affective stance. For an institution to acton the output, the analytical procedure has to be documented, reliability-checked, and validated. An accuracy number on a leaderboard does not by itself make the procedure something an insti

FAQs

What protocol is being tested in this study?

A validated procedure for classifying teaching-feedback comments into thematic categories and sentiment, based on a documented annotation guide, reliability checks, stratified cross-validation, and a held-out evaluation set.

How is durability evaluated as representation methods advance?

The same protocol is kept fixed while models and representations are varied across three generations, including frozen transformer embeddings and prompted large language models, to assess whether performance remains competitive.

Does the sentiment task transfer from Spanish to English?

Sentiment is transferred to English using a balanced 45,000-comment corpus validated against an independently annotated aspect-based education dataset, and the results show no sentiment advantage from a frontier model over a cheaper model on descriptive separation in English.

Jul 16, 2026

Upload by Ophelia