Establishing standards for human-annotated samples applied in supervised machine learning - Evidence from a Monte Carlo simulation

Aug 2, 2026

Upload by Oliver

Automated content analyses have become a popular tool in communication science, yet it remains open whether long-established standards for manual coding are sufficient when human-annotated data train supervised machine learning models. A Monte Carlo simulation tests how coder errors—random, individual systematic, and joint systematic—and dataset curation strategies—one coder per document, majority rule, and full agreement—accumulate losses in prediction accuracy across a two-stage pipeline. Strong coder agreement before training is identified as a key quality criterion, with Krippendorff’s alpha of at least 0.8 recommended to achieve satisfying predictive results, while systematic errors must be avoided, and best training samples arise from one-coder or majority coding approaches.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Keywords

1. Introduction

Intercoder reliability and quality criteria

Studies in Communication and Media RESEARCH-IN-BRIEF Establishing standards for human-annotated samples applied in supervised machine learning – Evidence from a Monte Carlo simulation Manuelle Inhaltsanalysen für das maschinelle Lernen – Etablierung von Standards durch eine Monte-Carlo-Simulation Corinna Oschatz, Marius Sältzer & Sebastian Stier Studies in Communicatihp:/doMirdgi/,017. /.,2242037S-.208293–-43-02,9D, aOmI:170..17.27011, :-410:067-2023-4-289 289 Open Access – -[https://www.nomos-elibrary.de/agb](https://www.nomos-elibrary.de/agb) Corinna Oschatz (Ass.-Prof. Dr.), University of Amsterdam, Amsterdam School of Communication Science (ASCoR), Postbus 15791, 1001 NG Amsterdam, The Netherlands. Contact: [c. m.oschatz@uva.nl](c. m.oschatz@uva.nl) Marius Sältzer (Prof. Dr.), University of Oldenburg, Institute for Social Sciences, Department of Digital Social Science, Ammerländer Heerstraße 114-118, 26129 Oldenburg, Germa[ny. Contact: marius.saeltzer@uol.de](ny. Contact: marius.saeltzer@uol.de) Sebastian Stier (Prof. Dr.), GESIS – Leibniz Institute for the Social Sciences, Department Computational Social Science, Unter Sachsenhausen 6-8, 50667 Cologne, Germany / Professor for Computational Social Science, School of Social Sciences, University of Mannheim, Mannheim, Germany. Contact: [sebastian.stier@gesis.org](sebastian.stier@gesis.org) 290 SCM, 12. Jg., 4/2023 [https://doi.org/10.5771/2192-4007-2023-4-289](https://doi.org/10.5771/2192-4007-2023-4-289), am 17.10.2024, 14:41:06 Open Access – -[https://www.nomos-elibrary.de/agb](https://www.nomos-elibrary.de/agb) RESEARCH-IN-BRIEF Establishing standards for human-annotated samples applied in supervised machine learning – Evidence from a Monte Carlo simulation Manuelle Inhaltsanalysen für das maschinelle Lernen – Etablierung von Standards durch eine Monte-Carlo-Simulation Corinna Oschatz, Marius Sältzer & Sebastian Stier Abstract: Automated content analyses have become a popular tool in communication science. While standard procedures for manual content analysis were established decades ago, it remains an open question whether these standards are sufficient for the use of human-annotated data to train supervised machine learning models. Scholars typically follow a two-stage procedure to obtain high prediction accuracy: manual content analysis followed by model training with human-annotated samples. We argue that a loss in prediction accuracy in supervised machine learning builds up over this two-stage procedure. In a Monte Carlo simulation, we tested (1) human coder errors (random, individual systematic, joint systematic) and (2) curation strategies for human-annotated datasets (one coder per document, majority rule, full agreement) as two sequential sources of accuracy loss of automated content analysis. Coder agreement prior to conducting manual content analysis remains an important quality criterion for automated content analyses. A Krippendorff’s alpha of at least 0.8 is desirable to achieve satisfying prediction results after machine learning. Systematic errors (individual and joint) must be avoided at all costs. The best training samples were obtained using one coder per document or the majority coding curation strategy. Ultimately, this paper can help researchers produce trustworthy predictions when combining manual coding and machine learning. Keywords: Supervised machine learning, prediction accuracy, impact of coder errors, impact of curation strategies, Monte Carlo simulation. Zusammenfassung: Automatisierte Inhaltsanalysen sind ein häufig genutztes Instrument zur Beantwortung kommunikationswissenschaftlicher Forschungsfragen. Während Standards für die manuelle Inhaltsanalyse bereits vor Jahrzehnten etabliert wurden, bleibt zu klären, ob diese Standards für den Einsatz manuell generierter Daten im maschinellen Lernen ausreichen. Wissenschaftler folgen in der Regel einem zweistufigen Verfahren, um mit ihren Modellen quali

FAQs

What problem does the research address in supervised machine learning with human-annotated data?

It evaluates whether standards from manual content analysis are sufficient when human-annotated samples are used to train supervised machine learning models, focusing on accuracy losses across the two-stage workflow.

Which two sources of accuracy loss are examined in the Monte Carlo simulation?

The study examines (1) human coder errors—random, individual systematic, and joint systematic—and (2) curation strategies for human-annotated datasets such as one coder per document, majority rule, and full agreement.

What coder agreement level is recommended to achieve satisfactory predictive results?

The research recommends a Krippendorff’s alpha of at least 0.8 before running automated content analysis to support satisfying prediction outcomes.

Aug 2, 2026

Upload by Oliver