[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117310-en":3,"doc-seo-117310-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},117310,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Establishing standards for human-annotated samples applied in supervised machine learning - Evidence from a Monte Carlo simulation","Automated content analyses have become a popular tool in communication science, yet it remains open whether long-established standards for manual coding are sufficient when human-annotated data train supervised machine learning models. A Monte Carlo simulation tests how coder errors—random, individual systematic, and joint systematic—and dataset curation strategies—one coder per document, majority rule, and full agreement—accumulate losses in prediction accuracy across a two-stage pipeline. Strong coder agreement before training is identified as a key quality criterion, with Krippendorff’s alpha of at least 0.8 recommended to achieve satisfying predictive results, while systematic errors must be avoided, and best training samples arise from one-coder or majority coding approaches.","Studies in Communication and Media  \nRESEARCH-IN-BRIEF  \nEstablishing standards for human-annotated samples applied in supervised machine learning – Evidence from a Monte Carlo  \nsimulation  \nManuelle Inhaltsanalysen für das maschinelle Lernen – Etablierung von Standards durch eine Monte-Carlo-Simulation  \nCorinna Oschatz, Marius Sältzer & Sebastian Stier  \nStudies in Communicatihp:/doMirdgi/,017. /.,2242037S-.208293–-43-02,9D, aOmI:170..17.27011, :-410:067-2023-4-289 289  \nOpen Access –  -[https://www.nomos-elibrary.de/agb](https://www.nomos-elibrary.de/agb)  \nCorinna Oschatz (Ass.-Prof. Dr.), University of Amsterdam, Amsterdam School of Communication Science (ASCoR), Postbus 15791, 1001 NG Amsterdam, The Netherlands. Contact: [c. m.oschatz@uva.nl](c. m.oschatz@uva.nl)  \nMarius Sältzer (Prof. Dr.), University of Oldenburg, Institute for Social Sciences, Department of Digital Social Science, Ammerländer Heerstraße 114-118, 26129 Oldenburg, Germa[ny. Contact: marius.saeltzer@uol.de](ny. Contact: marius.saeltzer@uol.de)  \nSebastian Stier (Prof. Dr.), GESIS – Leibniz Institute for the Social Sciences, Department Computational Social Science, Unter Sachsenhausen 6-8, 50667 Cologne, Germany / Professor for Computational Social Science, School of Social Sciences, University of Mannheim,  \nMannheim, Germany. Contact: [sebastian.stier@gesis.org](sebastian.stier@gesis.org)  \n290 SCM, 12. Jg., 4/2023  \n[https://doi.org/10.5771/2192-4007-2023-4-289](https://doi.org/10.5771/2192-4007-2023-4-289), am 17.10.2024, 14:41:06  \nOpen Access –  -[https://www.nomos-elibrary.de/agb](https://www.nomos-elibrary.de/agb)  \nRESEARCH-IN-BRIEF  \nEstablishing standards for human-annotated samples applied in supervised machine learning – Evidence from a Monte Carlo simulation  \nManuelle Inhaltsanalysen für das maschinelle Lernen – Etablierung von Standards durch eine Monte-Carlo-Simulation  \nCorinna Oschatz, Marius Sältzer & Sebastian Stier  \nAbstract: Automated content analyses have become a popular tool in communication science. While standard procedures for manual content analysis were established decades ago, it remains an open question whether these standards are sufficient for the use of human-annotated data to train supervised machine learning models. Scholars typically follow a two-stage procedure to obtain high prediction accuracy: manual content analysis followed by model training with human-annotated samples. We argue that a loss in prediction accuracy in supervised machine learning builds up over this two-stage procedure. In a Monte Carlo simulation, we tested (1) human coder errors (random, individual systematic, joint systematic) and (2) curation strategies for human-annotated datasets (one coder per document, majority rule, full agreement) as two sequential sources of accuracy loss of automated content analysis. Coder agreement prior to conducting manual content analysis remains an important quality criterion for automated content analyses. A Krippendorff’s alpha of at least 0.8 is desirable to achieve satisfying prediction results after machine learning. Systematic errors (individual and joint) must be avoided at all costs. The best training samples were obtained using one coder per document or the majority coding curation strategy. Ultimately, this paper can help researchers produce trustworthy predictions when combining manual coding and machine learning.  \nKeywords: Supervised machine learning, prediction accuracy, impact of coder errors, impact of curation strategies, Monte Carlo simulation.  \nZusammenfassung: Automatisierte Inhaltsanalysen sind ein häufig genutztes Instrument zur Beantwortung kommunikationswissenschaftlicher Forschungsfragen. Während Standards für die manuelle Inhaltsanalyse bereits vor Jahrzehnten etabliert wurden, bleibt zu klären, ob diese Standards für den Einsatz manuell generierter Daten im maschinellen Lernen ausreichen. Wissenschaftler folgen in der Regel einem zweistufigen Verfahren, um mit ihren Modellen quali","cbCaicff1Mxi86ij","https://ap.wps.com/l/cbCaicff1Mxi86ij","pdf",1157399,1,16,"English","en",105,"# Abstract\n# Keywords\n# 1. Introduction\n## Intercoder reliability and quality criteria","[{\"question\":\"What problem does the research address in supervised machine learning with human-annotated data?\",\"answer\":\"It evaluates whether standards from manual content analysis are sufficient when human-annotated samples are used to train supervised machine learning models, focusing on accuracy losses across the two-stage workflow.\"},{\"question\":\"Which two sources of accuracy loss are examined in the Monte Carlo simulation?\",\"answer\":\"The study examines (1) human coder errors—random, individual systematic, and joint systematic—and (2) curation strategies for human-annotated datasets such as one coder per document, majority rule, and full agreement.\"},{\"question\":\"What coder agreement level is recommended to achieve satisfactory predictive results?\",\"answer\":\"The research recommends a Krippendorff’s alpha of at least 0.8 before running automated content analysis to support satisfying prediction outcomes.\"}]",1785675110,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"establishing-standards-for-human-annotated-samples-applied-in-supervised-machine-learning-evidence-from-a-monte-carlo-simulation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/establishing-standards-for-human-annotated-samples-applied-in-supervised-machine-learning-evidence-from-a-monte-carlo-simulation/117310/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does the research address in supervised machine learning with human-annotated data?","Question",{"text":74,"@type":75},"It evaluates whether standards from manual content analysis are sufficient when human-annotated samples are used to train supervised machine learning models, focusing on accuracy losses across the two-stage workflow.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Which two sources of accuracy loss are examined in the Monte Carlo simulation?",{"text":79,"@type":75},"The study examines (1) human coder errors—random, individual systematic, and joint systematic—and (2) curation strategies for human-annotated datasets such as one coder per document, majority rule, and full agreement.",{"name":81,"@type":72,"acceptedAnswer":82},"What coder agreement level is recommended to achieve satisfactory predictive results?",{"text":83,"@type":75},"The research recommends a Krippendorff’s alpha of at least 0.8 before running automated content analysis to support satisfying prediction outcomes.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,118,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":28,"slug":117},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":119,"slug":120},30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]