The Challenges of Machine Learning for Trust and Safety - A Case Study on Misinformation Detection

Aug 4, 2026

Upload by Riley

This work examines the research-practice gap in applying machine learning to trust and safety, using misinformation detection as a case study. It surveys literature on automated misinformation detection across 248 well-cited papers, then evaluates subsets for dataset and code availability, design missteps, reproducibility, and generalizability. The analysis shows frequent dataset and method design errors and misalignment between reported detection tasks and real platform needs, limiting fully automated detection of human-generated misinformation. It also provides recommendations for ML evaluation in trust and safety and proposes directions for future research.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

AccessFree to download

Introduction

Research questions and research-practice gap

Literature review and replication study approach

[SoK] The Challenges of Machine Learning for Trust and Safety: A Case Study on Misinformation Detection Madelyne Xiao Princeton University Jonathan Mayer Princeton University arXiv :2308 . 12215v3 [ cs .LG] 19 Jun 2024 Abstract—We examine the disconnect between scholarship and practice in applying machine learning to trust and safety problems, using misinformation detection as a case study. We survey literature on automated detection of misinformation across a corpus of 248 well-cited papers in the field. We then examine subsets of papers for data and code availability, design missteps, reproducibility, and generalizability. Our paper corpus includes published work in security, natural language processing, and computational social science. Across these disparate disciplines, we identify common errors in dataset and method design. In general, detection tasks are often meaningfully distinct from the challenges that online services actually face. Datasets and model evaluation are often nonrepresentative of real-world contexts, and evaluation frequently is not independent of model training. We demonstrate the limitations of current detection methods in a series of three representative replication studies. Based on the results of these analyses and our literature survey, we conclude that the current state-of-the-art in fully-automated misinformation detection has limited efficacy in detecting human-generated misinformation. We offer recommendations for evaluating applications of machine learning to trust and safety problems and recommend future directions for research. 1. INTRODUCTION Online services face a daunting task: There is an unceasing deluge of user-generated content, on the order of hundreds of thousands of posts per minute on popular social media platforms [275] . Some of that content is false, hateful, harassing, extremist, or otherwise problematic. How can platforms reliably and proactively identify these “trust and safety” issues? Machine learning has proved an attractive approach in the academic literature, leading to large bodies of scholarship on misinformation detection [154], toxic speech classification [357], and other core trust and safety challenges (e.g., [358]) . The conceptual appeal of machine learning is that it could address the massive scale of user-generated content on large platforms and the capacity constraints of small platforms. Recent work claims impressive performance statistics: In the literature review that we conduct for this work, among publications that report performance metrics, about 70% of papers report over 80% accuracy on at least one detection task; some of these works report nearperfect performance [43], [44] . In the past year, news items from major tech companies have tempered these expectations. In June of 2023, OpenAI announced the deployment of a content moderation system based on GPT-4 to detect problematic content online; in the same press release, OpenAI’s head of safety systems admitted that human advisors would still need to “adjudicate borderline cases” [355] . In October of 2023, in a Bluesky post commenting on Twitter’s user-driven Community Notes program, Twitter’s former head of trust and safety stated that large-scale automated detection of misinformation remains a hard problem, and that no generalizable automated solutions are currently available [380] . These disclosures accord with our observation that, in practice, trust and safety functions at online services remain heavily manual: driven by user reports and carried out by human moderators. In this work, we investigate the disconnect between scholarship and practice in applications of machine learning to trust and safety problems. Our project is inspired by recent research that has identified shortcomings in machine learning applications for many problem domains, including information security [331], [336] . We use misinformation detection as a case study for trust and safety problems because

FAQs

What problem does the paper investigate in machine learning for trust and safety?

It studies the disconnect between how machine learning is researched and how trust and safety is handled in real online services, using misinformation detection as the case study.

How does the paper evaluate misinformation detection research?

It conducts a broad literature review of 248 papers, then examines subsets for data/code availability, design choices, reproducibility, and generalizability, followed by replication-style tests in representative studies.

What are the key findings about automated misinformation detection?

The paper finds that datasets and evaluation often do not reflect real-world contexts, evaluation may not be independent of training, and current fully automated methods have limited efficacy for detecting human-generated misinformation.

Aug 4, 2026

Upload by Riley