Measuring Machine Learning Harms from Stereotypes - Requires Understanding Who Is Harmed by Which Errors in What Ways

Aug 3, 2026

Upload by Violet

Machine learning research increasingly documents how models can propagate harmful stereotypes, yet limited work centers on the lived psychological experiences of people encountering such stereotypes. This study uses a gender-stereotype case in image search to examine reactions to model errors. Surveys show errors vary in whether they reflect stereotypes and in how harmful they are. Experiments randomly assign stereotype-reinforcing, stereotype-violating, or neutral errors, revealing experiential harm patterns that differ by gender and error type.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Limitations in existing stereotype-harm research

Stereotype reinforcement vs violation

Study overview

Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways Angelina Wang Stanford University, Cornell Tech USA [angelina.wang@cornell.edu](angelina.wang@cornell.edu) Xuechunzi Bai University of Chicago USA arXiv :2402 .04420v2 [ cs .CY] 24 May 2025 Solon Barocas Microsoft Research USA Su Lin Blodgett Microsoft Research USA Abstract Despite a proliferation of research on the ways that machine learning models can propagate harmful stereotypes, very little of this work is grounded in the psychological experiences of people exposed to such stereotypes. We use a case study of gender stereotypes in image search to examine how people react to machine learning errors. First, we use surveys to show that not all machine learning errors reflect stereotypes nor are equally harmful. Then, in experimental studies we randomly expose participants to stereotypereinforcing, -violating, and-neutral machine learning errors. We find stereotype-reinforcing errors induce more experiential harm, while having minimal impact on participants’ cognitive beliefs, attitudes, or behaviors. This experiential harm impacts participants who are women more than those who are men. However, certain stereotype-violating errors are more experientially harmful for men, potentially due to perceived threats to masculinity. We conclude by proposing a more nuanced perspective on the harms of machine learning errors—one that depends on who is experiencing what harm and why. CCS Concepts • Social and professional topics → Gender; • Computing methodologies → Object recognition. Keywords stereotypes, machine learning fairness, harms from biases ACM Reference Format: Angelina Wang, Xuechunzi Bai, Solon Barocas, and Su Lin Blodgett. 2025. Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways. In The 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25), June 23–26, 2025, Athens, Greece. ACM, New York, NY, USA, 17 pages. [https://doi.org/10.1145/3715275.3732046](https://doi.org/10.1145/3715275.3732046) 1 Introduction Over the past decade, researchers have demonstrated that machine learning models run the risk of learning stereotypical associations. Please use nonacm option or ACM Engage class to enable CC licenses This work is licensed under a Creative Commons Attribution 4 .0 International License. FAccT ’25, June 23–26, 2025, Athens, Greece © 2025 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-1482-5/2025/06 [https://doi.org/10.1145/3715275.3732046](https://doi.org/10.1145/3715275.3732046) For example, natural language processing models have been shown to associate women with homemakers and men with programmers [11], while computer vision models have been shown to associate women with shopping and men with driving [118] . These associations can cause machine learning models to make systematic errors, mistakenly reporting, for instance, that female doctors are nursesand that male nurses are doctors [94] . A rich literature has developed offering many more such examples, spurring calls to address the risk that machine learning models might make mistakes that perpetuate harmful stereotypes. Unfortunately, portions of this literature have suffered from three main limitations that impede effective intervention. First, while some research has asked crowd workers to annotate when errors invoke stereotypes or has drawn on pre-existing inventories of stereotypes [9, 11, 15, 16, 98, 106], machine learning researchers often rely on their own moral intuitions to determine which categories of associations to investigate (e.g., associations between gender and occupation) and to judge which specific associations (e.g., the association between women and nursing) are stereotypes. As a result, certain categories of associations that broader populations might perceive as stereotypical have n

FAQs

Why does the paper argue that existing stereotype-harm research is limited?

It highlights reliance on researchers’ moral intuitions to decide which associations are stereotypes and the tendency to treat all errors as equally harmful regardless of whether they reinforce or violate stereotypes.

How do the authors test how people react to different machine learning errors?

They run surveys and experiments in image search contexts, asking participants to evaluate whether errors relate to stereotypes and then randomly exposing participants to stereotype-reinforcing, stereotype-violating, or neutral errors.

What differences does the study find between stereotype-reinforcing and stereotype-violating errors?

Stereotype-reinforcing errors produce more experiential harm while having minimal effects on cognitive beliefs, attitudes, or behaviors; stereotype-violating errors show more experiential harm for men, potentially linked to perceived threats to masculinity.

Aug 3, 2026

Upload by Violet