When Debiasing Backfires Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation

Jul 16, 2026

Preprocessing-based stereotype mitigation methods in NLP, such as training on debiased corpora or editing datasets before/after learning, often reduce measured bias for targeted groups. Yet this work finds unintended shifts: stereotyping or counter-stereotyping can rise relative to neutral baselines for other demographics, even across unrelated categories. Experiments cover encoder-only and decoder-only models, multiple preprocessing strategies, and varied data scales on Wikipedia.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Introduction

Background

When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation Yahan Zheng Dartmouth College [yahan.zheng.gr@dartmouth.edu](yahan.zheng.gr@dartmouth.edu) Soroush Vosoughi Dartmouth College [soroush.vosoughi@dartmouth.edu](soroush.vosoughi@dartmouth.edu) John J. Guerrerio Dartmouth College [john.j.guerrerio.26@dartmouth.edu](john.j.guerrerio.26@dartmouth.edu) Weicheng Ma Oakland University [weichengma@oakland.edu](weichengma@oakland.edu) arXiv :2607 .07937v 1 [ cs .CL] 8 Jul 2026 Abstract Preprocessing-based methods for stereotype mitigation, such as pre-/post-training on debiased corpora, are widely used in NLP. While these approaches reduce measurable stereotypes for targeted groups, we find they often induce unintended shifts—side effects, where stereotyping or counter-stereotyping can increase relative to neutral baselines for other demographics, including across unrelated demographic categories. We demonstrate these side effects across two model families (encoderonly and decoder-only), multiple preprocessing strategies (removing stereotypical sentences, removing group mentions, and swapping group references), and both pre-and post-training at different data scales on Wikipedia. Standard benchmarks frequently miss these shifts. Using attention-rollout analysis, we observe that such side effects are not accompanied by large changes in attention flow, complicating mechanistic explanations. We discuss implications for evaluation, provide actionable diagnostics, and argue for side-effect-aware, transparent mitigation practices. 1 Introduction Pre-trained language models (PLMs) encode and propagate social stereotypes (Bolukbasi et al., 2016 ; Caliskan et al., 2017 ; Zhao et al., 2019), raising concerns about safe and responsible deployment. Among mitigation strategies, preprocessingbased methods seek to modify training data, for instance, by removing or altering stereotypical content, because they are simple to apply and impose no inference-time cost (Gallegos et al., 2024) . Intuitively, such interventions should hypothetically prevent models from learning unwanted associations by isolating them from stereotypical patterns. For consistency, we use the term stereotype to refer to any unwanted social bias encoded by language models from their training data, and mitigation or debiasing to refer to interventions aimed at reducing these encoded stereotypes. Despite the intuitive appeal and partial effectiveness of preprocessing-based methods, to date, no PLM has achieved complete freedom from stereotypes, raising questions about their true efficacy. More importantly, it is unclear whether datalevel mitigation eliminates harmful associations or merely redistributes them. We revisit this question empirically by curating debiased Wikipedia corpora targeting six demographic groups spanning three categories (gender, race, religion) . We use these corpora for both preand post-training of two PLMs (encoder-only TinyBERT (Jiao et al., 2020) and decoder-only GPT-2 (Radford et al., 2019)) and then measure changes in stereotype expression toward all groups. We report three findings: 1. Unintended shifts within and across categories. While stereotyping toward the group targeted for debiasing decreases, we frequently observe increased stereotyping or counterstereotyping for other groups (including across bias categories) . We define this phenomenon as a side effect: a case where a mitigation method decreases stereotype measures for the target group but simultaneously induces undesired changes for one or more non-target groups. Notably, these shifts are often asymmetric and hard to anticipate. We quantify trends on StereoSet and CrowS-Pairs (Nadeem et al., 2021 ; Nangia et al., 2020) . These patterns cannot be explained solely by changes in the distribution of stereotypical/anti-stereotypical training examples. 2. Robustness across settings. Side effects appear under three

FAQs

What phenomenon do the authors report about preprocessing-based debiasing?

They report unintended side effects: mitigation can reduce stereotyping for the target group while increasing stereotyping or counter-stereotyping for non-target groups, sometimes across different bias categories.

How do they test side effects across models and training setups?

They curate debiased Wikipedia corpora for six demographic groups and apply both pre-training and post-training to encoder-only TinyBERT and decoder-only GPT-2, using multiple preprocessing strategies and different training data scales.

Why do the authors argue existing benchmarks and analysis may miss these effects?

Standard benchmarks often fail to surface the shifts, and attention-rollout analysis shows only small changes in attention flow, making attention-based mechanistic explanations insufficient.

Jul 16, 2026

Upload by Genevieve