Mitigating Label Bias in Machine Learning - Fairness through Confident Learning - Paper

Aug 5, 2026

Upload by Levi

Discrimination arises when potentially biased agents overwrite underlying unbiased labels, producing biased datasets that unfairly harm specific groups and lead classifiers to reproduce these biases. This paper shows that, even with access only to biased labels, bias can be reduced by filtering the fairest instances within confident learning. Low self-confidence often signals label errors, but not always, especially for underrepresented groups, motivating a truncation of confidence scores, an expanded confidence interval, and co-teaching selection.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Abstract

Introduction

Fairness challenges in decision-making

Prior approaches to training fair models

Directly addressing biased data vs. model changes

Data selection under confident learning

Mitigating Label Bias in Machine Learning: Fairness through Confident Learning Yixuan Zhang 1 , Boyu Li2 , Zenan Ling3 , Feng Zhou4 * 1 China-Austria Belt and Road Joint Laboratory on AI and AM, Hangzhou Dianzi University, China 2Data Science Institute, University of Technology Sydney, Australia 3 School of Electronic Information and Communications, Huazhong University of Science and Technology, China 4 Center for Applied Statistics and School of Statistics, Renmin University of China, China [yixuan.zhang@hdu.edu.cn](yixuan.zhang@hdu.edu.cn), [boyu.li@student.uts.edu.au](boyu.li@student.uts.edu.au), [lingzenan@hust.edu.cn](lingzenan@hust.edu.cn), [feng.zhou@ruc.edu.cn](feng.zhou@ruc.edu.cn) Abstract Discrimination can occur when the underlying unbiased labels are overwritten by an agent with potential bias, resulting in biased datasets that unfairly harm specific groups and cause classifiers to inherit these biases. In this paper, we demonstrate that despite only having access to the biased labels, it is possible to eliminate bias by filtering the fairest instances within the framework of confident learning. In the context of confident learning, low self-confidence usually indicates potential label errors; however, this is not always the case. Instances, particularly those from underrepresented groups, might exhibit low confidence scores for reasons other than labeling errors. To address this limitation, our approach employs truncation of the confidence score and extends the confidence interval of the probabilistic threshold. Additionally, we incorporate with co-teaching paradigm for providing amore robust and reliable selection of fair instances and effectively mitigating the adverse effects of biased labels. Through extensive experimentation and evaluation of various datasets, we demonstrate the efficacy of our approach in promoting fairness and reducing the impact of label bias in machine learning models. Introduction In recent decades, we have observed a shift towards an AIdriven society, where machine learning techniques have profoundly affected various aspects of our lives, including finance (Khandani, Kim, and Lo 2010), recruitment (Faliagka et al. 2012) and law (Dressel and Farid 2018) . When algorithmic decisions have a significant impact on our lives, it is crucial for decision-makers or regulators to have confidence in the algorithm’s performance. While deep neural networks possess the capability to learn complex patterns from input data, they encounter a fundamental challenge in the data they learn from: the labels can be influenced by sensitive information, causing the neural networks to capture and reproduce these undesirable associations. Therefore, to ensure fairness in decision-making, it becomes essential to mitigate the impact of such undesirable relationships. Research on training fair machine learning models has then received a lot of attention. To achieve fairness, one * Corresponding author. Copyright © 2024, Association for the Advancement of Artificial Intelligence ([www.aaai.org](www.aaai.org)). All rights reserved. can incorporate fairness constraints into the learning objectives (Bilal Zafar et al. 2015; Cotter, Jiang, and Sridharan 2019; Donini et al. 2018; Rezaei et al. 2020; Roh et al. 2020), or modify the model’s predictions using threshold adjustments and calibration to align them with fairness constraints (Hardt, Price, and Srebro 2016; Kim, Ghorbani, and Zou 2019; Lohia et al. 2019; Petersen et al. 2021), or use data manipulation or representation learning methods before model training (Calmon et al. 2017; Choi et al. 2020; Jiang and Nachum 2020; Kamiran and Calders 2012) . Nonetheless, the aforementioned methods primarily focus on modifying the machine learning model to tackle the bias problem. There has been limited effort, as indicated by Jiang and Nachum (2020), in directly addressing the biased data itself, despite the fact that often the training data itself

FAQs

How does label bias create unfair outcomes in machine learning?

When unbiased ground-truth labels are overwritten by a potentially biased agent, the resulting biased dataset harms specific groups. Classifiers then inherit and reproduce these undesirable associations.

Why can low self-confidence scores be misleading?

Low confidence may indicate label errors, but in imbalanced or underrepresented settings, instances can receive low confidence for reasons other than incorrect labeling.

What does the proposed approach do to mitigate label bias using confident learning?

It truncates the confidence scores, extends the probabilistic threshold’s confidence interval, and uses a co-teaching paradigm to select more robust and reliable fair instances for retraining.

Aug 5, 2026

Upload by Levi