Bias Correction in Machine Learning-based Classification of Rare Events - Symposium on Data Science and Statistics (SDSS 2024)

Aug 2, 2026

Online platform businesses can be identified using web-scraped texts, but this classification is difficult because online platforms are rare. The work develops a machine learning text classification approach designed to minimize false positives in order to improve rare-event identification. The method incorporates Bayesian probability calibration and ensemble voting, which substantially reduces bias in prevalence estimates compared with uncalibrated probability outputs. The approach also cuts the need for extensive manual validation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages2

Reading Time5 min

AccessFree to download

Introduction

Rare subpopulations and online platforms

Overview of the proposed automated approach

Data and methods

Data sources and dataset construction

Modeling pipeline and ensemble training

Probability calibration

Results

Reducing false positives

Using model probabilities

Calibrating probabilities

Symposium on Data Science and Statistics (SDSS 2024) Bias Correction in Machine Learning-based Classification of Rare Events Luuk Gubbels Marco Puts Piet Daas Abstract Online platform businesses can be identified by using web-scraped texts. This is a classification problem that combines elements of natural language processing and rare event detection. Because online platforms are rare, accurately identifying them with Machine Learning algorithms is challenging. Here, we describe the development of a Machine Learning-based text classification approach that reduces the number of false positives as much as possible. It greatly reduces the bias in the estimates obtained by using calibrated probabilities and ensembles. Keywords: Calibration, Population, Ensembles 1 Introduction Obtaining reliable information from a small or rare subpopulation is a challenging topic for many researchers. Approaches commonly used to find rare or so-called hard-toidentify groups are a screening survey, network sampling, area sampling, or a combination (Snijkers et al. 2013). Examples of a rare subpopulation are online platforms. These are defined as: A digital service that facilitates interactions between two or more distinct but interdependent sets of users (whether firms or individuals) who interact through the service via the Internet (OECD, 2019) . To obtain a complete overview of all online platforms, a Support Vector Machine (SVM) classification model was developed and applied to the entire population of businesses with a website in the Netherlands. Here, it was found that: i) online platforms compose of 0.22% of the total population of businesses with a website and ii) considerable manual checking was needed in this process (Daas et al., 2023). The latter was predominantly the result of the large number of false positives produced by the model. In this paper, we describe the development of a fully automated approach that aims to reduce the number of false positive online platforms detected as much as possible. This, consequently, seriously reduces the bias in the total estimate of the number of online platforms obtained and the manual checking needed. 2 Data and methods All websites assigned to businesses in the Business Register (BR) of Statistics Netherlands were studied. This resulted in a list of 960,588 (unique) URLs of which 629,284 could actually be scraped (Daas et al. , 2023) . Web scraping was performed as described in Daas and van der Doef (2021) and up to a maximum of 200 pages was collected per URL. The texts were extracted and processed as described in Daas et al.(2023) . Per URL, the texts were combined. For online platform classification , a set of 569 online platforms (positives) were identified by experts. To this set, a random sample of 1328 non-platforms (negatives), from the scraped websites linked to the BR , were added. This resulted in an 1897-sized dataset with 30% platforms and 70% non-platforms. To ensure the independent evaluation of the findings , a 228-sized dataset was created, containing 69 positive and 159 negative cases, by sampling (and removing) them from the 1897-sized dataset. The resulting 1669 dataset was used for model development. All scripts were written in Python (v.3.7) and the Machine Learning algorithms implemented in scikitlearn (v.0.24) were used. The Bayesian calibration method of Puts and Daas (2021) was used to correct the intrinsic prevalence of probability-producing classification models. The code is available on GitHub (Puts, 2023) . Multiple algorithms were tested but in this paper, only the results of Logistic Regression based models are shown. This classification model provided results almost similar to that of the original SVM model but could be trained much faster (Gubbels, 2023) . Multiple models, up to 10, were trained on random resamples (bootstraps) of the 1669-sized training set. The bootstraps were randomly split into a 70% training and a 30% test set. A

FAQs

Why is classifying online platforms considered a rare-event problem?

Online platforms constitute only a small fraction of all businesses with websites. Reliable identification from this rare subpopulation is therefore difficult and can easily lead to biased estimates and many false positives.

What problem occurs when using raw model probabilities from Logistic Regression?

Using uncalibrated probabilities increases the estimated number of online platforms, which in turn increases bias. The probabilities rely on assumptions that are not satisfied in this setting, so they do not behave like actual probabilities.

How do ensembles and probability calibration reduce bias and false positives?

The approach combines multiple Logistic Regression models trained on bootstrapped resamples and aggregates their weighted votes. Bayesian calibration corrects the intrinsic prevalence produced by probability-based classifiers, yielding much lower bias and fewer false positives than the original SVM-based process.

Aug 2, 2026

Upload by Theodore