Sampling Bias Compensation for Robust Evaluation of Audio Classification Systems with Partially Labeled Evaluation Datasets

Jul 16, 2026

Upload by Maeve

Audio classification systems are often evaluated using fully annotated test sets, but real deployments cannot label large volumes of continuously collected audio. This paper addresses performance estimation under strict annotation-budget limits, where labeled subsets become biased and distort evaluation metrics. Importance weighting is studied as a correction mechanism by estimating density ratios in the deployed audio feature space. Three density-ratio estimators—KDE, logistic regression, and kNN—are compared using subsets generated via five active-learning sampling strategies on an audio scene classification benchmark, reducing the gap to true performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

Operational challenge of partial labeling

Sampling bias in evaluation metrics

Performance estimation via density-ratio importance weighting

Sampling Bias Compensation for Robust Evaluation of Audio Classification Systems with Partially Labeled Evaluation Datasets Javier Naranjo-Alcazar1 , Annamaria Mesaros 2, Tuomas Virtanen2, Pedro Zuccarello 1 1Instituto Tecnologico de Informatica (ITI), Valencia, Spain 2Tampere University, Tampere, Finland arXiv :2607 .04463v 1 [ cs . SD] 5 Jul 2026 Abstract—The performance of acoustic machine learning systems is commonly evaluated using fully annotated test sets. In real-world deployments, however, exhaustively labeling large volumes of continuously collected audio data is often infeasible. Consequently, performance assessment typically relies on a small labeled subset of the available data, introducing a sampling bias that can severely distort evaluation metrics. This paper studies methods for compensating the bias in evaluationlabeled subsets under strict annotation-budget constraints. We study whether importance weighting techniques can mitigate this discrepancy by compensating for the selection bias. Specifically, we implement and compare three density-ratio estimation methods: kernel density estimation (KDE), logistic regression, and k-nearest neighbors (kNN), utilizing feature-space representations of the deployed audio. To emulate realistic deployment scenarios, the labeled subsets are generated using five distinct sampling strategies based on active learning techniques. Experiments conducted on an audio scene classification (ASC) benchmark demonstrate that importance weighting consistently yields more realistic accuracy estimates, significantly reducing the gap between subset-based metricsand the true evaluation performance. Index Terms—Evaluation estimation, Sampling bias, Audio classification 1. INTRODUCTION The performance of machine listening systems is traditionally assessed using fully annotated benchmark datasets under controlled experimental conditions [1]–[3] . While this offline setting is appropriate for algorithm development, it rarely reflects the operational reality of realworld deployments. Once a model is trained on development data and deployed into production, it typically operates on continuous audio streams to perform ongoing inference [4] . In practical monitoring applications, evaluating operational performance relies on analyzing a collected sample of this deployment traffic. However, because exhaustively annotating every operational recording is logistically and economically infeasible, performance assessment must rely on a small, selected subset of the ongoing stream. Consequently, tracking the model’s long-term behavior depends entirely on evaluation metrics computed over this limited labeled snapshot, even though its representativeness with respect to the true deployment distribution remains uncertain. A representative example of this operational challenge can be found in large-scale environmental audio monitoring initiatives such as SorollIA [5], where extensive recording campaigns continuously generate substantially more audio than can realistically be annotated. In these real-time streaming scenarios, annotation efforts are usually guided by active learning (AL) strategies or other sample selection procedures designed to identify segments that maximize annotation value [6] . Consequently, the final evaluation subset is not obtained through random or exhaustive labeling, but rather by selecting a limited number of segments from a massive, continuously growing pool of collected audio. Moreover, the segments selected for annotation are typically chosen according to predefined criteria, such as maximizing diversity [6], [7], prioritizing uncertain samples [8], [9], or increasing the representation of rare acoustic conditions [10] . While these strategies are beneficial for building effective training datasets, they may produce a labeled subset whose distribution differs from that of the complete collection of recorded audio. This mismatch introduces selection bias, leading

FAQs

Why does evaluation sampling bias occur in real audio classification deployments?

In production, audio is collected continuously, but exhaustive labeling is infeasible. Evaluation therefore uses a small selected labeled subset, whose selection is guided by procedures such as active learning, making it non-representative of the full deployment distribution.

How does the paper compensate for sampling bias during evaluation?

The approach estimates density ratios between the full deployment distribution and the labeled subset in the audio feature space. These ratios are converted into importance weights and used to compute weighted evaluation metrics.

Which density-ratio estimation methods are compared, and what is their role?

The paper compares three methods: kernel density estimation (KDE), logistic regression, and k-nearest neighbors (kNN). Each method estimates the density ratio needed for importance weighting.

Jul 16, 2026

Upload by Maeve