Noisy Ostracods - A Fine-Grained, Imbalanced Real-World Dataset - For Benchmarking Robust Machine Learning and Label Correction Methods

Aug 5, 2026

Noisy Ostracods is a noisy, fine-grained dataset for crustacean ostracod genus and species classification with specialist annotations. From 71,466 collected specimens, 5.58% are estimated to contain problematic noise at the genus level. The dataset reflects realistic label challenges including open-set noise from newly discovered classes, pseudo-classes caused by mislabeling into new categories, and strong class imbalance (ρ=22429). Initial robust-learning baselines show limited gains over cross-entropy on raw noisy data, while noise detection underperforms naive cross-validation ensembling. The release supports development of noise-resilient learning and label-correction methods via dataset and evaluation protocols.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Abstract

Introduction

Related works

Noisy Ostracods: A Fine-Grained, Imbalanced Real-World Dataset for Benchmarking Robust Machine Learning and Label Correction Methods Jiamian Hu 1 , Yuanyuan Hong 1 , Yihua Chen 1,2 , He Wang 1,3 , Moriaki Yasuhara 1 1The University of Hong Kong 2The University of Tokyo 3Nanjing Institute of Geology and Palaeontology, CAS {jiamianh, u3001143, yihuaac}@connect.hku.hk, [hwang@nigpas.ac.cn](hwang@nigpas.ac.cn), yasuhara@hku.hk Abstract We present the Noisy Ostracods, a noisy dataset for genus and species classification of crustacean ostracods with specialists’ annotations. Over the 71466 specimens collected, 5.58% of them are estimated to be noisy (possibly problematic) at genus level. The dataset is created to addressing a real-world challenge: creating a clean fine-grained taxonomy dataset. The Noisy Ostracods dataset has diverse noises from multiple sources. Firstly, the noise is open-set, including new classes discovered during curation that were not part of the original annotation. The dataset has pseudo-classes, where annotators misclassified samples that should belong to an existing class into a new pseudo-class. The Noisy Ostracods dataset is highly imbalanced with a imbalance factor ρ = 22429 . This presents a unique challenge for robust machine learning methods, as existing approaches have not been extensively evaluated on fine-grained classification tasks with such diverse real-world noise. Initial experiments using current robust learning techniques have not yielded significant performance improvements on the Noisy Ostracods dataset compared to cross-entropy training on the raw, noisy data. On the other hand, noise detection methods have underperformed in error hit rate compared to naive cross-validation ensembling for identifying problematic labels. These findings suggest that the fine-grained, imbalanced nature, and complex noise characteristics of the dataset present considerable challenges for existing noiserobust algorithms. By openly releasing the Noisy Ostracods dataset, our goal is to encourage further research into the development of noise-resilient machine learning methods capable of effectively handling diverse, real-world noise in finegrained classification tasks. The dataset, along with its evaluation protocols, can be accessed at [https://github.com/H-Jamieu/Noisy_ostracods](https://github.com/H-Jamieu/Noisy_ostracods). 1 Introduction Ostracods are micro-crustaceans inhabiting in marine, non-marine and some semi-terrestrial habitats[1] . Their calcified shells preserved in the sediments provided rich material for paleoenvironmental reconstruction, ecological monitoring and bio-diversity analyses [2] . However, conducting such studies requires massive efforts in counting, sorting, and identifying ostracods [3] . With advancements in image acquisition technologies and deep learning methods [4, 5, 6], building an automatic identification system for ostracods has become a feasible solution to reduce human labor. Such a system necessitates a large amount of annotated data. In our practical work, the annotated dataset also serves as a taxonomy reference material for teaching. In these applications, any incorrect taxonomy 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks. (noise) discovered in the dataset raises user concerns about the trustworthiness of the identification results from the models trained on this data. Therefore, the need for robust, trustworthy machine learning algorithms and cleaned datasets is critical for the adaption of deep learning methods in the taxonomy field. Creating a clean dataset is both labor-intensive and time-consuming. Before the current version of Noisy Ostracods, we manually checked and retook corrupted photos over several rounds in a two-year period. Despite these efforts, new errors continue to be discovered by users, motivating us to conduct an extensive study on Learning with Noisy Labels (LNL)

FAQs

What makes the Noisy Ostracods dataset different from typical classification benchmarks?

It contains realistic, diverse label noise (open-set noise and pseudo-classes) and strong class imbalance, making it harder for robust learning and label correction methods.

How much noisy data is estimated at the genus level?

5.58% of the 71,466 specimens are estimated to be noisy or problematic at the genus level.

Do existing robust learning and noise detection methods perform well on this dataset?

Experiments indicate limited performance improvements from current robust-learning techniques versus cross-entropy on raw noisy data, and noise detection methods underperform naive cross-validation ensembling for finding problematic labels.

Aug 5, 2026

Upload by Theodora