A Novel Algorithm Can Generate Data to Train Machine Learning Models in Conditions of Extreme Scarcity of Real World Data

Q: Why is data scarcity a problem for training machine learning models?

Machine learning models require intensive training and large datasets. Collecting, curating, and operating real-world data can be expensive, raises privacy and legal issues, and may be impossible to obtain in scarce settings such as rare disease cohorts.

Q: How does the proposed algorithm generate synthetic data?

A genetic algorithm applies random mutations to candidate generated datasets. A neural network is trained using each dataset, and its performance on a batch of real-world data is used as a surrogate fitness measure for selecting fitter datasets.

Q: What results does the method achieve on the Iris and Breast Cancer datasets?

With real-world data abundance, models trained on generated data reach mean accuracies comparable to those trained on real data. Under simulated extreme scarcity, models trained on generated data show significantly higher mean accuracy than those trained on scarce real data.

Aug 3, 2026

Upload by Logic

Training effective machine learning models depends on access to large datasets, yet real-world data collection and management are constrained by cost, ethical and legal risks, and limited availability. The presented approach introduces a genetic-algorithm-based method that evolves candidate synthetic datasets by repeatedly training a neural network and using its performance on real data as a fitness surrogate. Selection pressure discards unfit datasets and improves fitness across generations. Experiments on Iris and Breast Cancer datasets show comparable accuracy under abundance and significant gains under simulated extreme scarcity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

AccessFree to download

Abstract

Background

Methods

Results

Conclusion

Introduction

Material and methods

Implementation

A novel algorithm can generate data to train machine learning models in conditions of extreme scarcity of real world data Olivier Niel, MD, PhD (1)(2) * (1) Kannerklinik, 4 rue Barblé, L1210 Luxembourg, Luxembourg (2) Faculty of Science, Technology and Medicine, University of Luxembourg, 2 Avenue de l'Université, L4365, Esch-sur-Alzette, Luxembourg * Correspondence: o.r.p.niel@free.fr (preferred), [olivier.niel@ext.uni.lu](olivier.niel@ext.uni.lu) Keywords: data generation, artificial intelligence, machine learning, neural network, genetic algorithm, data augmentation, synthetic data generation, Iris dataset, Breast cancer dataset Abstract Background: Training machine learning models requires large datasets. However, collecting, curating, and operating large and complex sets of real world data poses several problems, in terms of costs, ethical and legal issues, and data availability. To overcome these difficulties, we propose a novel algorithm which can generate large artificial datasets to train machine learning models, even in conditions of extreme scarcity of real world data. Methods: The data generation algorithm is based on a genetic algorithm, which mutates randomly generated datasets subsequently used for training a neural network. After training, the performance of the neural network on a batch of real world data is considered a surrogate for the fitness of the generated dataset used for its training. As selection pressure is applied to the population of generated datasets, unfit generated datasets are discarded, and the fitness of the fittest generated datasets increases throughout generations. Results: The performance of the data generation algorithm was measured on the Iris dataset and on the Breast Cancer Wisconsin diagnostic dataset. In conditions of real world data abundance, mean accuracy of machine learning models trained on generated data was comparable to mean accuracy of machine learning models trained on real world data (0.956 in both cases on the Iris dataset, p = 0.6996, and 0.9377 versus 0.9472 on the Breast Cancer dataset, p = 0.1189) . In conditions of simulated extreme scarcity of real world data, mean accuracy of machine learning models trained on generated data was significantly higher than mean accuracy of comparable machine learning models trained on scarce real world data (0.9533 versus 0.9067 on the Iris dataset, p < 0.0001, and 0.8692 versus 0.7701 on the Breast Cancer dataset, p = 0.0091) . Conclusion: Here we propose a novel algorithm, which can generate large artificial datasets to train machine learning models, in conditions of extreme scarcity of real world data, and also when cost or data sensitivity becomes an obstacle to the collection of large real world datasets. 1. Introduction Artificial intelligence algorithms are now being used extensively in most scientific disciplines. In particular, machine learning models have proven remarkably effective at solving complex classification or regression problems [1]. It is noteworthy that machine learning models need intensive training before they can be used; effective training requires significant computational resources, and, for intrinsic reasons, extremely large datasets [1] . However, collecting, curating, and operating large and complex sets of real world data poses several problems. First, these operations have substantial material and human costs. As an example, in 2021, global spending on big data and analytics reached 241 billion US dollars, and should peak above 655 billion US dollars by 2029 [2] . Second, data collection often raises ethical and legal issues, in terms of privacy and data protection. Indeed, when the nature of data becomes sensitive, which is the case with most personal data like medical health records, complex and costly arrangements are required to comply with local regulations and legislations, such as the General Data Protection Regulation in European countries, and to ensure that data

FAQs

Why is data scarcity a problem for training machine learning models?

Machine learning models require intensive training and large datasets. Collecting, curating, and operating real-world data can be expensive, raises privacy and legal issues, and may be impossible to obtain in scarce settings such as rare disease cohorts.

How does the proposed algorithm generate synthetic data?

A genetic algorithm applies random mutations to candidate generated datasets. A neural network is trained using each dataset, and its performance on a batch of real-world data is used as a surrogate fitness measure for selecting fitter datasets.

What results does the method achieve on the Iris and Breast Cancer datasets?

With real-world data abundance, models trained on generated data reach mean accuracies comparable to those trained on real data. Under simulated extreme scarcity, models trained on generated data show significantly higher mean accuracy than those trained on scarce real data.

Aug 3, 2026

Upload by Logic