Evaluation metrics and statistical tests for machine learning - Practical guide to model comparison

Aug 4, 2026

Evaluation metrics and statistical tests support reliable comparison of machine learning (ML) models, especially in supervised tasks such as binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval. The work explains how to select appropriate metrics, gather enough metric values for testing, and perform statistical tests with correct interpretation. Practical examples compare convolutional neural networks for X-ray lung infection classification and segmentation in positron emission tomography images.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Different machine learning tasks

Binary classification

Confusion matrix and core metrics

Statistical testing for comparing models

Practical examples with CNNs and medical imaging

[www. nature.com/scientificreports](www. nature.com/scientificreports) OPEN Evaluation metrics and statistical tests for machine learning Oona Rainio*, JarmoTeuho & Riku Klén Research on different machine learning (ML) has become incredibly popular during the past few decades. However, for some researchers not familiar with statistics, it might be difficult to understand how to evaluate the performance of ML models and compare them with each other. Here, we introduce the most common evaluation metrics used for the typical supervised ML tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval. We explain how to choose a suitable statistical test for comparing models, how to obtain enough values of the metric for testing, and how to perform the test and interpret its results. We also present a few practical examples about comparing convolutional neural networks used to classify X-rays with different lung infections and detect cancer tumors in positron emission tomography images. Keywords Evaluation metrics, Machine learning, Medical images, Statistical testing Due to our developed technology and access to huge amounts of digitized data, the number of different applications using machine learning (ML) has increased dramatically during the past few decades 1. Whereas ML techniques initially included only statistical methods and simple algorithms2, ML is currently used for different purposes across the fields of engineering, medicine, public health, finance, politics, and natural sciences, both in academia and industry3. However, because of this immerse interdisciplinary interest, some of the new ML researchers might not have a good grasp of basic statistical concepts. This prompts need for ongoing education about the proper use of statistics and appropriate metrics for evaluation of performance of ML algorithms. When new ML models are created, it is necessary to compare their performance to the already existing ones4. Evaluation serves two purposes: methods that do not perform well can be discarded, and the ones that seem promising can be further optimized. Also, especially in medicine, it is often useful to know whether an ML model outperforms an educated professional or not5–7. In supervised ML, we first divide our data for training and test sets, use the training data for training and validation of the model, predict all the instances of the test data, and compare the obtained predictions to the corresponding ground-truth values of the test set8. In this way, we can estimate whether the predictions of a new ML model are better than the predictions of a human or existing models in our test set. Despite complexity of final applications, ML models typically consists of relatively simple sub-tasks, such as binary or multi-class classification and regression. In addition, a special image processing ML technique called a convolutional neural network (CNN) can be used to perform image segmentation9 and object detectors are used to find desired targets in images or video footage10. Depending on the task in question, there are certain choices of evaluation metrics that can be used to assess the performance of supervised ML models11. There are also established statistical testing practices, especially for metrics used in binary classification8, 12. Nonetheless, the misuse of certain well-known tests, such as the paired t-test, is common4, and the required assumptions of the tests are often ignored11. Our aim here is to introduce the most common metrics for binary and multi-class classification, regression, image segmentation, and object detection. We explain the basics of statistical testing and what tests should be used in different situations related to supervised ML. At the end, we also give three examples about comparing the performance of CNNs for classifying X-rays related to lung infections and performing image segmentation fo

FAQs

Which supervised ML tasks are covered by the evaluation metrics and statistical testing approach?

The document covers common supervised tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval.

How does the document recommend comparing a new ML model against existing models?

It describes splitting data into training and test sets, training and validating with training data, predicting the test set, and comparing predictions to ground-truth labels using evaluation metrics.

What statistical testing guidance is emphasized for model comparison?

It explains how to choose suitable statistical tests, obtain enough metric values for testing, and interpret results while noting that assumptions and correct usage of tests (e.g., paired t-test misuse) matter.

Aug 4, 2026

Upload by Genevieve