An Entropic Metric for Measuring Calibration of Machine Learning Models

Aug 2, 2026

Uploaded by Levi

Machine learning model calibration determines whether predicted confidence probabilities reflect true correctness rates rather than only accuracy. The paper introduces Entropic Calibration Difference (ECD), a new calibration metric grounded in state-estimation research on target tracking. ECD separates and quantifies under-confidence and over-confidence instead of conflating them. The work analyzes why safer under-confident behavior can trade accuracy for statistical inefficiency, then evaluates ECD on simulated and real data. Results are compared against Expected Calibration Error (ECE) and Expected Signed Calibration Error (ESCE).

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Abstract

Introduction

Calibration of probabilities and limitations of accuracy

Motivation from modern neural networks being badly calibrated

Existing Metrics for Model Calibration

Brier Score

Probability adjustment and evaluation metrics

Proposed Entropic Calibration Difference (ECD)

Safe calibration and interpretation

Experiments and Results

Simulated and real data evaluation

Comparison with ECE and ESCE

Conclusion

An Entropic Metric for Measuring Calibration of Machine Learning Models Daniel James Sumler∗ , Lee Devlin∗ , Simon Maskell∗ , and Richard O. Lane† ∗ The University of Liverpool, UK. †QinetiQ, Great Malvern, UK. arXiv :2502 . 14545v1 [ cs .LG] 20 Feb 2025 Abstract—Understanding the confidence with which a machine learning model classifies an input datum is an important, and perhaps under-investigated, concept. In this paper, we propose a new calibration metric, the Entropic Calibration Difference (ECD). Based on existing research in the field of state estimation, specifically target tracking (TT), we show how ECD may be applied to binary classification machine learning models. We describe the relative importance of under- and over-confidence and how they are not conflated in the TT literature. Indeed, our metric distinguishes under- from over-confidence. We consider this important given that algorithms that are under-confident are likely to be “safer” than algorithms that are over-confident, albeit at the expense of also being over-cautious and so statistically inefficient. We demonstrate how this new metric performs on real and simulated data and compare with other metrics for machine learning model probability calibration, including the Expected Calibration Error (ECE) and its signed counterpart, the Expected Signed Calibration Error (ESCE). I. INTRODUCTION Calibration of probabilities is an important and oftenoverlooked concept when developing machine learning (ML) models. Usually, accuracy is the main metric used to calculate how well an ML model performs in terms of predicting a class for unseen data. Generally speaking, the closer the accuracy is to 100%, the better the model is deemed to be. However, this does not take into account the probability of predictions that the model outputs, which can be just as important, if not more, than the accuracy. In binary classification, a probability greater than a threshold, typically 0.5, is enough to decide whether an input belongs to one of two classes. While accuracy informs whether a classification is correct, a calibration metric informs how well the confidence probabilities match the true proportions of correct decisions. For example, a model that always outputsa probability of 0.6 for class label 1, but always gets this classification correct, should produce a poor calibration score, as even though the model has classified the output correctly, it has low confidence in that decision. Calibration has become even more important as of late, asthe research of Guo et al. [1] reveals that while modern neural networks are more accurate than ever, they are also badly calibrated. This could be attributed to the over-confidence of said networks due to the large amount of data they are able to be trained on. A well-calibrated model can be defined as one that outputs probabilities that are representative of the real-life occurrences from the unseen data. For example if, on average, 70% of people are correctly predicted to contract a certain disease, then one would expect the average probability outputted by a diagnosis model to be 0.7 . A mathematical representation of calibration can be seen in (1), where x ∈ {0, 1} is the true label, y ∈ Rd is an observed data sample of dimension d belonging to a binary class k ∈ {0, 1}, and pk is the confidence in class k, while P is the true probability. P (x = k|y) = pk (1) In this paper, we present a novel calibration metric that addresses some weaknesses of some of the most commonly-used existing metrics. In section II, we discuss existing calibration metrics that are widely discussed throughout the literature. In section III, we detail our motivations for “safe” calibration and why we feel our metric is necessary. Section IV explains our new metric and details how the results can be interpreted. In section V, we test our metric on simulated and real data, and compare it with other popular calibration metrics. Finally,

FAQs

Why does the paper discuss “safe” calibration?

It argues that under-confident models are often safer than over-confident ones, though this safety can come with over-cautious and statistically inefficient predictions.

Aug 2, 2026

Uploaded by Levi