稀疏概率分类器

Aug 6, 2026

Support vector machine outputs are often reused as confidence measures, yet no theory guarantees they truly correspond to classification uncertainty. When uncertainty matters, conditional probability estimators are more principled. This work targets ambiguity near the decision boundary, introducing a sparse probabilistic classifier via an adaptation of maximum likelihood implemented with logistic regression. The method produces calibrated probabilities only within a user-defined interval, improves computational efficiency, and yields better results than standard logistic regression with performance similar to SVMs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Motivation

From SVM scores to probabilistic assignments

Gray zone and calibrated decision rules

Motivation from class imbalance

Learning criterion

Bayes decision rule

Maximum likelihood

Sparse Probabilistic Classi􀀌ers Romain H􀀓erault [romain.herault@hds.utc.fr](romain.herault@hds.utc.fr) HEUDIASYC, Universit􀀓e de Technologie de Compi􀀒egne, BP 20529, 60205 Compi􀀒egne cedex, France Yves Grandvalet [yves.grandvalet@idiap.ch](yves.grandvalet@idiap.ch) IDIAP, Rue du Simplon 4, Case Postale 592, CH-1920 Martigny, Switzerland Abstract The scores returned by support vector machines are often used as a con􀀌dence measures in the classi􀀌cation of new examples. However, there is no theoretical argument sustaining this practice. Thus, when classi-􀀌cation uncertainty has to be assessed, it is safer to resort to classi􀀌ers estimating conditional probabilities of class labels. Here, we focus on the ambiguity in the vicinity of the boundary decision. We propose an adaptation of maximum likelihood estimation, instantiated on logistic regression. The model outputs proper conditional probabilities into a user-de􀀌ned interval and is less precise elsewhere. The model is also sparse, in the sense that few examples contribute to the solution. The computational e􀀎ciency is thus improved compared to logistic regression. Furthermore, preliminary experiments show improvements over standard logistic regression and performances similar to support vector machines. 1. Motivation There have been several attempts to turn the scores returned by support vector machines (SVMs) into probabilistic assignments (Platt, 2000; Grandvalet et al. , 2006) . However, there is no guaranty that these scores re􀀍ect a classi􀀌cation con􀀌dence; we even know that the conditional probabilities of class labels cannot be recovered unambiguously except at the decision boundary (Bartlett & Tewari, 2004) . Thus, when the classi􀀌cation uncertainty has to be assessed, estimating conditional probabilities is better motivated. Appearing in Proceedings of the 24 th International Conference on Machine Learning, Corvallis, OR, 2007 . Copyright 2007 by the author(s)/owner(s) . We propose to build probabilistic classi􀀌ers that are accurate on the “gray zone”, where class labels switch. Well-calibrated probabilities in this area allow to assess classi􀀌cation uncertainty. The classi􀀌er also provides relevant decision rules for the set of corresponding asymmetric misclassi􀀌cation losses, or equivalently, for the corresponding cone of the ROC curve. Focusing on a small range of conditional probabilities instead of estimating them on their full span has two advantages. First, the training objective is closer to the ultimate goal of minimizing the misclassi􀀌cation risk, and second, inaccuracy outside of the focus range is a key element for kernelized models, since Bartlett and Tewari (2004) proved that sparsity does not occur when the conditional probabilities can be unambiguously estimated everywhere. Sparsity refers here to the limited number of non-zero elements in a kernel expansion. It implies that many training examples have no in􀀍uence in the training process, thus improving its computational e􀀎ciency. The assessment of classi􀀌cation uncertainty and the sparsity of the model are important issues for the class imbalance problem, which is our original motivation for this work. When a vast majority of examples belong to the negative “uninteresting” class, and only a few interesting examples are available, learning tends to be biased towards the recognition of the majority class. This problem can be addressed by rebalancing the training distribution, either by over-sampling the minority class, or generating arti􀀌cial examples of the minority class (Chawla et al., 2002), or by downsampling the majority class. However, undersampling may discard relevant pieces of information and oversampling is not computationally e􀀎cient. Another tactic consists in post-processing standard classi􀀌cation techniques, such as tuning a bias term after learning to correct for the original decision bias, but this scheme fails to discover the changes in the shape or orientation of

FAQs

为什么不能直接把SVM的分数当作置信度？

因为SVM分数与分类置信度之间缺乏理论保证，除决策边界附近外，类别条件概率也不能被唯一恢复。

文中提出的模型如何处理决策边界附近的不确定性？

聚焦于“灰区”，即类别切换附近的条件概率歧义，通过在该区间内输出经过校准的条件概率来评估不确定性。

该方法为何被称为“稀疏”？

训练过程中只有少量样本对解产生贡献，使核展开中的非零元素数量受限，从而提升计算效率。

Aug 6, 2026

Upload by Lucas Martin