[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119681-en":3,"doc-seo-119681-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119681,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Variable-Based Calibration for Machine Learning Classifiers","Machine learning classifiers used in high-stakes settings need confidence scores that are calibrated to true predictive accuracy. This paper proposes variable-based calibration, defining how a model’s calibration behaves with respect to a specific variable of interest, extending conventional score-based calibration and metrics such as expected calibration error (ECE). Results show that near-perfect ECE may still hide substantial, feature-dependent calibration errors. The work provides detection, visualization, and quantification strategies, analyzes limitations of current score-based recalibration, and discusses implications for fairness and model interpretability.","arXiv :2209 . 15154v1 [ cs .LG] 30 Sep 2022  \nVariable-Based Calibration for Machine Learning Classi􀀌ers  \nMarkelle Kelly [kmarke@uci.edu](kmarke@uci.edu)  \nDepartment of Computer Science University of California, Irvine  \nPadhraic Smyth [smyth@ics.uci.edu](smyth@ics.uci.edu)  \nDepartment of Computer Science University of California, Irvine  \nAbstract  \nThe deployment of machine learning classi􀀌ers in high-stakes domains requires well-calibrated con􀀌dence scores for model predictions. In this paper we introduce the notion of variable-based calibration to characterize calibration properties of a model with respect to a variable of interest, generalizing traditional score-based calibration and metrics such as expected calibration error (ECE) . In particular, we 􀀌nd that models with near-perfect ECE can exhibit signi􀀌cant variable-based calibration error as a function of features of the data.  \nWe demonstrate this phenomenon both theoretically and in practice on multiple well-known datasets, and show that it can persist after the application of existing recalibration methods. To mitigate this issue, we propose strategies for detection, visualization, and quanti􀀌cation of variable-based calibration error. We then examine the limitations of current score-based recalibration methods and explore potential modi􀀌cations.  \nFinally, we discuss the implications of these 􀀌ndings, emphasizing that an understanding of calibration beyond simple aggregate measures is crucial for endeavors such as fairness and model interpretability.  \n1. Introduction  \nPredictive models built by machine learning algorithms are increasingly informing decisions across highstakes applications such as medicine (Rajkomar et al., 2019), employment (Chal􀀌n et al., 2016), and criminal justice (Zavr􀀔snik, 2021) . There is also broad recent interest in developing systems where humans and machine learning models collaborate to make predictions and decisions (Kleinberg et al., 2018; Bansalet al., 2021; De et al., 2021; Steyvers et al., 2022) . A critical aspect of using model predictions in such contexts is calibration. In particular, in order to trust the predictions from a machine learning classi􀀌er, these predictions must be accompanied by well-calibrated con􀀌dence scores.  \nIn practice, however, it has been well-documented that machine learning classi􀀌ers such as deep neural networks tend to produce poorly-calibrated class probabilities (Guo et al., 2017; Vaicenavicius et al., 2019; Ovadia et al., 2019) . As a result, a variety of recalibration techniques have been developed, which aim to ensure that a model's con􀀌dence (or score) matches its true accuracy. A widely used approach is post-hoc recalibration: methods which use a separate labeled dataset to learn a mapping from the original model's class probabilities to calibrated probabilities, often with a relatively simple one-dimensional mapping (e.g. , Platt (1999); Kull et al. (2017); Kumar et al. (2019)) . These methods have been shown to generally improve the the empirical calibration error of the model, as commonly measured by the expected calibration error (ECE) .  \nHowever, as we show in this paper, aggregate measures of score-based calibration error such as ECE can hide signi􀀌cant systematic miscalibration in other dimensions of a model's performance. To address this issue we introduce the notion of variable-based calibration to better understand how the calibration error of  \nKelly and Smyth  \n(a) Reliability diagram (for accuracy) (b) Variable-based calibration plot (for error)  \nFigure 1: Calibration plots for a neural network predicting cardiovascular disease, after recalibration with Platt scaling: (a) reliability diagram,(b) LOESS-smoothed estimates with con􀀌dence intervals of actual and model-predicted error as a function of patient age. This dataset consists of 70,000 records of patient data (49,000 train, 6,000 validation, 15,000 test), with a binary prediction task of determining the presence of ca","cbCaiuQ29xsCf7Gd","https://ap.wps.com/l/cbCaiuQ29xsCf7Gd","pdf",1084444,1,27,"English","en",105,"# Introduction\n## Calibration in high-stakes applications\n## Score-based recalibration and its limitations\n## Variable-based calibration: VECE","[{\"question\":\"What problem does variable-based calibration address?\",\"answer\":\"It addresses situations where traditional aggregate measures like ECE can conceal systematic miscalibration that varies with a specific variable of interest.\"},{\"question\":\"How is variable-based calibration different from score-based calibration?\",\"answer\":\"Score-based calibration evaluates calibration against confidence/score distributions, while variable-based calibration measures calibration error relative to a particular real-valued variable (e.g., patient age).\"},{\"question\":\"What do the paper’s experiments and theory show?\",\"answer\":\"They show that models with near-perfect ECE can still exhibit significant variable-based calibration error, the effect can persist after applying existing recalibration methods, and the paper proposes ways to detect and quantify it.\"}]","Variable-Based Calibration for Machine Learning Classifiers | PDF",1785725726,68,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"variable-based-calibration-for-machine-learning-classifiers","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/variable-based-calibration-for-machine-learning-classifiers/119681/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does variable-based calibration address?","Question",{"text":75,"@type":76},"It addresses situations where traditional aggregate measures like ECE can conceal systematic miscalibration that varies with a specific variable of interest.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is variable-based calibration different from score-based calibration?",{"text":80,"@type":76},"Score-based calibration evaluates calibration against confidence/score distributions, while variable-based calibration measures calibration error relative to a particular real-valued variable (e.g., patient age).",{"name":82,"@type":73,"acceptedAnswer":83},"What do the paper’s experiments and theory show?",{"text":84,"@type":76},"They show that models with near-perfect ECE can still exhibit significant variable-based calibration error, the effect can persist after applying existing recalibration methods, and the paper proposes ways to detect and quantify it.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]