[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85461-en":3,"doc-seo-85461-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85461,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",8,"Research & Report","Monitoring the Calibration of Probability Forecasts with an Application to Concept Drift Detection","Machine learning image classification models can be highly accurate yet produce probability estimates that do not match the empirical event frequencies. This work addresses the gap in methods for continuously monitoring calibration loss over time, rather than evaluating it only once. A cumulative-sum (CUSUM) procedure with dynamic limits is proposed to detect miscalibration in both traditional process monitoring and concept drift settings. The approach uses only predicted probabilities and observed outcomes, enabling early detection of operational changes that degrade field performance.","arXiv :2510 .25573v2 [ stat .ML] 12 Jul 2026  \nMonitoring the calibration of probability forecasts with an application to concept drift detection involving image  \nclassification  \nChristopher T. Franck, Anne R. Driscoll, Zoe Szajnfarber, William H. Woodall  \nJuly 14, 2026  \nAbstract  \nMachine learning approaches for image classification have led to impressive advancesin that field. For example, convolutional neural networks are able to achieve remarkable image classification accuracy across a wide range of applications in industry, defense, and other areas. While these machine learning models boast impressive accuracy, a related concern is how to assess and maintain calibration in the predictions these models make. A classification model is said to be well calibrated if its predicted probabilities correspond with the rates events actually occur. While there are many available methods to assess machine learning calibration and recalibrate faulty predictions, less effort has been spent on developing approaches that continually monitor predictive models for potential loss of calibration as time passes. We propose a cumulative sum-based approach with dynamic limits that enable detection of miscalibration in both traditional process monitoring and concept drift applications. This enables early detection of operational context changes that impact image classification performance in the field. The proposed chart can be used broadly in any situation where the user needs to monitor probability predictions overtime for potential lapses in calibration. Importantly, our method operates on probability predictions and event outcomes and does not require under-the-hood access to the machine learning model.  \nKeywords: image classification, calibration, cumulative sum (CUSUM) chart, dynamic control limits, probability prediction, statistical process monitoring, concept drift.  \n1 Introduction  \nBinary classification models such as logistic regression, tree-based approaches, and neural networks can furnish predictions that an event will occur in the probability space between zero and one. Perhaps surprisingly, these predictions may not actually reflect the empirical rate at which events occur, particularly when the model makes predictions outside of the training data. When predictions do not correspond to the actual event rate, we say a model is uncalibrated. Calibrated models provide predictions that correspond to the observed event rate. Thus, achieving and maintaining calibration is extremely important when model predictions are used to make decisions, as calibrated predictions help users understand the actual risk of prospective events. While calibration of prediction models has been well studied, there is a lack of methods available to prospectively monitor predictions to determine if and when predictive models lose calibration. There are a variety of potential industrial and quality applications where a monitoring procedure that assesses calibration could be useful. Perhaps a machine learning model is used in a manufacturing setting to predict the probability that apart fails a downstream inspection, but a change in a material supplier compromises the ability for the model to predict probabilities properly. In the medical monitoring context, neural network-based approaches can predict surgical mortality (for example, B¨orner et al. 2024), but such a model may lose calibration over time as new treatments appear and patient profiles change. The purpose of this paper is to propose a calibration monitoring cumulative sum (CUSUM) approach that indicates whether and at what time a model becomes uncalibrated.  \nThe core idea behind the proposed calibration CUSUM chart is illustrated in Figure 1. The model’s probability predictions are assessed at each time point (horizontal axis), and the calibration status of those predictions is summarized with our proposed CUSUM statistic (vertical axis) described in Section 4. In the left panel, the model ","cbCaipaIhXgcBe8t","https://ap.wps.com/l/cbCaipaIhXgcBe8t","pdf",1784426,1,29,"English","en",105,"# Introduction\n## Calibration and the need for monitoring\n## Proposed calibration CUSUM concept","[{\"question\":\"What does it mean for a probability forecasting model to be calibrated?\",\"answer\":\"A model is well calibrated when its predicted probabilities correspond to the event rates that actually occur.\"},{\"question\":\"Why is continuous monitoring of calibration important?\",\"answer\":\"Models can lose calibration as time passes, especially when predictions move outside training conditions or when concept drift and contextual changes occur.\"},{\"question\":\"How does the proposed method detect calibration loss?\",\"answer\":\"It uses a cumulative sum (CUSUM) statistic with dynamic control limits to flag when miscalibration occurs shortly after the chart exceeds the limits.\"}]",1784203717,73,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"monitoring-the-calibration-of-probability-forecasts-with-an-application-to-concept-drift-detection","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/monitoring-the-calibration-of-probability-forecasts-with-an-application-to-concept-drift-detection/85461/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What does it mean for a probability forecasting model to be calibrated?","Question",{"text":74,"@type":75},"A model is well calibrated when its predicted probabilities correspond to the event rates that actually occur.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Why is continuous monitoring of calibration important?",{"text":79,"@type":75},"Models can lose calibration as time passes, especially when predictions move outside training conditions or when concept drift and contextual changes occur.",{"name":81,"@type":72,"acceptedAnswer":82},"How does the proposed method detect calibration loss?",{"text":83,"@type":75},"It uses a cumulative sum (CUSUM) statistic with dynamic control limits to flag when miscalibration occurs shortly after the chart exceeds the limits.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]