[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118251-en":3,"doc-seo-118251-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},118251,687197207639,"Asher","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Valid Inference for Machine Learning Model Parameters","Machine learning models are commonly trained by minimizing a loss over training data, but this raises overfitting concerns and creates uncertainty about whether the learned parameter is optimal for the whole population. The paper builds valid confidence sets for the population-optimal parameter using only training data, with frequentist repeated-sampling guarantees. It further leverages the resulting confidence-set distribution to quantify confidence for arbitrary parameter-space regions and shows that this distribution can be accurately approximated via bootstrapping.","arXiv :2302 . 10840v2 [ stat .ML] 9 May 2024  \nValid Inference for Machine Learning Model Parameters  \nNeil Dey [ndey3@ncsu.edu](ndey3@ncsu.edu)  \nJonathan P. Williams [jwilli27@ncsu.edu](jwilli27@ncsu.edu)  \nDepartment of Statistics North Carolina State University Raleigh, NC 27607-6698, USA  \nAbstract  \nThe parameters of a machine learning model are typically learned by minimizing a loss function on a set of training data. However, this can come with the risk of overtraining; in order for the model to generalize well, it is of great importance that we are able to find the optimal parameter for the model on the entire population—not only on the given training sample. In this paper, we construct valid confidence sets for this optimal parameter of a machine learning model, which can be generated using only the training data without any knowledge of the population. We then show that studying the distribution of this confidence set allows us to assign a notion of confidence to arbitrary regions of the parameter space, and we demonstrate that this distribution can be well-approximated using bootstrapping techniques.  \nKeywords: PAC-Learning, Glivenko-Cantelli Classes, Random Sets, Imprecise Probability, Hypothesis Testing  \n1. Introduction  \nMany machine learning (ML) applications train a model by finding parameters that minimize a loss function (such as the zero-one loss or squared loss) on some training data. This approach generalizes well outside the training data if the average loss on the training data is sufficiently close to the expected loss (i.e. “risk”) for the entire population. Classical probably approximately correct (PAC) learning theory (e.g. Valiant, 1984; Kearns et al. , 1994; Mohri et al., 2018) provides theoretical rates at which the empirical risk function converges to the population risk in various settings, but provides no theoretical guarantees about estimation properties of the risk minimizer. In general, little work has been done to quantify the uncertainty in an empirical risk minimizer (ERM) .  \nGiven that an ERM is a function of the data, aleatory uncertainty is a natural quantification of the inherent randomness of the ERM: It is desirable to be able to construct confidence sets and hypothesis tests for risk minimizers of machine learning models that are“valid” in the sense that they posses frequentist repeated sampling guarantees. The utility of validity in machine learning contexts is perhaps most clearly illustrated by conformal prediction algorithm, introduced by Vovk et al. (2005) . Conformal prediction can take a point prediction method (for classification or regression problems) and yield valid prediction regions for new data, in the sense that the generated prediction region will contain the true label with any prespecified level of confidence. Thus, there exists a very useful validity guarantee on the prediction regions generated by conformal prediction-enhanced machine learning algorithms. Because this validity property is so desirable, conformal prediction has  \n©2022 Neil Dey and Jonathan P. Williams.  \nLicense: CC-BY 4.0, see [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/) .  \nDey and Williams  \nfound use in a variety of applications, such as online regression forests (Vasiloudis et al. , 2019), QSAR modelling for drug development (Eklund et al., 2015), convolutional neural networks for image classification (Matiz and Barner, 2019), and various deep-learning architectures (Messoudi et al., 2020), among others. However, conformal prediction cannot quantify uncertainty in the estimate of the ERM ([i.e. one](i.e. one) of the “inputs” of an untrained ML model)—only on the outputs of the trained model.  \nWe also note that in the context of the broader development in the literature of inferential approaches with finite-sample validity guarantees (not necessarily restricted to ERMs), papers on “e-values” and “e-processes” have been gaining prominence—m","cbCairch4o9lYJW8","https://ap.wps.com/l/cbCairch4o9lYJW8","pdf",661000,1,35,"English","en",105,"# Introduction\n## Valid confidence sets for risk minimizers\n## Validity via frequentist repeated sampling\n## Connections to conformal prediction and safer evidence measures\n## Related uncertainty quantification frameworks","[{\"question\":\"Why is uncertainty quantification important for machine learning model parameters?\",\"answer\":\"Training by empirical risk minimization may not yield a parameter that is optimal for the entire population. Quantifying uncertainty helps ensure the learned parameter generalizes beyond the training sample.\"},{\"question\":\"How does the paper construct valid confidence sets for the optimal parameter?\",\"answer\":\"The method constructs confidence sets for the population-optimal parameter using only the training data, while guaranteeing frequentist repeated-sampling validity.\"},{\"question\":\"How is confidence assigned to arbitrary regions of the parameter space?\",\"answer\":\"By studying the distribution of the constructed confidence set, the approach derives a notion of confidence for any specified region within the parameter space.\"}]","Valid Inference for Machine Learning Model Parameters | PDF",1785682642,88,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"valid-inference-for-machine-learning-model-parameters","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/valid-inference-for-machine-learning-model-parameters/118251/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is uncertainty quantification important for machine learning model parameters?","Question",{"text":75,"@type":76},"Training by empirical risk minimization may not yield a parameter that is optimal for the entire population. Quantifying uncertainty helps ensure the learned parameter generalizes beyond the training sample.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the paper construct valid confidence sets for the optimal parameter?",{"text":80,"@type":76},"The method constructs confidence sets for the population-optimal parameter using only the training data, while guaranteeing frequentist repeated-sampling validity.",{"name":82,"@type":73,"acceptedAnswer":83},"How is confidence assigned to arbitrary regions of the parameter space?",{"text":84,"@type":76},"By studying the distribution of the constructed confidence set, the approach derives a notion of confidence for any specified region within the parameter space.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]