[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-125652-en":3,"doc-seo-125652-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},125652,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","Efficient estimation and correction of selection-induced bias with order statistics","Model selection seeks a well-performing (often simpler) predictive model from a candidate pool, yet the selection rule itself can create bias when cross-validation estimates are dominated by noise. In finite samples, noisy utility estimates can lead to choosing a model that is not truly better for future data. As candidate pools grow and decisions compound, the bias can increase. This paper presents an efficient order-statistics-based estimator and correction method, with diagnostics and numerical validation, including forward search applications.","arXiv :2309 .03742v2 [ stat .ME] 14 Sep 2023  \nEfficient estimation and correction of selection-induced bias with  \norder statistics  \nYann McLatchie and Aki Vehtari  \nDepartment of Computer Science, Aalto University, Finland  \nAbstract. Model selection aims to identify a sufficiently well performing model that is possibly simpler than the most complex model among a pool of candidates. However, the decision-making process itself can inadvertently introduce non-negligible bias when the cross-validation estimates of predictive performance are marred by excessive noise. In finite data regimes, cross-validated estimates can encourage the statistician to select one model over another when it is not actually better for future data. While this bias remains negligible in the case of few models, when the pool of candidates grows, and model selection decisions are compounded (as in forward search), the expected magnitude of selection-induced bias is likely to grow too. This paper introduces an efficient approach to estimate and correct selection-induced bias based on order statistics. Numerical experiments demonstrate the reliability of our approach in estimating both selection-induced bias and over-fitting along compounded model selection decisions, with specific application to forward search. This work represents a light-weight alternative to more computationally expensive approaches to correcting selection-induced bias, such as nested cross-validation and the bootstrap. Our approach rests on several theoretic assumptions, and we provide a diagnostic to help understand when these may not be valid and when to fall back on safer, albeit more computationally expensive approaches. The accompanying code facilitates its practical implementation and fosters further exploration in this area.  \n1. Introduction  \nIn model selection, we are usually interested in identifying the most predictive model from a set of candidates. Specifically, we are interested in identifying a model from a collection of models whose out-of-sample predictive performance is best, either in the hope of generalising to future unseen data (Vehtari and Ojanen, 2012), or as a proxy for parameter recoverability (Scholz and B¨urkner, 2022) . When the number of predictors to consider is large, we might also be interested in achieving a smaller model that is capable of replicating the predictive behaviour of a larger model and use this instead for its improved interpretability, or to decrease data collection cost (Piironen et al., 2020) . In finite data regimes, however, there can be significant uncertainty in differentiating between models of similar performance. Over-optimism in the utility estimate of the selected model is known as selection-induced bias (Stone, 1974) . Alongside this, when we select a model which performs non-negligibly worse than the oracle asymptotically, we say that we have over-fit. The kernel of this work is an efficient and actionable selection-induced bias estimation based on order statistics. Concretely, we:  \n1. discuss over-optimism when making model selection decisions based on noisy crossvalidated predictive metrics;  \n2. propose a lightweight selection-induced bias estimation and correction tool based on order statistics;  \n3. provide a diagnostic for our estimate to understand when it is liable to be unsafe, and more computationally expensive approaches may be warranted;  \n4. show empirically that correcting selection-induced bias can help expose non-negligible over-fitting; and,  \n5. apply our bias correction to real and simulated data examples where we are interested in choosing between multiple candidate models, and in forward search.  \n1.1. Relation to previous work Predictive approaches to Bayesian model comparison have been qualitatively and quantitatively reviewed by Vehtari and Ojanen (2012) and Piironen and Vehtari (2017a) respectively. One popular method is choosing the maximum a posteriori (MAP) model,  \nKeywords: selection-induced","cbCaihxrHwfoQIb7","https://ap.wps.com/l/cbCaihxrHwfoQIb7","pdf",1064222,1,31,"English","en",105,"# Introduction\n## Relation to previous work","[{\"question\":\"What causes selection-induced bias in model selection?\",\"answer\":\"Selection-induced bias arises when cross-validation estimates of predictive performance are noisy, leading to over-optimistic utility for the selected model.\"},{\"question\":\"Why does selection-induced bias grow when the candidate pool becomes larger?\",\"answer\":\"With more candidates and compounded selection decisions (e.g., forward search), the expected magnitude of bias is likely to increase.\"},{\"question\":\"How does the proposed method estimate and correct the bias?\",\"answer\":\"The paper introduces an efficient estimation and correction approach based on order statistics, plus a diagnostic to assess when assumptions may be invalid and safer methods are needed.\"}]","Efficient estimation and correction of selection-induced bias with order statistics | PDF",1785900450,78,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"efficient-estimation-and-correction-of-selection-induced-bias-with-order-statistics","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/efficient-estimation-and-correction-of-selection-induced-bias-with-order-statistics/125652/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What causes selection-induced bias in model selection?","Question",{"text":75,"@type":76},"Selection-induced bias arises when cross-validation estimates of predictive performance are noisy, leading to over-optimistic utility for the selected model.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why does selection-induced bias grow when the candidate pool becomes larger?",{"text":80,"@type":76},"With more candidates and compounded selection decisions (e.g., forward search), the expected magnitude of bias is likely to increase.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the proposed method estimate and correct the bias?",{"text":84,"@type":76},"The paper introduces an efficient estimation and correction approach based on order statistics, plus a diagnostic to assess when assumptions may be invalid and safer methods are needed.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]