[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119198-en":3,"doc-seo-119198-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119198,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Assessing Robustness of Machine Learning Models using Covariate Perturbations","As machine learning models are increasingly used in critical decision-making systems, ensuring robustness under adversarial attacks and input-data changes becomes essential, particularly when models risk overfitting. This paper introduces a comprehensive framework that evaluates robustness via covariate perturbation techniques. It studies multiple perturbation strategies, including separate handling for numeric and non-numeric variables, summarizes perturbation effects to compare robustness across scenarios, and applies local robustness diagnosis to locate unstable data regions. Empirical studies on real-world datasets validate improved comparisons across models and reveal model instabilities while strengthening robustness.","Assessing robustness of machine learning models using covariate perturbations  \nArun Prakash R1, Anwesha Bhattacharyya, Joel Vaughan, and Vijayan N. Nair  \nModel Risk Management, Wells Fargo2  \nMay 2024  \nAbstract  \nAs machine learning models become increasingly prevalent in critical decision-making models and systems in fields like finance, healthcare, etc., ensuring their robustness against adversarial attacks and changes in the input data is paramount, especially in cases where models potentially overfit. This paper proposes a comprehensive framework for assessing the robustness of machine learning models through covariate perturbation techniques. We explore various perturbation strategies to assess robustness and examine their impact on model predictions, including separate strategies for numeric and non-numeric variables, summaries of perturbations to assess and compare model robustness across different scenarios, and local robustness diagnosis to identify any regions in the data where a model is particularly unstable. Through empirical studies on real world dataset, we demonstrate the effectiveness of our approach in comparing robustness across models, identifying the instabilities in the model, and enhancing model robustness.  \n1 Introduction  \nThe term “robustness” has a long history in statistics and data science. Early usage was concerned with need for inferential methods that were not overly sensitive to the presence of outliers in the data. (Box 1979) appears to be the first to discuss the notion of model robustness. His definition of robustness “as the property of a procedure which renders the answers it gives insensitive to departures, of a kind which occur in practice, from ideal assumptions” is very general and covers different types of robustness. Since Box’s pioneering paper, there has been extensive work in the statistical literature on model robustness. Early work dealt with parametric models, where one examines the behavior of procedures under model misspecifications. With increasingly large sample sizes, advances in computing, and renewed interest in flexible nonparametric models (using machine learning algorithms), we were able to relax restrictive parametric assumptions and fit increasingly complex models. These led to the opposite problem where the models become too flexible and can overfit the training data. One way to address this in the model fitting stage is through regularization, and there is a vast literature on regularization techniques and the associated hyperparameter tuning algorithms. Despite these approaches, there is still a need for methods to assess the robustness of a model after it has been fit to the training data. This is the focus of the present paper.  \nThe main reason for assessing model robustness is the generalizability of a model on unseen data. A common way to measure this is by comparing predictive performances between train and test datasets (which are typically hold-out data sets, in-time, out-of-time test sets, or subset of the entire dataset) . An overfit model fits to the noise in the training data and thus, despite producing low training error, but it can  \n1 Corresponding author (email: [arunprakash.r@wellsfargo.com](arunprakash.r@wellsfargo.com))  \n2 The views expressed in the paper are those of the authors and do not represent the views of Wells Fargo.  \nhave a comparatively large test error leading to poor performance when used in production, as discussed in (Groh 2022) . The large difference between training and test datasets is often called gap. To understand the reasons and regions where the over-fit occurs, it is useful to segment the test and training datasets into natural groups (by predictors or regions such as tails, areas with sparse data, etc.) and compare the gaps. Such comparisons provide useful insights that can be exploited to improve the model fit on the test dataset. This approach is examined in detail elsewhere in our work.  \nAnother technique to a","cbCaisrPWwuuPmcA","https://ap.wps.com/l/cbCaisrPWwuuPmcA","pdf",1364953,1,31,"English","en",105,"# Introduction\n## Motivation for assessing robustness\n## Measuring generalizability via train-test gaps\n## Complexity-based robustness concepts\n## Covariate-perturbation framework and scope","[{\"question\":\"What is the main approach proposed for assessing robustness?\",\"answer\":\"The paper proposes evaluating robustness by perturbing covariates (predictors) and examining the resulting reduction in predictive performance.\"},{\"question\":\"Why is robustness assessment especially important in critical decision-making systems?\",\"answer\":\"Robustness is crucial to maintain reliable predictions under adversarial attacks and changes in input data, particularly when models may overfit training data.\"},{\"question\":\"How does the framework handle different types of variables?\",\"answer\":\"It explores perturbation strategies with separate treatment for numeric and non-numeric variables to assess how each affects model predictions.\"}]","Assessing Robustness of Machine Learning Models using Covariate Perturbations | PDF",1785723052,78,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"assessing-robustness-of-machine-learning-models-using-covariate-perturbations","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/assessing-robustness-of-machine-learning-models-using-covariate-perturbations/119198/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main approach proposed for assessing robustness?","Question",{"text":75,"@type":76},"The paper proposes evaluating robustness by perturbing covariates (predictors) and examining the resulting reduction in predictive performance.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why is robustness assessment especially important in critical decision-making systems?",{"text":80,"@type":76},"Robustness is crucial to maintain reliable predictions under adversarial attacks and changes in input data, particularly when models may overfit training data.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the framework handle different types of variables?",{"text":84,"@type":76},"It explores perturbation strategies with separate treatment for numeric and non-numeric variables to assess how each affects model predictions.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]