[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117912-en":3,"doc-seo-117912-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117912,16904993612988,"Olivia Brown","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Performance Estimation Methods for Diagnosing Model Quality Degradation under Covariate Shift - Master Thesis","This study investigates performance degradation in machine learning models under dynamic data conditions, focusing on how covariate shift undermines deployed accuracy over time. Instead of prioritizing only new training strategies, it targets detection and diagnosis after deployment and the moment performance begins to decline. Three tests are introduced: Temporal Degradation to study degradation across samples, Continuous Retraining to model ongoing updates in production, and Performance Estimation using methods such as Direct Loss Estimation (DLE) to flag degradation without ground-truth labels. Results show diverse degradation behaviors across model families, with continuous retraining providing partial relief and performance estimation acting as an early warning system.","University of Padova  \nDepartment of Mathematics Tullio-Levi Civita Master Thesis in Data Science  \nPerformance Estimation Methods for Diagnosing Model Quality Degradation under Covariate Shift  \nSupervisor Master Candidate  \nBruno Scarpa Santiago Víquez Segura  \nUniversity of Padova  \nAcademic Year  \n2022-2023  \nii  \nDedication.  \nTo my mom, for encouraging me to follow the path that I enjoy the most. To Kathu, for telling me that I could. And to Maple, for reminding me to enjoy the little things in life.  \niv  \nAbstract  \nThis study explores the underexplored domain of performance degradation in machine learning models, a common challenge in dynamic data environments. While the academic community often prioritizes the development of new training methods to enhance model performance on benchmark datasets, little attention has been given to sustaining those high levels of performance post-deployment or identifying when performance starts to decade. In a rapidly evolving data landscape, where covariate shifts can significantly impact model performance, understanding and mitigating performance degradation issues becomes essential.  \nTo tackle this issue, this study introduces three comprehensive tests. The Temporal Degradation Test examines how various models perform when trained on different samples ofthe same dataset, shedding light on degradation patterns. The Continuous Retraining Test simulates a production environment by assessing the impact of continuous model retraining. Finally, the Performance Estimation Test explores the potential of performance estimation methods, such as Direct Loss Estimation (DLE), to identify degradation without ground truth data. Our findings reveal diverse degradation patterns influenced by machine learning methodologies, with continuous retraining offering partial relief but not complete resolution. Performance estimation methods emerge as vital early warning systems, enabling timely interventions to maintain model efficacy.  \nvi  \nContents  \nAbstract v  \nList of figures ix  \nList of tables xi  \nListing of acronyms xiii  \n1 Introduction 1  \n2 Methods 3  \n2.1 Temporal Degradation Test .......................... 3  \n2.2 Continuous Retraining Test .......................... 5  \n2.3 Performance Estimation Test ......................... 7  \n2.4 Methods and Hyperparameter Tunning ................... 8  \n2.4.1 LGBMRegressor ........................... 10  \n2.4.2 ElasticNet .............................. 10  \n2.4.3 RandomForestRegressor ....................... 11  \n2.4.4 MLPRegressor ............................ 11  \n3 Datasets 13  \n3.1 US Avocado Hass Sales ............................ 13  \n3.1.1 Data Pre-processing and feature engineering ............. 14  \n3.1.2 Exploratory data analysis ....................... 14  \n3.1.3 Baseline model and first post-deployment analysis........... 17  \n3.2 NYC Taxi Service Demand .......................... 19  \n3.2.1 Data Pre-processing and feature engineering ............. 20  \n3.2.2 Exploratory data analysis ....................... 21  \n3.2.3 Baseline model and first post-deployment analysis .......... 22  \n4 Results and Discussion 25  \n4.1 Temporal degradation results ......................... 25  \n4.1.1 Dataset: US Avocado Hass Sales ................... 25  \n4.1.2 Dataset: NYC Taxi Trip Demand .................. 30  \n4.2 Continuous retraining results ......................... 34  \n4.2.1 Dataset: US Avocado Hass Sales ................... 34  \n4.2.2 Dataset: NYC Taxi Trip Demand .................. 39  \n4.3 Performance estimation results ........................ 42  \n5 Conclusion 45  \n5.1 Summary of results .............................. 45  \n5.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45  \n5.3 Concluding words ............................... 46  \nReferences 47  \nAcknowledgments 49  \nListing of figures  \n2.1 Temporal Degradation Experiment ...................... 5  \n2.2 Continuous Retraining Experiment .....................","cbCaiop5IEnZrUds","https://ap.wps.com/l/cbCaiop5IEnZrUds","pdf",13092288,1,63,"English","en",105,"# Introduction\n## Performance degradation under covariate shift\n# Methods\n## Temporal Degradation Test\n## Continuous Retraining Test\n## Performance Estimation Test\n## Methods and hyperparameter tuning\n# Datasets\n## US Avocado Hass Sales\n## NYC Taxi Service Demand\n# Results and Discussion\n## Temporal degradation results\n## Continuous retraining results\n## Performance estimation results\n# Conclusion\n## Summary of results\n## Limitations","[{\"question\":\"What problem does the thesis address regarding machine learning models in production?\",\"answer\":\"It addresses how model performance can degrade after deployment when data distributions change, especially under covariate shift.\"},{\"question\":\"What are the three proposed tests in the study?\",\"answer\":\"The thesis proposes the Temporal Degradation Test, the Continuous Retraining Test, and the Performance Estimation Test.\"},{\"question\":\"How can performance be estimated without ground-truth data?\",\"answer\":\"The study explores performance estimation methods such as Direct Loss Estimation (DLE) to identify degradation signals without requiring ground truth.\"}]","Performance Estimation Methods for Diagnosing Model Quality Degradation under Covariate Shift - Master Thesis | PDF",1785680353,159,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"performance-estimation-methods-for-diagnosing-model-quality-degradation-under-covariate-shift-master-thesis","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/performance-estimation-methods-for-diagnosing-model-quality-degradation-under-covariate-shift-master-thesis/117912/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05","2026-08-02",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does the thesis address regarding machine learning models in production?","Question",{"text":76,"@type":77},"It addresses how model performance can degrade after deployment when data distributions change, especially under covariate shift.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What are the three proposed tests in the study?",{"text":81,"@type":77},"The thesis proposes the Temporal Degradation Test, the Continuous Retraining Test, and the Performance Estimation Test.",{"name":83,"@type":74,"acceptedAnswer":84},"How can performance be estimated without ground-truth data?",{"text":85,"@type":77},"The study explores performance estimation methods such as Direct Loss Estimation (DLE) to identify degradation signals without requiring ground truth.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,121,124,129,132,136],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":46,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":46,"category_name":138,"show_sort_weight":107,"slug":139},19,"General","general"]