[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-127763-en":3,"doc-seo-127763-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},127763,1099523882182,"Eliana","https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8",8,"Research & Report","Design Patterns for Machine Learning Based Systems with Human-in-the-Loop","Supervised machine learning systems remain difficult to develop and deploy due to limited prediction reliability and uncertainty about how to effectively embed human intelligence into automated decision-making. Human-in-the-Loop (HiL) mitigates gaps of fully automated approaches and improves practical applicability. The work compiles a developer-oriented catalog of HiL design patterns, explicitly accounting for human involvement costs and model retraining needs, and covering training, deployment, and cooperation pattern types.","Preprint-Authors’ version-Accepted for publication at IEEE Software  \nDesign Patterns for Machine Learning Based Systems with Human-in-the-Loop  \nJakob Smedegaard Andersen and Walid Maalej, University of Hamburg, Germany  \narXiv :2312 .00582v 1 [ cs . SE] 1 Dec 2023  \nAbstract—The development and deployment of systems using supervised machine learning (ML) remain challenging: mainly due to the limited reliability of prediction models and the lack of knowledge on how to effectively integrate human intelligence into automated decision-making. Humans involvement in the ML process is a promising and powerful paradigm to overcome the limitations of pure automated predictions and improve the applicability of ML in practice. We compile a catalog of design patterns to guide developers select and implement suitable human in-the-loop (HiL) solutions. Our catalog takes into consideration key requirements as the cost of human involvement and model retraining. It includes four training patterns, four deployment patterns, and two orthogonal cooperation patterns.  \nM  \nachine learning (ML) has become a major field of research with tremendous progress in recent years. With this, software developers  \nare increasingly confronted with the need to integrate ML models into their systems. In particular, supervised ML has become essential to leverage insights hidden in large datasets, optimize workflows, and gain competitive advantages. Supervised ML aims at learning patterns from pre-labelled examples in order to make accurate predictions about new unseen data.  \nDespite the research progress, the development and deployment of ML approaches for real world applications still poses critical engineering [1], [2] and deployment challenges [3] . Getting ML models to work accurately in practice is drastically different from idealized research and development environments. Typical limitations include not meeting accuracy and reliability requirements, the lack of user acceptance, model hallucination, few but critical prediction mistakes, as well as the shortage of adequate and rich data to (re)train models. To mitigate the limitations of pure automated approaches, research has thus suggested the Humanin-the-Loop (HiL) paradigm [4] .  \nIn HiL, humans are empowered to continuously provide feedback to the system at each stage of the ML pipeline: including data collection and preparation, model training and evaluation, operation and monitoring (as Figure 1 shows) . HiL focuses on feedback  \nXXXX-XXX © 2023 IEEE  \nDigital Object Identifier 10.1109/XXX.0000.0000000  \nfrom domain experts and actual users of the system, i.e. people with expertise in a specific application domain but not necessarily technical knowledge. For supervised ML, this feedback is usually either corrections of labels or additional labels. Domain-, application- or context-specific feedback has the advantage of incorporating specific knowledge from the actual domain, which a model is unaware of—thus with a high potential for improvement. HiL considers human feedback an essential part of the ML process, rather than replacing humans with full automation. Simple, time-consuming, and repetitive tasks should be automated as much as possible, while the human should step into the loop from time to time to solve creative, challenging, difficult, or interesting cases.  \nWhile the concept behind HiL is promising, its research space is rather convoluted and its operationalizability and applicability in software development is rather unexplored. Moreover, the design space requires a careful consideration:  \n• Human assistance in the creation and use of ML models usually comes with a high cost and should thus be carefully designed and minimized.  \n• Also retraining ML models can, in certain cases, be very expensive and with significant carbon footprint.  \n• Both models and humans may make (different types of) mistakes which require a careful tradeoff discussion and decision that fits the use cases and do","cbCaiqPLpopTFyrU","https://ap.wps.com/l/cbCaiqPLpopTFyrU","pdf",390377,1,9,"English","en",105,"# Overview of Human-in-the-Loop for Supervised ML\n## Motivation: Reliability and Deployment Challenges\n## HiL Paradigm in the ML Pipeline\n## Feedback Sources and Types","[{\"question\":\"What challenges motivate the Human-in-the-Loop (HiL) approach for supervised ML?\",\"answer\":\"Supervised ML deployment struggles with limited reliability of prediction models and insufficient knowledge on integrating human intelligence into automated decisions. Practical systems also face accuracy and reliability gaps, user acceptance issues, and data limitations for training or retraining.\"},{\"question\":\"How does HiL change the supervised ML workflow?\",\"answer\":\"HiL enables humans to provide continuous feedback at multiple pipeline stages, including data collection and preparation, model training and evaluation, and operation and monitoring. This keeps human expertise involved throughout the lifecycle rather than relying on full automation.\"},{\"question\":\"What kinds of feedback does HiL typically provide in supervised ML?\",\"answer\":\"Feedback often comes from domain experts and actual users and commonly takes the form of label corrections or additional labels. Context-specific feedback can incorporate knowledge the model may not otherwise capture.\"}]","Design Patterns for Machine Learning Based Systems with Human-in-the-Loop | PDF",1785941480,23,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"design-patterns-for-machine-learning-based-systems-with-human-in-the-loop","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/design-patterns-for-machine-learning-based-systems-with-human-in-the-loop/127763/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-24","2026-08-05",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What challenges motivate the Human-in-the-Loop (HiL) approach for supervised ML?","Question",{"text":76,"@type":77},"Supervised ML deployment struggles with limited reliability of prediction models and insufficient knowledge on integrating human intelligence into automated decisions. Practical systems also face accuracy and reliability gaps, user acceptance issues, and data limitations for training or retraining.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does HiL change the supervised ML workflow?",{"text":81,"@type":77},"HiL enables humans to provide continuous feedback at multiple pipeline stages, including data collection and preparation, model training and evaluation, and operation and monitoring. This keeps human expertise involved throughout the lifecycle rather than relying on full automation.",{"name":83,"@type":74,"acceptedAnswer":84},"What kinds of feedback does HiL typically provide in supervised ML?",{"text":85,"@type":77},"Feedback often comes from domain experts and actual users and commonly takes the form of label corrections or additional labels. Context-specific feedback can incorporate knowledge the model may not otherwise capture.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,121,124,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]