[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119189-en":3,"doc-seo-119189-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119189,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Learning Run-time Safety Monitors for Machine Learning components","Machine learning components deployed in autonomous systems for safety-critical tasks must remain reliably assured after deployment, despite changes in the operating environment. Runtime performance monitoring becomes essential, especially when ground truth is unavailable during operation. This work presents a process to create safety monitors using degraded datasets and machine learning, deploying the monitor in parallel with the original ML component to predict safety risk from model outputs under current inputs. Experiments on public speed sign datasets validate the approach.","arXiv :2406 . 16220v1 [ cs .LG] 23 Jun 2024  \nLearning Run-time Safety Monitors for Machine Learning components  \nOzan Vardal 1 , Richard Hawkins 1 , Colin Paterson 1 , Chiara Picardi 1 , Daniel Omeiza2 , Lars Kunze2 , and Ibrahim Habli 1  \n1 Department of Computer Science, University of York, York, UK  \n{ozan.vardal,richard.hawkins,[colin.paterson](colin.paterson}@york.ac.uk)[}](colin.paterson}@york.ac.uk)[@york.ac.uk](colin.paterson}@york.ac.uk)  \n2 Department of Engineering, University of Oxford, Oxford, UK  \nAbstract. For machine learning components used as part of autonomous systems (AS) in carrying out critical tasks it is crucial that assurance of the models can be maintained in the face of post-deployment changes (such as changes in the operating environment of the system) . A critical part of this is to be able to monitor when the performance of the model at runtime (as a result of changes) poses a safety risk to the system.  \nThis is a particularly difficult challenge when ground truth is unavailable at runtime. In this paper we introduce a process for creating safety monitors for ML components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output. We demonstrate the viability of our approach through some initial experiments using publicly available speed sign datasets.  \n1 Introduction  \nThe use of machine learning (ML) in perception and understanding is essential for many autonomous systems (AS) . Where such systems are used for safety related tasks it is critical that the safety of the ML components can be assured prior to deployment. Post deployment, we must be able to demonstrate that the system continues to operatate safely throughout operation in complex and dynamic environments. In [18] we introduced transfer assurance, a process for assuring ML components used in AS. Transfer assurance is used when the ML component is required to be updated in response to changes in the AS or, crucially, in response to changes in the environment in which it operates. Figure 1 shows an overview of our transfer assurance process, which is split into three stages containing six activities. The first stage considers the initial development of the ML component for deployment into an operational AS and makes use of the AMLAS assurance process [8] . This stage results in the creation of an ML component along with its safety case and a set of appropriate ML safety monitors. These ML safety monitors are deployed on the AS along with the ML component and are used to identify changes in the system or environment which invalidate the safety case for the ML component. The second and the  \n2 Vardal et al.  \nthird phases deal with analysing and responding to such changes to maintain acceptable safety of the AS. This paper focuses on the first stage of the transfer assurance process, in particular the creation of effective safety monitors which allow us to understand the impact of change on ML models at run-time (activities 2 and 3 in Figure 1) .  \nFig. 1: Three Stage Transfer Assurance Process with activities shown in green and artefacts in yellow.  \nThe challenge for activity 2 is to create monitors capable of detecting when ML component outputs are potentially unsafe due to changes in operating conditions. This is particularly challenging because we are unable to determine ground-truth during operation and because changes in operating conditions are combinatorial in nature, meaning that only by understanding the complex interplay of features can we determine the impact on safety.  \nWe address this challenge by training an ML safety monitor with degraded data to reflect the potential impact of real-world factors. These degraded datasets are presented to the ML component and labelled to reflect their impact on model performance. This labelled data is then used as training ","cbCaiv7K8PIb6j3S","https://ap.wps.com/l/cbCaiv7K8PIb6j3S","pdf",904203,1,15,"English","en",105,"# Introduction\n## Related work","[{\"question\":\"Why are runtime safety monitors important for machine learning components in autonomous systems?\",\"answer\":\"They help maintain assurance when post-deployment changes alter operating conditions, which can create safety risks. Monitoring is crucial because performance may degrade without immediate ground truth.\"},{\"question\":\"How does the proposed method create safety monitors without ground truth at runtime?\",\"answer\":\"It trains a safety monitor using degraded datasets that model real-world factors and labels reflecting impacts on performance. The trained monitor then predicts safety risk during operation.\"},{\"question\":\"What is the role of the safety monitor during deployment alongside the ML component?\",\"answer\":\"The monitor runs in parallel using the same runtime inputs and provides a safety risk prediction for the ML output. If risk is significant, the system may trigger mitigation actions and reassess the safety case.\"}]","Learning Run-time Safety Monitors for Machine Learning components | PDF",1785723009,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"learning-run-time-safety-monitors-for-machine-learning-components","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/learning-run-time-safety-monitors-for-machine-learning-components/119189/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why are runtime safety monitors important for machine learning components in autonomous systems?","Question",{"text":75,"@type":76},"They help maintain assurance when post-deployment changes alter operating conditions, which can create safety risks. Monitoring is crucial because performance may degrade without immediate ground truth.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed method create safety monitors without ground truth at runtime?",{"text":80,"@type":76},"It trains a safety monitor using degraded datasets that model real-world factors and labels reflecting impacts on performance. The trained monitor then predicts safety risk during operation.",{"name":82,"@type":73,"acceptedAnswer":83},"What is the role of the safety monitor during deployment alongside the ML component?",{"text":84,"@type":76},"The monitor runs in parallel using the same runtime inputs and provides a safety risk prediction for the ML output. If risk is significant, the system may trigger mitigation actions and reassess the safety case.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]