[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123538-en":3,"doc-seo-123538-105":30,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123538,7971461741311,"Ophelia","https://ap-avatar.wpscdn.com/avatar/74000253aff267980c6?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345379180704826",8,"Research & Report","Safety Monitoring of Machine Learning Perception Functions - a Survey","Machine Learning (ML) models, including deep neural networks, are widely used in autonomous systems for perception tasks, but this raises new dependability risks when outputs drive safety-critical decisions. Failures may cause catastrophic consequences in domains such as self-driving vehicles and surgical robotics. This survey reviews safety monitoring of ML-based perception under safety-critical constraints, organizing the literature around threat identification, requirements elicitation, failure detection, reaction, and evaluation, while outlining remaining challenges and directions for future research.","arXiv :2412 .06869v 1 [ cs .LG] 9 Dec 2024  \nORI GINAL AR TI CLE  \nSafety Monitoring of Machine Learning Perception Functions: a Survey  \nRaul Sena Ferreira∗1,2  \nJoris Guérin∗1,2,3  \nKevin Delmas4  \nJérémie Guiochet1,2  \nHélène  \nWaeselynck1  \n1LAAS/CNRS, Toulouse, France 2Université de Toulouse, Toulouse, France 3Espace-Dev, IRD, Université de Montpellier, Montpellier, France  \n4ONERA, Toulouse, France  \nCorrespondence  \n*Equal contribution [raulsenaferreira@gmail.com](raulsenaferreira@gmail.com)[ ](raulsenaferreira@gmail.com)[joris.guerin@ird.fr](joris.guerin@ird.fr)[ ](joris.guerin@ird.fr)[kevin.delmas@onera.fr](kevin.delmas@onera.fr)[ ](kevin.delmas@onera.fr)[jeremie.guiochet@laas.fr](jeremie.guiochet@laas.fr)[ ](jeremie.guiochet@laas.fr)[helene.waeselynck@laas.fr](helene.waeselynck@laas.fr)  \nAbstract  \nMachine Learning (ML) models, such as deep neural networks, are widely applied in autonomous systems to perform complex perception tasks. New dependability challenges arise when ML predictions are used in safety-critical applications, like autonomous cars and surgical robots. Thus, the use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system despite the occurrence of faults. This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context. In this review, we structure the existing literature to highlight key factors to consider when designing such monitors: threat identification, requirements elicitation, detection of failure, reaction, and evaluation. We also highlight the ongoing challenges associated with safety monitoring and suggest directions for future research.  \nK E Y W O R D S  \nFault tolerance, Runtime monitoring, Machine learning perception, Safety-critical autonomous systems  \n1  INTRODUCTION  \nRecent advances in Machine Learning (ML) have allowed autonomous systems to leave the safe environment of research labs to perform complex tasks, where failures can have catastrophic consequences. Examples of such safety-critical systems include self-driving cars, 1 surgical robots, 2 and unmanned aerial vehicles in urban environments.3 These autonomous systems frequently use large ML models like neural networks for complex sensor signal interpretation, i.e., perception, 4 or decision-making, i.e., control.5 This paper focuses on safety mechanisms for critical physical systems relying on ML to process sensor signals.  \nIn various autonomous system applications, essential perception tasks can only be solved using ML. For example, in highly uncontrolled settings, such as self-driving cars 6 or UAV emergency landing, 7 deep neural networks must be used to detect pedestrians in RGB images. This information cannot be obtained from other approaches and is crucial to guarantee the system’s safety. Despite the great success of modern ML-based perception, it introduces new dependability challenges: 8,9,10 1. The lack of well-defined specification: ML models are learned from examples instead of manually coded, making their operational boundaries elusive, and preventing formal safety guarantees. 2. The black-box nature of the models: traceability and transparency of ML predictions is difficult. 3. The high-dimensionality of data: validation of the complete operational design domain is impossible. 4. The over-confidence of neural networks: output scores cannot be used as is to detect failures since a model can deliver wrong outputs with high confidence. 11 Hence, conventional offline safety measures, like fault prevention, removal, and forecasting 12 are often not sufficient to ensure safety and to certify these systems. Online fault tolerance mechanisms, such as Safety Monitors (SM), emerge as a promising alternative to improve safety in critical systems relying on ML perception. This paper focuses on SMs, which aims to keep the system in an acceptable state during operation, despite faults or adverse sce","cbCait5UhSfyr5MI","https://ap.wps.com/l/cbCait5UhSfyr5MI","pdf",879088,1,26,"English","en",105,"# Introduction\n## Safety-critical autonomous perception with ML\n# Safety Monitors and their role\n## Why ML perception outputs need monitoring\n## Characteristics of safety monitors\n# Survey scope and structure\n## Error detection focus in existing studies","[{\"question\":\"What distinguishes safety monitors from traditional offline safety measures?\",\"answer\":\"Offline measures such as fault prevention or forecasting may be insufficient because they cannot fully guarantee safety for complex learned models. Online mechanisms like safety monitors provide runtime fault tolerance by checking safety properties during operation.\"}]","Safety Monitoring of Machine Learning Perception Functions - a Survey | PDF",1785817198,66,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":28},"safety-monitoring-of-machine-learning-perception-functions-a-survey","",{"@graph":36,"@context":77},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/safety-monitoring-of-machine-learning-perception-functions-a-survey/123538/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"What distinguishes safety monitors from traditional offline safety measures?","Question",{"text":75,"@type":76},"Offline measures such as fault prevention or forecasting may be insufficient because they cannot fully guarantee safety for complex learned models. Online mechanisms like safety monitors provide runtime fault tolerance by checking safety properties during operation.","Answer","https://schema.org",{"og:url":52,"og:type":79,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":81,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,115,120,123,127],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":46,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":46,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":113,"slug":114},30,"research-report",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":46,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":98,"slug":130},19,"General","general"]