[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-120658-en":3,"doc-seo-120658-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},120658,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","A fast and flexible machine learning approach to data quality monitoring - Abstract","A machine learning framework enables real-time monitoring of particle detectors by testing whether incoming experimental data batches match a reference sample that represents normal operating behavior. The method performs a likelihood-ratio hypothesis test using a fast, flexible kernel-based model with nonparametric learning capacity, allowing it to approximate complex data relationships from sufficient data. The resulting detection algorithm is efficient and broadly applicable to different anomaly types. Performance is demonstrated on multivariate data from a drift tube chambers muon detector.","A fast and flexible machine learning approach to data quality monitoring  \narXiv :2301 .08917v1 [hep-ex] 21 Jan 2023  \nGaia Grosso  \nDipartimento di Fisica e Astronomia Università di Padova INFN, Sez. di Padova Padova, Italy  \nCERN, Experimental Physics Department Geneva, Switzerland [gaia.grosso@cern.ch](gaia.grosso@cern.ch)  \nNicolò Lai  \nDipartimento di Fisica e Astronomia Università di Padova Padova, Italy  \n[nicolo.lai@studenti.unipd.it](nicolo.lai@studenti.unipd.it)  \nMarco Letizia  \nMaLGa Center-DIBRIS Università di Genova INFN, Sez. di Genova Genova, Italy  \n[marco.letizia@edu.unige.it](marco.letizia@edu.unige.it)  \nJacopo Pazzini  \nDipartimento di Fisica e Astronomia Università di Padova INFN, Sez. di Padova Padova, Italy [jacopo.pazzini@unipd.it](jacopo.pazzini@unipd.it)  \nMarco Rando  \nMaLGa Center-DIBRIS Università di Genova Genova, Italy  \n[marco.rando@edu.unige.it](marco.rando@edu.unige.it)  \nAndrea Wulzer  \nDipartimento di Fisica e Astronomia Università di Padova Padova, Italy [andrea.wulzer@cern.ch](andrea.wulzer@cern.ch)  \nMarco Zanetti  \nDipartimento di Fisica e Astronomia  \nUniversità di Padova  \nINFN, Sez. di Padova  \nPadova, Italy  \n[Marco.Zanetti@cern.ch](Marco.Zanetti@cern.ch)  \nAbstract  \nWe present a machine learning based approach for real-time monitoring of particle detectors. The proposed strategy evaluates the compatibility between incoming batches of experimental data and a reference sample representing the data behavior in normal conditions by implementing a likelihood-ratio hypothesis test. The core model is powered by recent large-scale implementations of kernel methods, nonparametric learning algorithms that can approximate any continuous function given enough data. The resulting algorithm is fast, efficient and agnostic about the type of potential anomaly in the data. We show the performance of the model on multivariate data from a drift tube chambers muon detector.  \nMachine Learning and the Physical Sciences workshop, NeurIPS 2022 .  \n1 Introduction  \nModern high-energy physics experiments consist of complex detectors where hundreds of millions of sensors are read out as frequently as every few nanoseconds. The electrical signals are amplified, processed and combined before the trigger selection and the final storage. At each of these steps some errors can occur and invalidate the whole process. Monitoring systems are deployed to assess the quality of the data and keep the flow under control throughout all the stages of the acquisition. Data quality monitoring (DQM) is a challenging task from a statistical point of view due to its high intrinsic dimensionality and the high level of human supervision required. The unforeseen events incoming during an experimental run can be several. Some of them can be anticipated and recognised while they occur, others cannot. The detection of well known dysfunctions could be nonetheless missed due to the huge amount of channels that should be simultaneously monitored. Developing flexible highly automatized techniques to supervise multiple variables at once is thus fundamental to reduce the risk of failures [1, 2, 3, 4] . This work proposes the use of a recent machine learning approach to compare collected data with a sample of reference events that depicts the correct detector readings. This reference sample can be, for instance, a set of measurements in a controlled scenario or simulated events. The basic idea is to perform a hypothesis test powered by a fast and flexible machine learning (ML) algorithm. In practice, we leverage the ability of binary classifiers to implicitly model the underlying data-generating distributions and estimate the likelihood ratio test statistics. This is then used to assess whether the hypothesis underlying the observed data (alternative hypothesis) agrees with the assumption of normal behavior (null hypothesis) . If a set of measurements significantly deviates from the reference sample, the learned likelihood ratio can be used to cha","cbCair3Wa3kKva1G","https://ap.wps.com/l/cbCair3Wa3kKva1G","pdf",2425377,1,6,"English","en",105,"# 1 Introduction\n# 2 Experimental setup and data samples\n# 3 ML model and core strategy\n# 4 Results overview\n# Conclusions and further developments","[{\"question\":\"How does the proposed method monitor detector data quality in real time?\",\"answer\":\"It compares incoming data batches with a reference dataset representing normal detector behavior, using a likelihood-ratio hypothesis test driven by a machine learning model.\"},{\"question\":\"What role does the kernel-based learning model play?\",\"answer\":\"The kernel-based nonparametric model acts as a fast, flexible binary classifier that estimates likelihood-ratio test statistics and captures underlying data distributions.\"},{\"question\":\"What kind of anomalies can the approach detect?\",\"answer\":\"The approach is designed to be agnostic to the specific anomaly type, flagging measurement sets that significantly deviate from the reference sample in feature space.\"}]","A fast and flexible machine learning approach to data quality monitoring - Abstract | PDF",1785731204,15,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"a-fast-and-flexible-machine-learning-approach-to-data-quality-monitoring-abstract","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/a-fast-and-flexible-machine-learning-approach-to-data-quality-monitoring-abstract/120658/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does the proposed method monitor detector data quality in real time?","Question",{"text":75,"@type":76},"It compares incoming data batches with a reference dataset representing normal detector behavior, using a likelihood-ratio hypothesis test driven by a machine learning model.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What role does the kernel-based learning model play?",{"text":80,"@type":76},"The kernel-based nonparametric model acts as a fast, flexible binary classifier that estimates likelihood-ratio test statistics and captures underlying data distributions.",{"name":82,"@type":73,"acceptedAnswer":83},"What kind of anomalies can the approach detect?",{"text":84,"@type":76},"The approach is designed to be agnostic to the specific anomaly type, flagging measurement sets that significantly deviate from the reference sample in feature space.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]