[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-126649-en":3,"doc-seo-126649-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},126649,549768064778,"Finn","https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8",8,"Research & Report","NOISE REDUCTION IN METRIC, EVENT, LOG, AND TRACE (MELT) DATA USING DISTRIBUTED MACHINE LEARNING","In the Observability domain, metric, event, log and trace (MELT) datasets are produced at high volume and high frequency and are inherently related. Existing monitoring and analytics methods often treat each type separately, limiting holistic correlation across the full environment. This work presents scalable noise-reduction techniques that dynamically adjust data collection and enable change-point detection, anomaly detection, log pattern detection, and causal inference with two-phase filtering via Edge Processors and Global Processors.","Technical Disclosure Commons  \nDefensive Publications Series  \nApril 2023  \nNOISE REDUCTION IN METRIC, EVENT, LOG, AND TRACE (MELT) DATA USING DISTRIBUTED MACHINE LEARNING  \nSrinivasan Srinivasan  \nLinda Zhou  \nManikandan Mari Vera Kumara  \nFollow this and additional works at: [https://www.tdcommons.org/dpubs_series](https://www.tdcommons.org/dpubs_series)  \nRecommended Citation  \nSrinivasan, Srinivasan; Zhou, Linda; and Vera Kumara, Manikandan Mari, \"NOISE REDUCTION IN METRIC, EVENT, LOG, AND TRACE (MELT) DATA USING DISTRIBUTED MACHINE LEARNING\", Technical Disclosure Commons,(April 10, 2023)  \n[https://www.tdcommons.org/dpubs_series/5788](https://www.tdcommons.org/dpubs_series/5788)  \nThis work is licensed under a Creative Commons Attribution 4.0 License.  \nThis Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons.  \nNOISE REDUCTION IN METRIC, EVENT, LOG, AND TRACE (MELT) DATA  \nUSING DISTRIBUTED MACHINE LEARNING  \nAUTHORS:  \nSrinivasan Arashanipalai  \nLinda Zhou  \nManikandan Mari Vera Kumara  \nABSTRACT  \nIn the Observability domain, metric, event, log and trace (MELT) are basic data types generated by the infrastructure and applications. These datasets are not only ingested at high volume and high frequency but also related. Currently, available solutions are for individual data types, i.e., metric monitoring, log analytics, trace flow analysis, etc. These solutions do not provide a holistic view of the entire environment with MELT correlation. To address these types of challenges, techniques are presented herein that support a scalable, flexible, dynamic, and adaptive noise reduction system. While the system is running as expected, data is collected at a lower frequency. When the first sign of trouble appears, such a system may automatically increase collection frequency for change point detection, anomaly detection, log pattern detection, and causal inference. Aspects of the presented techniques employ a two-phase filtering mechanism comprising Edge Processors and Global Processors to intelligently apply machine learning techniques to scale up and down monitoring and root cause analysis capabilities.  \nDETAILED DESCRIPTION  \nTo diagnose the cause of an anomaly related to business transactions or key performance indicators (KPIs), users need to drill down into the related logs, events, and traces. In typical cloud native applications, the volume of traces and logs that are associated with a business transaction represents a huge overhead. The problem is further aggravated by the fact that the logs are usually unstructured, low-level, noisy, and lack the information about changes to the states of resources. Consequently, users need to employ both traces and logs, as logs relate to intra-service behaviors while traces pertain to inter-service behaviors.  \n1 6848  \nPublished by Technical Disclosure Commons, 2023 2  \nTo identify and reduce noise, an integrated approach is used to decide the relationships and the dependencies between metrics and the related logs and traces. Such an approach should satisfy several design goals.  \nThe first design goal encompasses scalability. In cloud native deployments it is quite common for thousands of requests related to metric, event, log, and trace (MELT) data types which need to be processed each second. The second design goal encompasses low overhead. The approach should identify the incoming data as either significant or noise without impacting the throughout. The third design goal encompasses configurability. The approach should allow users to specify quotas and those constraints should be used to decide whether to mark data as noisy or significant.  \nThe fourth design goal encompasses flexibility. The approach should allow a user to configure the solution to fit their specific requirements. For example, in addition to anout-of","cbCaigWLBKSB2n7E","https://ap.wps.com/l/cbCaigWLBKSB2n7E","pdf",764022,1,15,"English","en",105,"# Detailed Description\n## Observability and MELT data challenges\n## Integrated approach for noise identification\n## Design goals for the proposed system\n## Key definitions: noise and relevance score\n## Two-phase filtering with Edge and Global processors","[{\"question\":\"为什么需要在MELT数据中进行噪声降低？\",\"answer\":\"MELT数据在根因分析时带来低功能价值并造成显著开销，同时日志常常是非结构化、低层级且嘈杂，缺少资源状态变化信息。噪声降低可以减少用户钻取成本并提升诊断效率。\"},{\"question\":\"文中提出的噪声降低系统如何动态应对异常？\",\"answer\":\"系统在运行正常时以较低频率采集数据；当出现早期问题迹象时，会自动提高采集频率以支持变点检测、异常检测、日志模式检测与因果推断。\"},{\"question\":\"两阶段过滤机制（Edge Processors与Global Processors）的作用是什么？\",\"answer\":\"两阶段过滤用于判断入站数据是否应被分类为噪声。该分布式过滤避免单一集中式过滤带来的瓶颈，并可扩展监控与根因分析能力。\"}]","NOISE REDUCTION IN METRIC, EVENT, LOG, AND TRACE (MELT) DATA USING DISTRIBUTED MACHINE LEARNING | PDF",1785934032,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"noise-reduction-in-metric-event-log-and-trace-melt-data-using-distributed-machine-learning","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/noise-reduction-in-metric-event-log-and-trace-melt-data-using-distributed-machine-learning/126649/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么需要在MELT数据中进行噪声降低？","Question",{"text":75,"@type":76},"MELT数据在根因分析时带来低功能价值并造成显著开销，同时日志常常是非结构化、低层级且嘈杂，缺少资源状态变化信息。噪声降低可以减少用户钻取成本并提升诊断效率。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"文中提出的噪声降低系统如何动态应对异常？",{"text":80,"@type":76},"系统在运行正常时以较低频率采集数据；当出现早期问题迹象时，会自动提高采集频率以支持变点检测、异常检测、日志模式检测与因果推断。",{"name":82,"@type":73,"acceptedAnswer":83},"两阶段过滤机制（Edge Processors与Global Processors）的作用是什么？",{"text":84,"@type":76},"两阶段过滤用于判断入站数据是否应被分类为噪声。该分布式过滤避免单一集中式过滤带来的瓶颈，并可扩展监控与根因分析能力。","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]