[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-122666-en":3,"doc-seo-122666-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},122666,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",6,"Technology","Macaw - The Machine Learning Magnetometer Calibration Workflow - Summary","Earth Systems Science relies on complex data pipelines that combine heterogeneous sources and apply sequential filtering and analysis, often implemented as manually executed scripts with limited benefits from modern scientific workflow management systems. Macaw (MAgnetometer CAlibration Workflow) ports a neural-network-based calibration approach for non-dedicated satellite platform magnetometers into a workflow engine. Experiments compare the original HPC cluster runs with a commodity-cluster implementation, reducing CPU hours by 50.2%, memory hours by 59.5%, and runtime by 17.5% through parallelization.","Macaw: The Machine Learning Magnetometer Calibration Workflow  \nJonathan Bader􀀃z , Kevin Styp-Rekowski􀀃z , Leon Doehlerz , Soeren Beckerz , Odej Kaoz , z Technische Universitt Berlin, Germany, ffi[rstname.lastname](rstname.lastnameg@tu-berlin.de)[g](rstname.lastnameg@tu-berlin.de)[@tu-berlin.de](rstname.lastnameg@tu-berlin.de)  \narXiv :2210 .08897v2 [ cs .DC] 18 Jul 2023  \nAbstract—In Earth Systems Science, many complex data pipelines combine different data sources and apply data filtering and analysis steps. Typically, such data analysis processes are historically grown and implemented with many sequentially executed scripts. Scientific workflow management systems (SWMS) allow scientists to use their existing scripts and provide support for parallelization, reusability, monitoring, or failure handling. However, many scientists still rely on their sequentially called scripts and do not profit from the out-of-the-box advantages a SWMS can provide.  \nIn this work, we transform the data analysis processes of a Machine Learning-based approach to calibrate the platform magnetometers of non-dedicated satellites utilizing neural networks into a workflow called Macaw (MAgnetometer CAlibration Workflow). We provide details on the workflow and the steps needed to port these scripts to a scientific workflow. Our experimental evaluation compares the original sequential script executions on the original HPC cluster with our workflow implementation on a commodity cluster. Our results show that through porting, our implementation decreased the allocated CPU hours by 50.2% and the memory hours by 59.5%, leading to significantly less resource wastage. Further, through parallelizing single tasks, we reduced the runtime by 17.5% .  \nIndex Terms—Geomagnetism, Platform Magnetometer, Scientific Workflow, Workflow Management System, Resource Efficiency  \nI. INTRODUCTION  \nScientists frequently have to analyze enormous amounts of data that can easily exceed terabytes of input [1], [2] . Examples of such data analysis challenges are abundant in many scientific domains such as bioinformatics, where many sequences need to be examined in parallel [3]–[7], in Earth observation [8]–[10], where hundreds of high-resolution pictures are analyzed, or in material science, where many molecules are analyzed in parallel [11], [12] .  \nThe data processing is frequently executed on one of three types of infrastructures, a workstation (powerful personal computer), a commodity cluster (ensemble of multiple independent computers with commodity hardware, interconnected via a commodity communication network), or a HighPerformance Computing (HPC) cluster (multiple independent high-performance computers interconnected via a very fast interconnect, e.g., Infiniband or Omni-Path) [13],[14] . Different infrastructure types require different usages. While a scientist can simply run the data processing scripts on his workstation without any changes, the usage of a commodity cluster or a  \nHPC cluster might require code adaptions, e.g., the connection to a resource manager.  \nThe problem of different infrastructure types is further aggravated since scientists frequently move their data between different compute infrastructures. To provide a reproducible data pipeline that can be executed on various infrastructures with different resource managers, scientific workflow management systems (SWMS) can be used. SWMS like Nextflow [15], Pegasus [16], or Snakemake [17] claim to enable reproducible data analysis pipelines, providing out of the box parallelization, monitoring, and failure handling [10] . They are provisioning adapters for different resource managers like Slurm [18] or Kubernetes 1 to support the execution on different clusters. Further, lightweight virtualization technologies like Docker are used to enable reproducibility. In addition, with SWMS, scientists can easily reuse their existing scripts and link them to a data analysis workflow instead of significantly rewriting their existing code.  \n","cbCaiaT9vMk18JLr","https://ap.wps.com/l/cbCaiaT9vMk18JLr","pdf",315360,1,7,"English","en",105,"# Introduction\n## Data pipeline challenges and infrastructure types\n## Role of scientific workflow management systems (SWMS)\n# Macaw workflow concept and calibration goal\n## Neural-network-based magnetometer regression\n## Porting from sequential scripts to Nextflow\n# Evaluation methodology and results\n## CPU, memory, and runtime improvements","[{\"question\":\"What problem does Macaw address in magnetometer calibration?\",\"answer\":\"Macaw addresses the need to calibrate platform magnetometers of non-dedicated satellites using a machine-learning regression model while packaging the calibration process into a workflow for better execution support and portability.\"},{\"question\":\"How does Macaw change the original calibration implementation?\",\"answer\":\"It transforms an originally sequential set of scripts into a scientific workflow implemented with Nextflow to enable parallelization, reproducibility, and error handling.\"},{\"question\":\"What improvements were observed in the evaluation compared with the original HPC scripts?\",\"answer\":\"The workflow implementation reduced allocated CPU hours by 50.2% and memory hours by 59.5%, and reduced runtime by 17.5% via parallelizing single tasks.\"}]","Macaw - The Machine Learning Magnetometer Calibration Workflow - Summary | PDF",1785812052,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"macaw-the-machine-learning-magnetometer-calibration-workflow-summary","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/technology/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/macaw-the-machine-learning-magnetometer-calibration-workflow-summary/122666/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does Macaw address in magnetometer calibration?","Question",{"text":75,"@type":76},"Macaw addresses the need to calibrate platform magnetometers of non-dedicated satellites using a machine-learning regression model while packaging the calibration process into a workflow for better execution support and portability.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does Macaw change the original calibration implementation?",{"text":80,"@type":76},"It transforms an originally sequential set of scripts into a scientific workflow implemented with Nextflow to enable parallelization, reproducibility, and error handling.",{"name":82,"@type":73,"acceptedAnswer":83},"What improvements were observed in the evaluation compared with the original HPC scripts?",{"text":84,"@type":76},"The workflow implementation reduced allocated CPU hours by 50.2% and memory hours by 59.5%, and reduced runtime by 17.5% via parallelizing single tasks.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,117,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":114,"show_sort_weight":115,"slug":116},"Healthcare",40,"healthcare",{"id":118,"doc_module":4,"doc_module_name":46,"category_name":119,"show_sort_weight":120,"slug":121},8,"Research & Report",30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]