[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118125-en":3,"doc-seo-118125-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},118125,5909877438554,"Maeve","https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272",8,"Research & Report","Baler - Machine Learning Based Compression of Scientific Data","Storing and sharing increasingly large datasets limits progress across scientific research and industrial operations. This paper presents the development and applications of Baler, a machine learning based data compression tool designed for use across scientific disciplines and industry. Performance is evaluated for compressing High Energy Physics (HEP) data and demonstrated on Computational Fluid Dynamics (CFD) toy data as a proof-of-principle, alongside cross-disciplinary guidance for feasibility studies of learning based compression.","arXiv :2305 .02283v2 [physics .comp-ph] 16 Feb 2024  \nBaler-Machine Learning Based Compression of Scientific Data  \nF. Bengtsson 1 C. Doglioni2 P.A. Ekman 1 A. Gallén 1 P. Jawahar2 A. Orucevic-Alagic 1  \nM. Camps Santasmasas2 N. Skidmore2 O. Woolland2  \n1 Lund University  \n2 University of Manchester  \nAbstract: Storing and sharing increasingly large datasets is a challenge across scientific research and industry. In this paper, we document the development and applications of Baler-a Machine Learning based data compression tool for use across scientific disciplinesand industry. Here, we present Baler’s performance for the compression of High Energy Physics (HEP) data, as well as its application to Computational Fluid Dynamics (CFD) toy data as a proof-of-principle. We also present suggestions for cross-disciplinary guidelines to enable feasibility studies for machine learning based compression for scientific data.  \n1 Introduction  \nMany different fields of science share a common issue; storing ever-growing datasets. By the end of the next decade, the Large Hadron Collider (LHC) experiments will have over an order of magnitude more data to analyze than currently [1–3]; the Square Kilometre Array (SKA) experiment is expected to record 8.5EB of data over its 15-year lifespan [4] and fields such as Computational Fluid Dynamics (CFD) rely on TB-sized simulation samples that need to be stored and shared. Without significant R&D, the datasets expected tobe collected by big-data science experiments are projected to exceed the available storage resources (see e.g. Fig. 2 of Ref. [1] for the case of the ATLAS experiment at the LHC) . This cross-disciplinary issue is not limited to scientific research and extends to industrial operations [5] .  \n1.1 Lossy data compression in high energy physics  \nA common mitigation strategy to this problem involves compressing data using lossless algorithms, see e.g. Refs. [6–8] . Once the storage limit is reached, one is forced to discard parts of the dataset, or only save certain features of the data. Generally, this can be done without impacting the overall scientific program of the experiments, for example by using a data selection system called trigger that only stores data satisfying certain pre-determined characteristics that ensure the dataset will be aligned with the experiment’s main scientific goals. However, saving only a subset of data is not ideal for processes where additional  \nstatistical power is necessary, e.g. for rare signals buried in high-rate backgrounds. In these cases, one can foresee using lossy compression algorithms that reduce the data size ideally beyond what lossless compression algorithms can do [9], using approximation and partial data discarding, at the expense of data fidelity. One limitation of lossy compression is that to obtain high compression ratios with low data loss, the compression algorithm must be tailored to the input data; for instance, MP3 [10] is an example of a lossy compression algorithm that uses techniques specifically suited for waves and frequencies. Thereby, a general solution to this cross-disciplinary problem is hard to obtain by traditional methods. As a solution to this problem, we present Baler, a lossy data compression tool based on the machine learning autoencoder architecture, which tailors the compression to the user’s dataset. It is also important to note that for such a tool to be usable in a scientific experiment, the loss in data quality must be controlled and it must also be deemed to be tolerable/negligible with respect to other sources of experimental jitter.  \n1.2 Autoencoders for lossy data compression  \nAutoencoders (AEs) [11] are a class of unsupervised deep neural networks characterized by an encoder, a central latent space, a decoder, and a target space of the same dimensionality as the input space, as illustrated in Figure 1. The encoder, is a neural network that maps each input, x, to an abstract latent point z, generally o","cbCaigxoWDoiaDmT","https://ap.wps.com/l/cbCaigxoWDoiaDmT","pdf",3604686,1,21,"English","en",105,"# Introduction\n## Lossy data compression in high energy physics\n## Autoencoders for lossy data compression\n# Baler methodology","[{\"question\":\"What problem does Baler address in scientific data sharing and storage?\",\"answer\":\"Growing experiments produce datasets that can exceed available storage resources, forcing tradeoffs in what is kept and how it is compressed. Baler targets this by providing a tailored lossy compression approach.\"},{\"question\":\"Why use lossy compression instead of only lossless methods?\",\"answer\":\"Lossless compression alone may not achieve sufficient size reduction once storage limits are reached. Lossy methods can reduce data further by allowing controlled loss of fidelity.\"},{\"question\":\"How do autoencoders enable compression in Baler?\",\"answer\":\"Autoencoders use an encoder to map inputs to a lower-dimensional latent space and a decoder to reconstruct the output. The latent representation acts as the compressed data, while the decoder serves as decompression.\"}]","Baler - Machine Learning Based Compression of Scientific Data | PDF",1785681729,53,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"baler-machine-learning-based-compression-of-scientific-data","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/baler-machine-learning-based-compression-of-scientific-data/118125/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does Baler address in scientific data sharing and storage?","Question",{"text":75,"@type":76},"Growing experiments produce datasets that can exceed available storage resources, forcing tradeoffs in what is kept and how it is compressed. Baler targets this by providing a tailored lossy compression approach.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why use lossy compression instead of only lossless methods?",{"text":80,"@type":76},"Lossless compression alone may not achieve sufficient size reduction once storage limits are reached. Lossy methods can reduce data further by allowing controlled loss of fidelity.",{"name":82,"@type":73,"acceptedAnswer":83},"How do autoencoders enable compression in Baler?",{"text":84,"@type":76},"Autoencoders use an encoder to map inputs to a lower-dimensional latent space and a decoder to reconstruct the output. The latent representation acts as the compressed data, while the decoder serves as decompression.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]