[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-120345-en":3,"doc-seo-120345-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},120345,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",8,"Research & Report","Decoupling Structural and Quantitative Knowledge in ReLU-based Deep Neural Networks","Rapid expansion of AI applications increases both economic and environmental costs from training deep neural networks (DNNs). To address limitations of further optimizing conventional DNN training, this paper introduces an approach that decouples structural information, represented by non-linear functions, from quantitative knowledge, represented by model parameters. Experiments show that ReLU-based DNNs can be deployed as globally linear models where different network parts activate per sample, yielding piece-wise linear outputs. The resulting design enables independent training objectives, including faster re-training and incremental or federated re-training by combining models trained on different datasets.","Decoupling Structural and Quantitative Knowledge in ReLU-based Deep Neural Networks  \nJosé Duato  \n[jduato@qsimov.com](jduato@qsimov.com)[ ](jduato@qsimov.com)Qsimov Quantum Computing S.L. Talavera de la Reina, Spain  \nJose I. Mestre  \n[jmiravet@uji.es](jmiravet@uji.es)[ ](jmiravet@uji.es)Universitat Jaume I Castelló de la Plana, Spain  \nManuel F. Dolz  \n[dolzm@uji.es](dolzm@uji.es)[ ](dolzm@uji.es)Universitat Jaume I Castelló de la Plana, Spain  \nEnrique S. Quintana-Ortí  \n[quintana@disca.upv.es](quintana@disca.upv.es)[ ](quintana@disca.upv.es)Universitat Politècnica de València Valencia, Spain  \nAbstract  \nThe relentless growth of artificial intelligence applications has led to substantial economic and environmental costs associated with training deep neural networks (DNNs) . Recognizing the challenges in further optimizing conventional DNN training, in this paper we propose a novel approach that decouples structural information (non-linear functions) from quantitative knowledge (model parameters), and provide strong experimental evidence to demonstrate that these two types of knowledge can be trained independently. We evidence that ReLU-based DNNs can be deployed as globally linear models, from which different parts of the DNN are active for each sample, thus emulating the piece-wise linear outputs generated by ReLU activation functions. Leveraging this linear model foundation, this kind of DNN supports various objectives, including faster re-training times and combining multiple copies trained on different datasets for incremental and federated re-training.  \nCCS Concepts: • Computing methodologies → Neural networks; Learning linear models.  \nKeywords: Artificial Intelligence, Deep Neural Networks, ReLU function, Structural and Quantitative Knowledge  \nACM Reference Format:  \nJosé Duato, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí, and José Cano. 2025. Decoupling Structural and Quantitative Knowledge in ReLU-based Deep Neural Networks. In The 5th Workshop on Machine Learning and Systems (EuroMLSys ’25), March 30-April 3, 2025, Rotterdam, Netherlands. ACM, New York, NY, USA, 7 pages.  \n[https://doi.org/10.1145/3721146.3721950](https://doi.org/10.1145/3721146.3721950)  \nThis work is licensed under a Creative Commons AttributionNonCommercial-NoDerivatives 4 .0 International License.  \nEuroMLSys ’25, Rotterdam, Netherlands  \n© 2025 Copyright held by the owner/author(s) .  \nACM ISBN 979-8-4007-1538-9/2025/03  \n[https://doi.org/10.1145/3721146.3721950](https://doi.org/10.1145/3721146.3721950)  \nJosé Cano  \n[jose.canoreyes@glasgow.ac.uk](jose.canoreyes@glasgow.ac.uk)[ ](jose.canoreyes@glasgow.ac.uk)University of Glasgow Glasgow, United Kingdom  \n1 Introduction  \nDeep Learning (DL) has become a highly effective solution for addressing a growing number of challenges. Nevertheless, the time and energy costs to train complex Deep Neural Networks (DNNs) are growing over time [2, 5], outpacing the computational advances contributed by hardware accelerators in recent years [10, 12, 17] .  \nThe massive training costs of DNNs stem from several factors. To achieve accurate recognition of complex features, DNNs must undergo extensive training on large datasets with sophisticated architectures that have numerous tunable parameters. Consequently, training complex DNNs requires a large number of arithmetic operations, typically using the conventional Stochastic Gradient Descent (SGD) or any ofits variants. This is partly due to challenges like the vanishing gradient problem, which leads to slow convergence, and the need for techniques to avoid getting trapped in local minima. While non-linearities in DNNs are crucial for capturing real-world phenomena, they also increase the computational demands and training time for large networks.  \nIn addition to the previous discussion, for many applications data evolves dynamically over time, necessitating re-training. The consequence is that, since DNNs learn via some method that minimizes ","cbCaiozdnl4jPC2v","https://ap.wps.com/l/cbCaiozdnl4jPC2v","pdf",769618,1,7,"English","en",105,"# Abstract\n# Introduction\n## Training costs and challenges\n## Motivation for re-training over evolving data\n## Proposed linear-model approach\n# Contributions\n## Active paths and activation patterns\n## Hypothesis for decoupling SK and QK\n## Proof-of-Concept validation","[{\"question\":\"What does the paper mean by decoupling structural and quantitative knowledge in ReLU-based DNNs?\",\"answer\":\"Structural knowledge (SK) is the ability to activate and deactivate network paths via non-linear structure, while quantitative knowledge (QK) is the numerical weight and bias parameters. The approach trains or updates these aspects separately.\"},{\"question\":\"How does the proposed method emulate ReLU piece-wise linear behavior?\",\"answer\":\"It activates a specific subset of a linear model for each input sample. Different active subsets across samples reproduce the piece-wise linear outputs generated by ReLU activations.\"},{\"question\":\"Why is this decoupling useful for re-training?\",\"answer\":\"By separating activation-related structure from parameter updates, the method targets training and re-training that can be faster and less costly, supporting incremental and federated re-training using models trained on different datasets.\"}]","Decoupling Structural and Quantitative Knowledge in ReLU-based Deep Neural Networks | PDF",1785729589,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"decoupling-structural-and-quantitative-knowledge-in-relu-based-deep-neural-networks","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/decoupling-structural-and-quantitative-knowledge-in-relu-based-deep-neural-networks/120345/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What does the paper mean by decoupling structural and quantitative knowledge in ReLU-based DNNs?","Question",{"text":75,"@type":76},"Structural knowledge (SK) is the ability to activate and deactivate network paths via non-linear structure, while quantitative knowledge (QK) is the numerical weight and bias parameters. The approach trains or updates these aspects separately.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed method emulate ReLU piece-wise linear behavior?",{"text":80,"@type":76},"It activates a specific subset of a linear model for each input sample. Different active subsets across samples reproduce the piece-wise linear outputs generated by ReLU activations.",{"name":82,"@type":73,"acceptedAnswer":83},"Why is this decoupling useful for re-training?",{"text":84,"@type":76},"By separating activation-related structure from parameter updates, the method targets training and re-training that can be faster and less costly, supporting incremental and federated re-training using models trained on different datasets.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]