[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119228-en":3,"doc-seo-119228-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119228,687197207639,"Asher","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","On the Temperature of Machine Learning Systems","A thermodynamic theory for machine learning (ML) systems is developed by treating ML systems as analogs of physical systems with energy and entropy. The framework defines system states, interprets model training and refresh as state phase transitions, and connects initial potential energy to model loss functions with a minimum-potential-energy principle. For diverse energy forms and initialization strategies, the temperature during phase transition is derived analytically and asymptotically, serving as an indicator of data distribution and training complexity. Deep neural networks are modeled as layered heat engines, and work efficiency is tied to activation functions to classify heat-engine types.","arXiv :2404 . 13218v1 [ cs .LG] 19 Apr 2024  \nOn the Temperature of Machine Learning Systems  \nDong Zhang ∗  \nApril 23, 2024  \nAbstract  \nWe develop a thermodynamic theory for machine learning (ML) systems. Similar to physical thermodynamic systems which are characterized by energy and entropy, ML systems possess these characteristics as well. This comparison inspire us to integrate the concept of temperature into ML systems grounded in the fundamental principles of thermodynamics, and establish a basic thermodynamic framework for machine learning systems with non-Boltzmann distributions. We introduce the concept of states within a ML system, identify two typical types of state, and interpret model training and refresh as a process of state phase transition. We consider that the initial potential energy of a ML system is described by the model’s loss functions, and the energy adheres to the principle of minimum potential energy. For a variety of energy forms and parameter initialization methods, we derive the temperature of systems during the phase transition both analytically and asymptotically, highlighting temperature as a vital indicator of system data distribution and ML training complexity. Moreover, we perceive deep neural networks as complex heat engines with both global temperature and local temperatures in each layer. The concept of work efficiency is introduced within neural networks, which mainly depends on the neural activation functions. We then classify neural networks based on their work efficiency, and describe neural networks as two types of heat engines.  \nKeywords: machine learning system, thermodynamics, temperature, entropy, energy, phase transition, heat engine, work efficiency  \n1 Introduction  \nFrom the perspective of information theory, data carries entropy. The concept of entropy originated from thermodynamics and statistical mechanics, where it describes the disorder or randomness in a physical system. Claude Shannon later extended the idea of entropy to information theory to measure the uncertainty of random variables [1], while Norbert Wiener also discussed entropy in the context of cybernetics, especially differential entropy [2] . The employment of entropy in machine learning isan adaptation from information theory. For example, cross-entropy and information gain are used for splitting nodes in decision trees and random forests. In unsupervised learning, entropy can be used to evaluate the quality of clusters. Overall, the usage of entropy in data systems and machine learning is fundamentally rooted in the principles of thermodynamics and information theory, demonstrating a diverse and interdisciplinary application of the concept.  \nOn the other hand, a physical system has energy, as well as entropy. If the concept of entropy can be introduced into a data system, does data also have energy? In the field of machine learning, there is a category of models known as energy-based models (EBMs) [3, 4] . The origins of these EBMs can be traced back to the Ising model in statistical physics [5, 6] and the Amari-Hopfield network [7, 8] . The Boltzmann Machines (BMs) were proposed as stochastic recurrent neural networks [9], inspired by the Ising model as well as spin-glass model in physics [10] . To simplify the training process and improve  \n∗ Email: [dongzhanghz@gmail.com](dongzhanghz@gmail.com)  \ncomputational efficiency, the Restricted Boltzmann Machines (RBMs) were later developed [11, 12] . The RBMs introduced a restriction that the neurons must form a bipartite graph, which significantly improved the training efficiency. Since the advent of RBMs, a variety of methods and applications have been proposed under the umbrella of EBMs, contributing to the evolution and expansion of this field [13, 14, 15, 16, 17, 18, 19] . The fundamental concept of an EBM is to define an energy function that satisfies Eµ(x) = −log pµ(x), or pµ(x) = exp[−Eθ(x)/Zθ], where Eµ(x) is the energy function with parameter set µ","cbCaip2JIZ6ATR2z","https://ap.wps.com/l/cbCaip2JIZ6ATR2z","pdf",1736265,1,44,"English","en",105,"# Introduction\n## Entropy in information theory and machine learning\n## Energy-based models and generalized energy\n## Temperature-like quantities in ML systems","[{\"question\":\"How does the document define a thermodynamic view of machine learning systems?\",\"answer\":\"It models ML systems using thermodynamic quantities, treating energy and entropy as analogs of physical systems. This enables integrating temperature into ML from thermodynamic principles.\"},{\"question\":\"What role do model loss functions play in the proposed framework?\",\"answer\":\"The document states that the initial potential energy of an ML system is described by the model’s loss functions. It then uses a minimum potential energy principle to guide the energy behavior.\"},{\"question\":\"How is temperature characterized during model phase transitions?\",\"answer\":\"Temperature is derived both analytically and asymptotically for different energy forms and parameter initialization methods. It is presented as a key indicator of data distribution and training complexity.\"}]","On the Temperature of Machine Learning Systems | PDF",1785723202,111,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"on-the-temperature-of-machine-learning-systems","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/on-the-temperature-of-machine-learning-systems/119228/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does the document define a thermodynamic view of machine learning systems?","Question",{"text":75,"@type":76},"It models ML systems using thermodynamic quantities, treating energy and entropy as analogs of physical systems. This enables integrating temperature into ML from thermodynamic principles.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What role do model loss functions play in the proposed framework?",{"text":80,"@type":76},"The document states that the initial potential energy of an ML system is described by the model’s loss functions. It then uses a minimum potential energy principle to guide the energy behavior.",{"name":82,"@type":73,"acceptedAnswer":83},"How is temperature characterized during model phase transitions?",{"text":84,"@type":76},"Temperature is derived both analytically and asymptotically for different energy forms and parameter initialization methods. It is presented as a key indicator of data distribution and training complexity.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]