[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117588-en":3,"doc-seo-117588-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117588,1374391974468,"Eden","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Exploiting the conservation of probability in simple machine learning models","The thesis investigates the development of simple unsupervised machine learning models with two layers, designed for clarity and interpretability. A low-dimensional hidden “coding” layer is connected to the visible data layer, and the model executes a probability-preserving jump from hidden to visible states. The proposed Markov Chain Machines (MCMs) generalize Restricted Boltzmann Machines by decoupling latent-state distribution from conditional visible distribution. Mathematical formulation, training, and initialization are provided, alongside entropy-based merging and recursive splitting. Experiments on MNIST show faster convergence, improved likelihood maximization, better hidden unit utilization, smoother likelihood landscapes, and more interpretable weights, supporting applications in clustering, generative modeling, and structured representation learning.","University of Padova  \nDepartment of Physics and Astronomy ”Galileo Galilei”  \nMaster Thesis in Physics of Data  \nExploiting the conservation of  \nprobability in simple machine learning  \nmodels  \nSupervisor Master Candidate  \nProf. Marco Baiesi Manjodh Singh  \nUniversity of Padova  \nCo-supervisor Student ID  \nProf. Alberto Rosso 2122168  \nUniversité Paris-Saclay  \nAcademic Year  \n2024-2025  \n“The simplest explanation is usually the right one.”  \n—William of Ockham  \nAbstract  \nThe thesis investigates the development of simple unsupervised machine learning models with two layers. The first hidden layer, or coding, of low dimension is connected to the data layer. The model jumps from the hidden layer to the visible one, preserving probability, and is therefore simple and easy to understand. The proposed models, referred to as Markov Chain Machines (MCMs), build upon and generalize the structure of Restricted Boltzmann Machines (RBMs) by decoupling the latent-state distribution from the conditional distribution over visible states. This disentanglement improves interpretability, optimizes latent space utilization, and facilitates more stable training dynamics. The model represents essentially a rediscovery ofSigmoid Belief Networks. We provide a mathematical formulation of MCMs, their training procedure, and initialization strategies, and explore two novel extensions: entropy-based merging and recursive splitting. Numerical experiments on the MNIST dataset show that MCMs outperform RBMs in convergence speed, likelihood maximization, and hidden unit utilization, while producing smoother likelihood landscapesand more interpretable weights. The findings suggest promising applications of MCMs in clustering, generative modeling, and structured representation learning.  \nContents  \nAbstract iii  \nList of figures vii  \nListing of acronyms ix  \n1 Introduction 1  \n2 Markovian Generative Models 3  \n2.1 General Model Structure ................................... 3  \n2.2 Energy-based Markovian models ............................... 6  \n2.3 Bernoulli Mixture model ................................... 9  \n3 An Overview of Restricted Boltzmann Machines 13  \n3.1 Physical derivation of RBMs ................................. 14  \n3.2 Restricted Boltzmann Machines ............................... 16  \n3.2.1 Training objective .................................. 17  \n3.2.2 Log-likelihood gradients .............................. 18  \n3.2.3 Exact training .................................... 18  \n3.2.4 Approximate training: Contrastive Divergence ................... 19  \n3.2.5 Weight initialization ................................ 21  \n3.3 Clustering and composition .................................. 22  \n4 Markov Chain Machines 23  \n4.1 Motivation: decoupling sampling frequencies and jump probabilities ............ 23  \n4.2 Gradient computation .................................... 25  \n4.3 Weight initialization ...................................... 27  \n4.4 One hot encoding restriction ................................. 28  \n5 Unsupervised Subclasses learning 31  \n5.1 Entropy-based merging .................................... 31  \n5.2 Recursive learning ....................................... 33  \n6 Numerical results: MNIST 37  \n6.1 Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38  \n6.1.1 PCA representation ................................. 40  \n6.2 Clustering ........................................... 44  \n6.3 Composition ......................................... 44  \n7 Conclusion 51  \nReferences 53  \nAcknowledgments 55  \nListing of figures  \n2.1 Graphical representation of the Markovian generative process: a latent variable is sampled from p (z) and used to generate visible data via p (x | z) ....................... 4  \n3.1 Architectural illustration ofa RBM demonstrating the key structural constraint that gives the  \nmodel its name. The network consists of two layers: a visible layer with 784 units (","cbCaigShi6xtpckf","https://ap.wps.com/l/cbCaigShi6xtpckf","pdf",1645083,1,67,"English","en",105,"# 1 Introduction\n# 2 Markovian Generative Models\n## 2.1 General Model Structure\n## 2.2 Energy-based Markovian models\n## 2.3 Bernoulli Mixture model\n# 3 An Overview of Restricted Boltzmann Machines\n## 3.1 Physical derivation of RBMs\n## 3.2 Restricted Boltzmann Machines\n# 4 Markov Chain Machines\n## 4.1 Motivation: decoupling sampling frequencies and jump probabilities\n## 4.2 Gradient computation\n## 4.3 Weight initialization\n## 4.4 One hot encoding restriction\n# 5 Unsupervised Subclasses learning\n## 5.1 Entropy-based merging\n## 5.2 Recursive learning\n# 6 Numerical results: MNIST\n## 6.1 Reconstruction\n## 6.2 Clustering\n## 6.3 Composition\n# 7 Conclusion\n# References\n# Acknowledgments","[{\"question\":\"What are Markov Chain Machines (MCMs) and how do they relate to Restricted Boltzmann Machines (RBMs)?\",\"answer\":\"MCMs are simple two-layer unsupervised models that preserve probability through a jump from a low-dimensional hidden coding layer to visible states. They generalize RBMs by decoupling the latent-state distribution from the conditional distribution over visible states.\"},{\"question\":\"What improvements does decoupling provide in MCMs?\",\"answer\":\"Decoupling enhances interpretability, optimizes utilization of the latent space, and supports more stable training dynamics. It also leads to smoother likelihood landscapes compared with RBMs in the reported experiments.\"},{\"question\":\"What novel extensions of MCMs are explored, and what datasets are used for evaluation?\",\"answer\":\"The thesis investigates entropy-based merging and recursive splitting as two novel extensions. Numerical experiments are conducted on the MNIST dataset, evaluating convergence speed, likelihood maximization, and latent unit utilization.\"}]","Exploiting the conservation of probability in simple machine learning models | PDF",1785677118,169,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"exploiting-the-conservation-of-probability-in-simple-machine-learning-models","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/exploiting-the-conservation-of-probability-in-simple-machine-learning-models/117588/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What are Markov Chain Machines (MCMs) and how do they relate to Restricted Boltzmann Machines (RBMs)?","Question",{"text":75,"@type":76},"MCMs are simple two-layer unsupervised models that preserve probability through a jump from a low-dimensional hidden coding layer to visible states. They generalize RBMs by decoupling the latent-state distribution from the conditional distribution over visible states.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What improvements does decoupling provide in MCMs?",{"text":80,"@type":76},"Decoupling enhances interpretability, optimizes utilization of the latent space, and supports more stable training dynamics. It also leads to smoother likelihood landscapes compared with RBMs in the reported experiments.",{"name":82,"@type":73,"acceptedAnswer":83},"What novel extensions of MCMs are explored, and what datasets are used for evaluation?",{"text":84,"@type":76},"The thesis investigates entropy-based merging and recursive splitting as two novel extensions. Numerical experiments are conducted on the MNIST dataset, evaluating convergence speed, likelihood maximization, and latent unit utilization.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]