[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-128813-105":3,"detail-sidebar-cat-0-en-105":73,"doc-detail-128813-en":122},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":66,"head_meta":68,"extra_data":70,"updated_unix":72},105,"en","natural-gradient-methods-in-statistics-and-machine-learning-doctoral-dissertation","Natural Gradient Methods in Statistics and Machine Learning - Doctoral Dissertation","","Many inference tasks can be formulated as optimization problems, where standard gradient-based methods often require many iterations and become computationally expensive. This thesis develops efficient approaches based on natural gradient methods to accelerate convergence. It introduces a natural-gradient-based optimizer for probability distribution parameters via applications to maximum likelihood estimation and variational inference, covering skew-elliptical families, elliptical copulas, and mixture models. It also presents new expectation propagation variants with improved convergence and easier tuning for black-box inference.",{"@graph":14,"@context":65},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/natural-gradient-methods-in-statistics-and-machine-learning-doctoral-dissertation/128813/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/natural-gradient-methods-in-statistics-and-machine-learning-doctoral-dissertation/128813.png","ImageObject",300,407,{"name":42,"@type":43},"Maeve","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-09-17","2026-08-06",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",9,{"@type":57,"mainEntity":58},"FAQPage",[59],{"name":60,"@type":61,"acceptedAnswer":62},"What are the advantages of the new expectation propagation (EP) variants proposed in the thesis?","Question",{"text":63,"@type":64},"Using a natural-gradient interpretation of Minka’s EP algorithm, the thesis motivates two new EP variants tailored for black-box inference. They converge faster, are easier to tune, and avoid using debiasing estimators.","Answer","https://schema.org",{"og:url":32,"og:type":67,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":69,"canonical":32},"index,follow",{"doc_id":71,"site_id":7},128813,1786003643,{"code":4,"msg":74,"data":75},"success",[76,80,84,88,93,98,103,107,111,114,118],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":77,"show_sort_weight":78,"slug":79},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":81,"show_sort_weight":82,"slug":83},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Exam",70,"exam",{"id":89,"doc_module":4,"doc_module_name":25,"category_name":90,"show_sort_weight":91,"slug":92},5,"Comic",60,"comic",{"id":94,"doc_module":4,"doc_module_name":25,"category_name":95,"show_sort_weight":96,"slug":97},6,"Technology",50,"technology",{"id":99,"doc_module":4,"doc_module_name":25,"category_name":100,"show_sort_weight":101,"slug":102},7,"Healthcare",40,"healthcare",{"id":104,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":105,"slug":106},8,30,"research-report",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},"Religion & Spirituality",20,"religion-spirituality",{"id":109,"doc_module":4,"doc_module_name":25,"category_name":112,"show_sort_weight":109,"slug":113},"World Cup","world-cup",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":115,"slug":117},10,"Lifestyle","lifestyle",{"id":119,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":89,"slug":121},19,"General","general",{"code":4,"msg":74,"data":123},{"doc_id":71,"user_id":124,"nickname":42,"user_avatar":125,"doc_module":4,"category_id":104,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":126,"file_id":127,"file_url":128,"file_type":129,"file_size":130,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":131,"language":132,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":133,"faqs":134,"seo_title":135,"seo_description":12,"update_tm":72,"read_time":136},2336474466712,"https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd","Natural gradient methods in statistics and machine learning  \nJonathan Jak Sum So  \nDepartment of Engineering  \nUniversity of Cambridge  \nThis dissertation is submitted for the degree of Doctor of Philosophy  \nDarwin College April 2025  \nDeclaration  \nThis thesis is the result of my own work and includes nothing which is the outcome of work done in collaboration except as declared in the preface and specified in the text. It is not substantially the same as any work that has already been submitted, or, is being concurrently submitted, for any degree, diploma or other qualification atthe University of Cambridge or any other University or similar institution except as declared in the preface and specified in the text. It does not exceed the prescribed word limit for the relevant Degree Committee.  \nJonathan Jak Sum So April 2025  \nAbstract  \nMany inference problems can be expressed as optimisations. These optimisations can often be challenging, requiring many iterations to converge using standard methods, making them computationally demanding. In this thesis, we present methods for performing such optimisations efficiently, drawing heavily on the use of natural gradient methods (Amari, 1998) .  \nIn the first main contribution chapter of this thesis, we present a novel naturalgradient-based method for optimising the parameters of probability distributions where a direct application of natural gradients would be computationally demanding. We apply this method to maximum likelihood estimation and variational inference tasks involving a number of distributions. These include: skew-elliptical distributions, which can model multivariate real-valued data with characteristics beyond those which can be captured by the Gaussian family, such as asymmetry or heavy-tailedness; elliptical copulas, which are commonly used for modelling correlation structure in high-dimensional statistics; and various mixture distributions, which represent complex probability distributions as combinations of simpler distributions, allowing for features such as multimodality, which may not be possible to represent in the component distributions alone. Our method expands the set of distribution families that can efficiently be targeted with natural gradients, and as we demonstrate, can result insignificantly faster convergence than standard methods.  \nIn the second main contribution chapter, we use a novel natural-gradient interpretation of the expectation propagation (EP) algorithm of Minka (2001) to motivate two new natural-gradient-based EP variants that have particularly desirable properties in black-box inference settings. Black-box inference methods allow practitioners to answer questions of inference without requiring expert knowledge of the underlying inference techniques, and typically place few restrictions on the model of interest. EP has several desirable computational properties in black-box settings, but existing EP variants face multiple challenges. Our new variants have several advantages over their predecessors that allow them to address these challenges. Namely, they converge faster, are easier to tune, and do not make use of debiasing estimators. By facilitating the use of EP in  \nsuch settings, our advances have the potential to reduce the computational demands of performing black-box inference.  \nWe hope that our contributions will prove to be useful in their own right, and also that they may facilitate or inspire further advances in statistics, machine learning, or indeed any other field in which problems of inference are to be found.  \nAcknowledgements  \nI am grateful to all the members of the CBL (past and present) for helping to create such a wonderful environment for pursuing a PhD. The people are too numerous to list, but I will always cherish the time I have spent here. I would like to thank my friend and collaborator Hermanni for introducing me to the world of identifiable generative models, and for accompanying me in my quest to find","cbCaijsLeOpHltKI","https://ap.wps.com/l/cbCaijsLeOpHltKI","pdf",3050941,182,"English","# Abstract\n# Declaration\n# Main Contributions\n## Natural-gradient-based optimisation for distribution parameters\n## Natural-gradient interpretation and new EP variants\n# Acknowledgements","[{\"question\":\"What are the advantages of the new expectation propagation (EP) variants proposed in the thesis?\",\"answer\":\"Using a natural-gradient interpretation of Minka’s EP algorithm, the thesis motivates two new EP variants tailored for black-box inference. They converge faster, are easier to tune, and avoid using debiasing estimators.\"}]","Natural Gradient Methods in Statistics and Machine Learning - Doctoral Dissertation | PDF",459]