[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86505-en":3,"doc-seo-86505-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},86505,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","M+Adam Low-Precision Training via Additive–Multiplicative Optimization","Training with quantized weights can cut memory, compute, and energy costs, yet accuracy often degrades when optimization runs directly in low precision without high-precision master weights. M+Adam identifies a core failure: standard additive optimizers can stall under low precision because coarse mantissa resolution rounds updates to zero at large weight magnitudes. Additive and multiplicative update modes fail complementarily; M+Adam combines them. Monotone descent is proven under standard smoothness assumptions, and experiments on LLaMA-style pretraining show consistent gains using BF16, FP8, and FP4 master weights.","M+Adam: Low-Precision Training via Additive–Multiplicative Optimization  \nXiaoyuan Liang 1 Sebastian Loeschcke 2 Mads Toftrup 3 Anima Anandkumar 1  \narXiv :2607 . 106 1 1v 1 [ cs .LG] 12 Jul 2026  \nAbstract  \nTraining with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and not make progress, especially at large weight magnitudes due to coarse mantissa resolution. To overcome this, multiplicative updates have been previously proposed, in place of additive updates in standard optimizers. While successful under extremely low precision, such as under the logarithmic number system, they suffer from failures near zero and across sign changes. The failure modes of additive and multiplicative updates are therefore complementary. To exploit this, we propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are zeroed out under rounding. We prove monotone descent for M+Adam under standard smoothness assumptions.  \nAcross LLaMA-style pretraining with 60M–1B models, 1–8 × Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.  \n1. Introduction  \nTraining large language models (LLMs) in full precision (FP32) incurs substantial memory, compute, and energy costs. Modern accelerators are optimized for reducedprecision formats such as FP8 and FP4, via specialized hardware like tensor cores (Micikevicius et al., 2022) . As a result, most large-scale training pipelines adopt mixed-precision training, executing forward and backward passes in reduced precision (Micikevicius et al., 2018) . However, they still  \n1 California Institute of Technology 2University of Copenhagen 3Aarhus University. Correspondence to: Xiaoyuan Liang \u003C[xliang2@caltech.edu](xliang2@caltech.edu) >.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \nAdam (additive)  \nMadam (multiplicative)  \nQuantization bins under FP system  \nFigure 1. Additive (Adam), multiplicative (Madam), and combined M+Adam updates under low precision. FP grids under low precision cause additive updates to be lost to rounding at large magnitudes and multiplicative updates to stall near zero, while M+Adam avoids both failure modes.  \n\n| Adam Madam M+ADAM |  |  |  |\n| --- | --- | --- | --- |\n| Additive updates | ✓ | × | ✓ |\n| Multiplicative updates | × | ✓ | ✓ |\n| Can escape zero | ✓ | × | ✓ |\n| Non-zero update at large |w| | × | ✓ | ✓ |\n\nTable 1. Comparison of update mechanisms and their failure modes across the studied optimizers.  \nretain high-precision copies (known as master weights) for optimization, since training directly in low precision results in significant degradation of accuracy and is often unstable (Zamirai et al., 2021) .  \nRecently, the precision of master weights has been reduced, e.g. BF16 instead of FP32, using techniques such as stochastic rounding (SR) (Connolly et al., 2021) to keep them unbiased. While this reduces the memory overhead compared to FP32 master weights, they still exhibit degraded convergence or instability relative to full-precision FP32 training. This is because standard optimizers, such as Adam, fail to make progress at large weight magnitudes under low precision (larger quantization bins in Fig.2) , since updates smaller than the local quantization step are rounded to zero.  \nTo overcome this failure mode, alternative optimizers have been proposed based on multiplicative updates, such as Madam (Bernstein et al., 2020) . These are effective at taking larger steps at higher magnitudes (larger quantization bins in Fig.2), where standard optimizers with additive ","cbCaicny9c21UM5O","https://ap.wps.com/l/cbCaicny9c21UM5O","pdf",1899785,3,1,45,"English","en",105,"# Abstract\n# Introduction\n## Failure modes in additive and multiplicative updates\n## Proposed optimizer: M+Adam","[{\"question\":\"What failure mode prevents standard optimizers from working well in low precision?\",\"answer\":\"Under low precision, additive updates can be rounded to zero because coarse mantissa resolution makes updates smaller than the local quantization step ineffective, especially at large weight magnitudes.\"},{\"question\":\"Why do multiplicative optimizers like Madam have different failure modes?\",\"answer\":\"Multiplicative updates can stall near zero and struggle across sign changes because relative updates may not produce enough absolute change to move weights through zero when the sign must flip.\"},{\"question\":\"How does M+Adam combine additive and multiplicative updates to improve training?\",\"answer\":\"M+Adam uses additive (Adam-style) steps to handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are lost to rounding.\"}]",1784212242,113,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"madam-low-precision-training-via-additivemultiplicative-optimization","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/madam-low-precision-training-via-additivemultiplicative-optimization/86505/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-26","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What failure mode prevents standard optimizers from working well in low precision?","Question",{"text":75,"@type":76},"Under low precision, additive updates can be rounded to zero because coarse mantissa resolution makes updates smaller than the local quantization step ineffective, especially at large weight magnitudes.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why do multiplicative optimizers like Madam have different failure modes?",{"text":80,"@type":76},"Multiplicative updates can stall near zero and struggle across sign changes because relative updates may not produce enough absolute change to move weights through zero when the sign must flip.",{"name":82,"@type":73,"acceptedAnswer":83},"How does M+Adam combine additive and multiplicative updates to improve training?",{"text":84,"@type":76},"M+Adam uses additive (Adam-style) steps to handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are lost to rounding.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]