[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85426-en":3,"doc-seo-85426-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85426,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","Memory Savings at What Cost: A Study of Alternatives to Backpropagation","Forward-mode automatic differentiation (FMAD) and zero-order (ZO) optimization are proposed as memory-efficient, backpropagation-free alternatives for large language model fine-tuning, but prior evaluations typically benchmark only against vanilla backpropagation and omit key memory-efficient variants like activation checkpointing. This study provides unified theoretical and empirical comparisons of BP, checkpointed BP, FMAD, and ZO for LLMs and vision-language models, revealing distinct trade-offs in memory, computational cost, convergence speed, accuracy, and failure modes under perturbation constraints.","Memory Savings at What Cost? A Study of Alternatives to Backpropagation  \nKunjal Panchal 1 Sunav Choudhary 2 Yuriy Brun 1 Hui Guan 1  \narXiv :2506 .2 1833v2 [ cs .LG] 10 Jul 2026  \nAbstract  \nForward-mode automatic differentiation (FMAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagationfree alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing. We present a unified theoretical and empirical comparison of BP, checkpointed BP, FMAD, and ZO for LLM and vision-language model training, showing that while FMAD and ZO reduce activation memory, they trade memory for higher computational cost and longer wallclock time to convergence, resulting in lower accuracy and slower training, especially under constrained perturbation budgets. Across models, BP with checkpointing outperforms FMAD and ZO variants, including variance-reduced methods, achieving up to 31.1% higher accuracy, 34.8% faster convergence, and 3.8 × fewer computationsat comparable memory usage, while also revealing instability-related failure modes in FMAD and ZO. Overall, our results correct a one-sided benchmarking narrative by showing that memoryefficient methods entail fundamentally different trade-offs, and that ignoring these distinctions has led to misleading conclusions about LLM optimization in prior work. Our source code is available at [https://github.com/Astuary/](https://github.com/Astuary/)[ ](https://github.com/Astuary/)Gradient  Estimation Methods.  \n1. Introduction  \nBackpropagation (BP) (Rumelhart et al., 1986) is the standard algorithm for gradient computation in deep learning 1College of Information and Computer Sciences, University of Massachusetts, Amherst 2Adobe, San Jose. Correspondence to:  \nKunjal Panchal \u003C[kpanchal@umass.edu](kpanchal@umass.edu)>.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \nand remains the dominant approach for training and finetuning large language models (LLMs), due to its convergence efficiency and widespread support in automatic differentiation frameworks such as PyTorch (Paszke et al., 2019) and JAX (Bradbury et al., 2018) . However, BP incurs high memory overhead when training large models such as LLMs, as it must store intermediate activations for the backward pass. To address this limitation, recent research has explored alternative gradient estimation methods for LLM optimization, including forward-mode automatic differentiation (FMAD) (Baydin et al., 2017 ; 2022 ; Panchal et al., 2024) and zero-order (ZO) optimization (Richardson, 1955 ; Malladi et al., 2023), which approximate gradients using directional derivatives or finite-difference evaluations under random weight perturbations. These methods are often promoted as memory-efficient or hardware-friendly alternatives to BP for LLM training and fine-tuning, particularly in resource-constrained or non-differentiable settings (Panchalet al., 2024 ; Xu et al., 2024 ; Malladi et al., 2023) .  \nDespite growing interest, prior work on FMAD and ZO for LLM training and fine-tuning suffers from two critical limitations that leave their practical value inadequately understood. First, existing comparisons (Gautam et al., 2024 ; Zhang et al., 2024) often overlook activation checkpointing (Chen et al., 2016), a widely supported and effective BP variant for large-scale models that substantially reduces memory usage by recomputing rather than storing intermediate activations. Second, as shown in Table 1, key considerations for LLM optimization (such as computational cost and wall-clock time to convergence) are frequently omitted, leaving even comparisons against vanilla BP incomplete. This one-sided narrative of ZO and FMAD as superior to BP in LLM settings motivates our study: we aim ","cbCainhA4t1SHGSv","https://ap.wps.com/l/cbCainhA4t1SHGSv","pdf",2352456,1,44,"English","en",105,"# Introduction\n## Background: Backpropagation and memory overhead\n## Alternative gradient estimation methods (FMAD, ZO)\n## Limitations in prior comparisons\n# Comparative study: BP vs checkpointed BP vs FMAD vs ZO\n## Trade-offs across memory, compute, and convergence\n## Empirical benchmarks and metrics","[{\"question\":\"What motivation does the paper give for studying FMAD and ZO versus backpropagation?\",\"answer\":\"FMAD and ZO are promoted as memory-efficient, backpropagation-free alternatives for fine-tuning large language models, but their benefits are often assessed only against vanilla backpropagation.\"},{\"question\":\"What key limitations in prior work does the paper address?\",\"answer\":\"Prior comparisons often omit activation checkpointing and frequently exclude important optimization metrics such as computational cost and wall-clock convergence time, leading to an incomplete and one-sided narrative.\"},{\"question\":\"What main conclusion does the study reach about memory-efficient methods?\",\"answer\":\"Memory-efficient approaches like FMAD and ZO reduce activation memory but incur higher computational cost and slower convergence, and checkpointed backpropagation outperforms them on accuracy and efficiency under comparable memory usage.\"}]",1784203378,111,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"memory-savings-at-what-cost-a-study-of-alternatives-to-backpropagation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/memory-savings-at-what-cost-a-study-of-alternatives-to-backpropagation/85426/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What motivation does the paper give for studying FMAD and ZO versus backpropagation?","Question",{"text":74,"@type":75},"FMAD and ZO are promoted as memory-efficient, backpropagation-free alternatives for fine-tuning large language models, but their benefits are often assessed only against vanilla backpropagation.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What key limitations in prior work does the paper address?",{"text":79,"@type":75},"Prior comparisons often omit activation checkpointing and frequently exclude important optimization metrics such as computational cost and wall-clock convergence time, leading to an incomplete and one-sided narrative.",{"name":81,"@type":72,"acceptedAnswer":82},"What main conclusion does the study reach about memory-efficient methods?",{"text":83,"@type":75},"Memory-efficient approaches like FMAD and ZO reduce activation memory but incur higher computational cost and slower convergence, and checkpointed backpropagation outperforms them on accuracy and efficiency under comparable memory usage.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]