[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-297057-105":59,"doc-detail-297057-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","on-the-learning-of-non-autoregressive-transformers-paper","On the Learning of Non-Autoregressive Transformers - Paper","","Non-autoregressive Transformers (NAT) aim to reduce decoding latency by predicting full sentences in parallel, yet this parallelism weakens left-to-right dependency modeling and makes training difficult. The work presents theoretical and empirical analyses of NAT learning challenges and unifies prior success patterns. It shows likelihood-only training approximates marginal distributions while dropping token dependencies measured by conditional total correlation. It then reformulates objectives in a proxy-distribution likelihood view to reduce information loss, explaining observations and enabling new method design.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/on-the-learning-of-non-autoregressive-transformers-paper/297057/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/on-the-learning-of-non-autoregressive-transformers-paper/297057.png","ImageObject",300,407,{"name":92,"@type":93},"Mason","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-11","2026-09-18",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",13,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"Why is NAT learning challenging compared with autoregressive Transformers?","Question",{"text":113,"@type":114},"NATs predict tokens in parallel under an independence assumption, which drops left-to-right dependencies. This makes standard likelihood training lead to poor generation quality.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What is the key insight about maximizing likelihood in NATs?",{"text":118,"@type":114},"Maximizing likelihood can approximate marginal distributions but eliminates dependencies between tokens. The lost dependency information can be quantified using the dataset’s conditional total correlation.",{"name":120,"@type":111,"acceptedAnswer":121},"How do the authors explain why some NAT objectives work despite low likelihood?",{"text":122,"@type":114},"They formalize prior objectives within a unified framework showing their success can be seen as maximizing likelihood on a proxy distribution. This view leads to reduced information loss and matches empirical results.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},297057,1790472398,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":145,"read_time":146,"preview_page_count":4},5909887256941,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","On the Learning of Non-Autoregressive Transformers  \nFei Huang * 1 † Tianhua Tao * 1 Hao Zhou 2 † Lei Li 3 Minlie Huang 1  \nAbstract  \nNon-autoregressive Transformer (NAT) is a family of text generation models, which aims to reduce the decoding latency by predicting the whole sentences in parallel. However, such latency reduction sacrifices the ability to capture left-toright dependencies, thereby making NAT learning very challenging. In this paper, we present theoretical and empirical analyses to reveal the challenges of NAT learning and propose a unified perspective to understand existing successes. First, we show that simply training NAT by maximizing the likelihood can lead to an approximation of marginal distributions but drops all dependencies between tokens, where the dropped information can be measured by the dataset’s conditional total correlation. Second, we formalize many previous objectives in a unified framework and show that their success can be concluded as maximizing the likelihood on a proxy distribution, leading to a reduced information loss. Empirical studies show that our perspective can explain the phenomena in NAT learning and guide the design of new training methods.  \n1. Introduction  \nNon-Autoregressive Transformers (NATs, Gu et al., 2018 ; Gu et al., 2019 ; Ma et al., 2019 ; Ding et al., 2021a ; Gu & Kong, 2021) have received growing attention due to their significantly lower decoding latency and approaching accuracy compared to the autoregressive Transformers (ATs)  \n*Equal contribution †This work is done while Fei Huang was a research intern and Hao Zhou was a research scientist at ByteDance AI Lab. 1The CoAI group, Tsinghua University. Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, DCST, Tsinghua University. 2Institute for AI Industry Research, Tsinghua University. 3University of California Santa Barbara. Correspondence to: Hao Zhou \u003C[haozhou0806@gmail.com](haozhou0806@gmail.com) >, Minlie Huang \u003Cai[huang@tsinghua.edu.cn](huang@tsinghua.edu.cn) >.  \nProceedings of the 39 th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 2022 . Copyright 2022 by the author(s) .  \nWe ight space direction 2  \n~~ ~~ 4.4  \n~~ ~~ 5.0  \n~~ ~~ 5.7  \n~~ ~~ 6.6  \n~~ ~~ 8.0  \n~~ ~~ 9.8  \nWe ight space direction 2  \n20  \n18  \n16  \n14  \n10  \n6  \n0  \nFigure 1: The log-likelihood and the BLEU score on a 2D section in NAT’s weight space. GLAT+KD (⋆) and MLE ( ▲) are trained via different objectives starting from an initial checkpoint (♦) for 10k steps, where MLE achieve higher log-likelihood but lower BLEU, and GLAT is the opposite. Each point of the contour map is linearly interpolated from the three checkpoints. All values are evaluated on one translation benchmark, the validation set of WMT14 En-De.  \nin text generation (Qian et al., 2021b ; Huang et al., 2022b) . NATs generate the whole sequence in parallel based on the assumption that each token can be predicted independently. However, unlike ATs that can be easily trained via Maximum Likelihood Estimation (MLE), NAT learning is very challenging because it drops the left-to-right dependencies. Gu et al. (2018) show that directly training NATs via MLE leads to implausible outputs with repeated tokens, revealing their inability to preserve the consistency in generated texts.  \nTo address the problem, many training methods have been proposed. For example, knowledge distillation (KD, Kim & Rush, 2016 ; Gu et al., 2018) supervises NATs with target sentences distilled from an AT teacher model. GLAT (Qian et al., 2021a) improves the training by utilizing a masked language model objective. These methods only change the training objectives without modifying the model, but they demonstrate significant improvements in generation quality.  \nDespite the empirical successes in NAT learning, there still exists a surprising characteristic no","cbCaip1Z5DUxldKq","https://ap.wps.com/l/cbCaip1Z5DUxldKq","pdf",937112,21,"English","# Abstract\n# 1. Introduction\n## Background: NATs vs. autoregressive Transformers\n## Challenges of likelihood-based NAT learning\n## Prior methods: knowledge distillation and GLAT\n## Empirical phenomenon motivating two research questions\n## Motivation from likelihood vs. generation quality figures","[{\"question\":\"Why is NAT learning challenging compared with autoregressive Transformers?\",\"answer\":\"NATs predict tokens in parallel under an independence assumption, which drops left-to-right dependencies. This makes standard likelihood training lead to poor generation quality.\"},{\"question\":\"What is the key insight about maximizing likelihood in NATs?\",\"answer\":\"Maximizing likelihood can approximate marginal distributions but eliminates dependencies between tokens. The lost dependency information can be quantified using the dataset’s conditional total correlation.\"},{\"question\":\"How do the authors explain why some NAT objectives work despite low likelihood?\",\"answer\":\"They formalize prior objectives within a unified framework showing their success can be seen as maximizing likelihood on a proxy distribution. This view leads to reduced information loss and matches empirical results.\"}]","On the Learning of Non-Autoregressive Transformers - Paper | PDF",1789740013,53]