[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-297058-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-297058-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","on-the-learning-of-non-autoregressive-transformers","On the Learning of Non-Autoregressive Transformers","","Non-autoregressive Transformers (NATs) aim to reduce decoding latency by predicting entire sentences in parallel, but this parallelism makes learning difficult because left-to-right dependencies are not captured. The paper provides theoretical and empirical analyses of NAT learning challenges and a unified viewpoint explaining existing successes. It shows that likelihood-maximizing training approximates marginal distributions while removing token dependencies, and the removed information can be quantified. It further formalizes prior objectives as likelihood maximization on a proxy distribution to reduce information loss, and experiments validate the framework.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/on-the-learning-of-non-autoregressive-transformers/297058/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/on-the-learning-of-non-autoregressive-transformers/297058.png","ImageObject",300,407,{"name":42,"@type":43},"\tWilliam","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-11","2026-09-18",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",12,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why is learning NATs more challenging than training autoregressive Transformers?","Question",{"text":63,"@type":64},"NATs predict tokens in parallel and drop left-to-right dependencies, so the independent prediction assumption makes likelihood-based training ineffective at capturing the needed structure.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What problem arises when training NATs by maximizing likelihood (MLE)?",{"text":68,"@type":64},"MLE training approximates marginal distributions but removes dependencies between tokens, which can be measured using the dataset’s conditional total correlation, leading to poorer generation quality despite higher likelihood.",{"name":70,"@type":61,"acceptedAnswer":71},"How can earlier NAT objectives still improve generation quality despite low likelihood?",{"text":72,"@type":64},"The paper shows that many prior objectives can be framed as maximizing likelihood on a proxy distribution, which reduces information loss and improves generation quality even when the resulting likelihood remains low.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},297058,1790359384,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},5909887254083,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","On the Learning of Non-Autoregressive Transformers  \nFei Huang * 1 † Tianhua Tao * 1 Hao Zhou 2 † Lei Li 3 Minlie Huang 1  \narXiv :2206 .05975v1 [ cs .CL] 13 Jun 2022  \nAbstract  \nNon-autoregressive Transformer (NAT) is a family of text generation models, which aims to reduce the decoding latency by predicting the whole sentences in parallel. However, such latency reduction sacrifices the ability to capture left-toright dependencies, thereby making NAT learning very challenging. In this paper, we present theoretical and empirical analyses to reveal the challenges of NAT learning and propose a unified perspective to understand existing successes. First, we show that simply training NAT by maximizing the likelihood can lead to an approximation of marginal distributions but drops all dependencies between tokens, where the dropped information can be measured by the dataset's conditional total correlation. Second, we formalize many previous objectives in a unified framework and show that their success can be concluded as maximizing the likelihood on a proxy distribution, leading to a reduced information loss. Empirical studies show that our perspective can explain the phenomena in NAT learning and guide the design of new training methods.  \n1. Introduction  \nNon-Autoregressive Transformers (NATs, Gu et al., 2018 ; Gu et al., 2019 ; Ma et al., 2019 ; Ding et al., 2021a ; Gu & Kong, 2021) have received growing attention due to their significantly lower decoding latency and approaching accuracy compared to the autoregressive Transformers (ATs)  \n*Equal contribution †This work is done while Fei Huang was a research intern and Hao Zhou was a research scientist at ByteDance AI Lab. 1The CoAI group, Tsinghua University. Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, DCST, Tsinghua University. 2Institute for AI Industry Research, Tsinghua University. 3University of California Santa Barbara. Correspondence to: Hao Zhou \u003C[haozhou0806@gmail.com](haozhou0806@gmail.com) >, Minlie Huang \u003Cai[huang@tsinghua.edu.cn](huang@tsinghua.edu.cn) >.  \nProceedings of the 39 th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 2022 . Copyright 2022 by the author(s) .  \nWe ight space direction 2  \n~~ ~~ 4.4  \n~~ ~~ 5.0  \n~~ ~~ 5.7  \n~~ ~~ 6.6  \n~~ ~~ 8.0  \n~~ ~~ 9.8  \nWe ight space direction 2  \n20  \n18  \n16  \n14  \n10  \n6  \n0  \nFigure 1: The log-likelihood and BLEU score on a 2D section in NAT's weight space. GLAT+KD (F) and MLE (N) are trained via different objectives starting from an initial checkpoint (􀀇) for 10k steps, where MLE achieve higher log-likelihood but lower BLEU, and GLAT is the opposite. Each point of the contour map is linearly interpolated from the three checkpoints. All values are evaluated on one translation benchmark, the validation set ofWMT14 En-De.  \nin text generation (Qian et al., 2021b ; Huang et al., 2022b) . NATs generate the whole sequence in parallel based on the assumption that each token can be predicted independently. However, unlike ATs that can be easily trained via Maximum Likelihood Estimation (MLE), NAT learning is very challenging because it drops the left-to-right dependencies. Gu et al. (2018) show that directly training NATs via MLE leads to implausible outputs with repeated tokens, revealing their inability to preserve the consistency in generated texts.  \nTo address the problem, many training methods have been proposed. For example, knowledge distillation (KD, Kim & Rush, 2016 ; Gu et al., 2018) supervises NATs with target sentences distilled from an AT teacher model. GLAT (Qian et al., 2021a) improves the training by utilizing a masked language model objective. These methods only change the training objectives without modifying the model, but they demonstrate significant improvements in generation quality.  \nDespite the empirical successes in NAT learning, there still exis","cbCaineD2VIXfoOe","https://ap.wps.com/l/cbCaineD2VIXfoOe","pdf",1091139,21,"English","# Introduction\n## Background: NATs vs. autoregressive Transformers\n## Challenges in NAT learning\n## Prior training methods and objectives","[{\"question\":\"Why is learning NATs more challenging than training autoregressive Transformers?\",\"answer\":\"NATs predict tokens in parallel and drop left-to-right dependencies, so the independent prediction assumption makes likelihood-based training ineffective at capturing the needed structure.\"},{\"question\":\"What problem arises when training NATs by maximizing likelihood (MLE)?\",\"answer\":\"MLE training approximates marginal distributions but removes dependencies between tokens, which can be measured using the dataset’s conditional total correlation, leading to poorer generation quality despite higher likelihood.\"},{\"question\":\"How can earlier NAT objectives still improve generation quality despite low likelihood?\",\"answer\":\"The paper shows that many prior objectives can be framed as maximizing likelihood on a proxy distribution, which reduces information loss and improves generation quality even when the resulting likelihood remains low.\"}]","On the Learning of Non-Autoregressive Transformers | PDF",1789740053,53]