[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-149865-en":3,"doc-seo-149865-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},149865,962085571259,"Putri","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Attention Does Not Explain","Attention mechanisms are widely used in neural NLP models and are often presented as providing interpretability by assigning importance distributions over input units. This work tests whether attention weights truly relate to model outputs. Extensive experiments across text classification, question answering, and Natural Language Inference show that attention weights largely do not yield faithful explanations. Learned attention is frequently uncorrelated with gradient-based feature importance, and different attention distributions can produce equivalent predictions.","Attention is not Explanation  \nSarthak Jain  \nNortheastern University [jain.sar@husky.neu.edu](jain.sar@husky.neu.edu)  \nByron C. Wallace  \nNortheastern University [b.wallace@northeastern.edu](b.wallace@northeastern.edu)  \narXiv : 1902 . 10186v3 [ cs .CL] 8 May 2019  \nAbstract  \nAttention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful “explanations\" for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanationsand should not be treated as though they do. Code to reproduce all experiments is available at [https://github.com/successar/](https://github.com/successar/)[ ](https://github.com/successar/)AttentionExplanation.  \n1 Introduction and Motivation  \nAttention mechanisms (Bahdanau et al., 2014) induce conditional distributions over input units to compose a weighted context vector for downstream modules. These are now a near-ubiquitous component of neural NLP architectures. Attention weights are often claimed (implicitly or explicitly) to afford insights into the “inner-workings”of models: for a given output one can inspect the inputs to which the model assigned large attention weights. Li et al. (2016) summarized this commonly held view in NLP: “Attention provides an important way to explain the workings of neural models\". Indeed, claims that attention provides  \nafter 15 minutes watching the movie i was asking myself what todo leave the theater sleep or try to keep watching the movie to see if there was anything worth i finally watched the movie what a waste of time maybe i am not a 5 years old kid anymore  \noriginal ↵  \nafter 15 minutes watching the movie i was asking myself what todo leave the theater sleep or try to keep watching the movie to see if there was anything worth i finally watched the movie what a waste of time maybe i am not a 5 years old kid anymore  \nadversarial ↵˜  \nf (x|↵, ✓) = 0 .01 f (x|↵˜, ✓) = 0 .01  \nFigure 1: Heatmap of attention weights induced over a negative movie review. We show observed model attention (left) and an adversarially constructed set of attention weights (right) . Despite being quite dissimilar, these both yield effectively the same prediction (0.01) .  \ninterpretability are common in the literature, e.g.,(Xu et al., 2015 ; Choi et al., 2016 ; Lei et al., 2017 ; Martins and Astudillo, 2016 ; Xie et al., 2017 ; Mullenbach et al., 2018) .1  \nImplicit in this is the assumption that the inputs (e.g., words) accorded high attention weights are responsible for model outputs. But as far as weare aware, this assumption has not been formally evaluated. Here we empirically investigate the relationship between attention weights, inputs, and outputs.  \nAssuming attention provides a faithful explanation for model predictions, we might expect the following properties to hold. (i) Attention weights should correlate with feature importance measures (e.g., gradient-based measures); (ii) Alternative (or counterfactual) attention weight configurations ought to yield corresponding changes in prediction (and if they do not then are equally plausible as explanations) . We report that neither property is consistently observed by a BiLSTM with a standard attention mechanism in the con","cbCaidhSa38X6qfo","https://ap.wps.com/l/cbCaidhSa38X6qfo","pdf",1259036,1,16,"English","en",105,"# Introduction and Motivation\n## Attention mechanisms and interpretability claims\n## Research questions and contributions","[{\"question\":\"本文研究的核心问题是什么？\",\"answer\":\"研究注意力权重与模型输出之间是否存在有意义的对应关系，尤其是注意力是否能作为预测的“忠实解释”。\"},{\"question\":\"实验覆盖了哪些 NLP 任务？\",\"answer\":\"实验涵盖文本分类、问答（QA）以及自然语言推断（NLI）等任务，并评估注意力权重的可解释性。\"},{\"question\":\"研究的主要结论是什么？\",\"answer\":\"标准注意力模块提供的注意力权重通常并不能提供有意义的解释：注意力权重与基于梯度的特征重要性往往不相关，且不同注意力分布可能产生等价预测。\"}]","Attention Does Not Explain | PDF",1787808726,40,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"attention-does-not-explain","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/attention-does-not-explain/149865/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-09-04","2026-08-27",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"本文研究的核心问题是什么？","Question",{"text":76,"@type":77},"研究注意力权重与模型输出之间是否存在有意义的对应关系，尤其是注意力是否能作为预测的“忠实解释”。","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"实验覆盖了哪些 NLP 任务？",{"text":81,"@type":77},"实验涵盖文本分类、问答（QA）以及自然语言推断（NLI）等任务，并评估注意力权重的可解释性。",{"name":83,"@type":74,"acceptedAnswer":84},"研究的主要结论是什么？",{"text":85,"@type":77},"标准注意力模块提供的注意力权重通常并不能提供有意义的解释：注意力权重与基于梯度的特征重要性往往不相关，且不同注意力分布可能产生等价预测。","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":29,"slug":119},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]