[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-140255-105":59,"doc-detail-140255-en":117},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":110,"head_meta":112,"extra_data":114,"updated_unix":116},105,"en","the-curious-case-of-absolute-position-embeddings-paper-findings","The Curious Case of Absolute Position Embeddings - Paper Findings","","Transformer language models represent word order through positional information, most often absolute position embeddings (APEs) learned during pretraining. Natural language, however, depends on relative position, and the ability of APEs to reflect this relativity has remained insufficiently tested. This study shows that models over-rely on APEs: shifting the sentence start to a non-zero position degrades performance on zero-to full-shot tasks across model families and sizes, raising concerns about APE efficacy for relative position modeling and motivating deeper analysis of word-order processing strategies.",{"@graph":69,"@context":109},[70,84,100],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/the-curious-case-of-absolute-position-embeddings-paper-findings/140255/",{"url":83,"name":65,"@type":85,"author":86,"headline":65,"publisher":89,"fileFormat":92,"inLanguage":63,"description":67,"dateModified":93,"datePublished":94,"encodingFormat":92,"isAccessibleForFree":95,"interactionStatistic":96},"DigitalDocument",{"name":87,"@type":88},"Sarah ","Person",{"url":74,"name":90,"@type":91},"DocShare","Organization","application/pdf","2026-09-11","2026-08-24",true,{"@type":97,"interactionType":98,"userInteractionCount":29},"InteractionCounter",{"@type":99},"ViewAction",{"@type":101,"mainEntity":102},"FAQPage",[103],{"name":104,"@type":105,"acceptedAnswer":106},"What problem does the study investigate about absolute position embeddings (APEs)?","Question",{"text":107,"@type":108},"It examines whether APEs let Transformer models capture relative position information, since language understanding depends more on relative than absolute word order.","Answer","https://schema.org",{"og:url":83,"og:type":111,"og:title":65,"og:site_name":90,"og:description":67},"article",{"robots":113,"canonical":83},"index,follow",{"doc_id":115,"site_id":62},140255,1787571343,{"code":4,"msg":5,"data":118},{"doc_id":115,"user_id":119,"nickname":87,"user_avatar":120,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":121,"file_id":122,"file_url":123,"file_type":124,"file_size":125,"view_count":29,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":126,"language":127,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":128,"faqs":129,"seo_title":130,"seo_description":67,"update_tm":116,"read_time":26},962085320529,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","The Curious Case of Absolute Position Embeddings  \nKoustuv Sinha‡†∗ Amirhossein Kazemnejad ‡∗ Siva Reddy‡ Joelle Pineau†‡ Dieuwke Hupkes† Adina Williams†  \n‡ McGill University / Mila-Quebec AI; † Meta AI {koustuv.sinha,[amirhossein.kazemnejad}@mail.mcgill.ca](amirhossein.kazemnejad}@mail.mcgill.ca)  \nAbstract  \nTransformer language models encode the notion of word order using positional information. Most commonly, this positional information is represented by absolute position embeddings (APEs), that are learned from the pretraining data. However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information. Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero-to full-shot tasks, across a range of model families and model sizes. Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.  \n1 Introduction  \nRecently, Transformer (Vaswani et al., 2017) language models (TLMs) have been widely used for natural language applications. Such models incorporate positional encodings: vectors encoding information about the order of words in context. Many models, such as RoBERTa (Liu et al., 2019), GPT3 (Brown et al., 2020) and OPT (Zhang et al., 2022), utilize absolute position embeddings (APEs) that directly encode absolute (linear) word order. APEs appear to contribute to the performance of such models; although when they are removed, some models become sensitive to ablative word scrambles (Sinha et al., 2021), while others work optimally (Haviv et al., 2022) . Thus, what precisely APEs contribute remains unclear.  \n*  \nEqual contributions.  \nZero starting position  \nWho could Thomas observe without distracting Nathan ?  \nNon-zero starting position  \nWho could Thomas observe without distracting Nathan ?  \nFigure 1: Transformer models with absolute positional embeddings have different representations for sentences starting from non-zero positions.  \nIt is conceivable that APEs may enable the model to handle the relative distances between words. If models were somehow learning relative position information despite using absolute positional embeddings, we would expect sentence encodings tobe the same in most cases, regardless of where they appear in the context window. For example, the meaning of “smoking kills” should be constant in“Kim said smoking kills”(positions 2–3) and “It was commonly believed by most adult Americans in the 90s that smoking kills”(positions 13–14), despite the fact that these words appear in different absolute positions. Given this, our central question is: do APEs enable the model to learn the relative distances between the words in a sentence?  \nPrior work has attempted to explore the consequences of APEs using probing methods (Wanget al., 2021) . APEs have been found to not capture the meaning of absolute or relative positions (Wang and Chen, 2020) . APEs have also been found tobias model output with positional artefacts (Luo et al., 2021), leading to better performance on token to position de-correlation (Ke et al., 2021) . Haviv et al. (2022) even find that causal TLMs perform adequately even without an explicit APEs. However, a systematic study on relativity of positional encodings is still needed.  \nTo better understand the relativity of absolute  \n4449  \nFindings of the Association for Computational Linguistics: EMNLP 2022 , pages 4449–4472 December 7-11, 2022 ©2022 Association for Computational Linguistics  \nposition embeddings, we f","cbCaicjDXRMEbeBk","https://ap.wps.com/l/cbCaicjDXRMEbeBk","pdf",836773,24,"English","# Abstract\n# Introduction\n## Positional encodings and absolute position embeddings\n## Prior work and open questions\n# Method: phase shifting and task evaluation\n## Zero-shot, few-shot, and full-shot tasks\n# Findings and implications","[{\"question\":\"What problem does the study investigate about absolute position embeddings (APEs)?\",\"answer\":\"It examines whether APEs let Transformer models capture relative position information, since language understanding depends more on relative than absolute word order.\"}]","The Curious Case of Absolute Position Embeddings - Paper Findings | PDF"]