[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84440-en":3,"doc-seo-84440-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84440,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge","Large Language Models excel at multi-hop reasoning within the training distribution but struggle with unseen compositions, a failure described as the “curse of two-hop reasoning.” The work attributes this brittleness to missing supervision on the bridge entity and introduces identity bridge, a minimal constraint enforcing an identity mapping on bridge tokens. With this supervision, even a one-layer transformer with uniform attention can obtain out-of-distribution two-hop generalization. Theoretical analysis shows implicit regularization, while empirical results align GPT-2 performance with Emb–MLP and extend to fine-tuned LLMs.","arXiv :2509 .24653v2 [ cs .LG] 13 Jul 2026  \nUnveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge  \nPengxiao Lin1,2, ∗ Zheng-An Chen1,∗ Zhi-Qin John Xu1,2,3 †  \n1School of Mathematical Sciences, Shanghai Jiao Tong University  \n2Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University  \n3Shanghai Seres Information Technology Co., Ltd, Shanghai 200040, China.  \nAbstract  \nLarge Language Models (LLMs) excel at multi-hop reasoning in distribution, yet fail on unseen compositions, a phenomenon known as the “curse of two-hop reasoning”. In this work, we argue that this phenomenon can be attributed to a missing supervision on the bridge entity. We formalize this gap by introducing identity bridge, a minimal supervision that enforces a identity mapping on bridge tokens. Under this supervision, even a one-layer transformer with uniform attention (Emb–MLP) can achieve out-of-distribution (OOD) two-hop generalization. We provide a theoretical analysis demonstrating that identity bridge induces an implicit regularization effect, leading the model to establish a direct subject-to-answer association. From an empirical perspective, the performance of standard GPT-2 models aligns closely with simple Emb–MLP models across varying levels of problem complexity. Finally, analyses of fine-tuned mainstream LLMs indicate that correct two-hop predictions consistently coincide with the establishment of a subject-to-answer relationship, extending our findings to realistic settings.  \n1 Introduction  \nLarge language models (LLMs) achieve strong performance on a wide range of multi-step reasoning tasks, especially when assisted by chain-of-thought (CoT) Wei et al. (2022); Yao et al. (2023); Kojima et al. (2022) . Despite their strong CoT-assisted performance, LLMs exhibit very fragile reasoning abilities when forced to produce the final answer directly Bai et al. (2025) . On the one hand, models sometimes exhibit signatures consistent with latent multi-step processing, yet such evidence is suggestive rather than definitive and is often difficult to disentangle from shortcut strategies Ding et al. (2024); Lin et al. (2025) . On the other hand, controlled synthetic settings reveal striking and systematic failures, including the reversal curse Berglund et al. (2024); Allen-Zhu & Li (2024; 2025); Qi et al.(2023) and the two-hop curse Balesni et al. (2025); Dziri et al. (2023) . These phenomena suggest that seemingly minor design choices in synthetic data and training objectives can crucially interact with optimization-induced inductive biases, leading to brittle multi-hop inference. While recent work has offered partial analyses and empirical observations Yang et al. (2024); Biran et al. (2024); Wang et al. (2024); Ye et al. (2025); Liu et al. (2026), a theoretically grounded explanation of mechanisms behind multi-hop reasoning remains incomplete. This motivates the following questions.  \n1. Why is OOD compositional generalization in two-hop reasoning so brittle, and what overlooked factors cause this brittleness in a way that admits rigorous characterization?  \n2. When a transformer appears to generalize in two-hop settings, what mechanism underlies this behavior: a genuine two-step implicit inference, or a shortcut solution that mimics composition?  \n∗ Equal contribution.  \n†Corresponding author: [xuzhiqin@sjtu.edu.cn](xuzhiqin@sjtu.edu.cn).  \nIn this work, we revisit the two-hop curse, where models fit two single-hop relations (a → band b → c in the training set) yet fail to answer the composed query (a → c at evaluation), following the setup and definitions of Wang et al. (2024) . We highlight an often overlooked factor that standard synthetic setups do not enforce the bridge entity to be identical in the input and the output. Without this constraint, training can fit the two single-hop tasks using disjoint features, causing composition to break down. We introduce identity bridge, a minimal fix that adds a zero","cbCaido8gHgHkoBE","https://ap.wps.com/l/cbCaido8gHgHkoBE","pdf",893718,1,29,"English","en",105,"# Abstract\n# Introduction\n## Problem motivation: OOD compositional generalization and the two-hop curse\n## Identity bridge: minimal supervision on bridge entities\n## Mechanistic analysis with a simplified Emb–MLP model\n## Empirical validation with GPT-2 and other mainstream LLMs\n## Summary of contributions","[{\"question\":\"What causes the “curse of two-hop reasoning” in transformers?\",\"answer\":\"The paper argues the brittleness comes from missing supervision on the bridge entity, allowing the model to fit single-hop relations using disjoint features that do not compose at evaluation.\"},{\"question\":\"What is identity bridge, and how does it help?\",\"answer\":\"Identity bridge adds a minimal supervision that enforces an identity mapping for each bridge token, effectively including a zero-hop self-mapping so composition generalizes to unseen two-hop queries.\"},{\"question\":\"Does the proposed mechanism represent real multi-step inference or a shortcut?\",\"answer\":\"The authors characterize the one-layer solution as a form of shortcut pattern rather than step-by-step implicit reasoning, and they validate related behavior with higher-complexity settings and multiple LLM families.\"}]",1784195641,73,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"unveiling-the-mechanisms-of-multi-hop-reasoning-in-transformers-via-identity-bridge","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/unveiling-the-mechanisms-of-multi-hop-reasoning-in-transformers-via-identity-bridge/84440/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What causes the “curse of two-hop reasoning” in transformers?","Question",{"text":75,"@type":76},"The paper argues the brittleness comes from missing supervision on the bridge entity, allowing the model to fit single-hop relations using disjoint features that do not compose at evaluation.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is identity bridge, and how does it help?",{"text":80,"@type":76},"Identity bridge adds a minimal supervision that enforces an identity mapping for each bridge token, effectively including a zero-hop self-mapping so composition generalizes to unseen two-hop queries.",{"name":82,"@type":73,"acceptedAnswer":83},"Does the proposed mechanism represent real multi-step inference or a shortcut?",{"text":84,"@type":76},"The authors characterize the one-layer solution as a form of shortcut pattern rather than step-by-step implicit reasoning, and they validate related behavior with higher-complexity settings and multiple LLM families.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]