[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-657628-105":59,"doc-detail-657628-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","gilt-graph-infused-layers-transformer-language-model","GiLT - Graph-Infused Layers Transformer Language Model","","Augmenting Transformer language models with linguistic structure improves syntactic generalization beyond standard training that omits explicit parses. This paper introduces the Graph-Infused Layers Transformer Language Model (GiLT), injecting dependency-graph structural information into attention weights while avoiding extra structural tokens in language modeling. GiLT incrementally constructs dependency graphs during token prediction and modulates attention using graph attributes. Experiments show improved syntactic generalization with competitive perplexity, and downstream gains after fine-tuning from a pretrained language model.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/gilt-graph-infused-layers-transformer-language-model/657628/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/gilt-graph-infused-layers-transformer-language-model/657628.png","ImageObject",300,407,{"name":92,"@type":93},"sagouaw","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-30","2026-10-26",true,{"@type":102,"interactionType":103,"userInteractionCount":34},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What problem does GiLT address in Transformer language modeling?","Question",{"text":112,"@type":113},"Transformer LMs typically lack explicit linguistic structure in their design and training. GiLT addresses this by integrating dependency-graph information into the model’s attention mechanism.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does GiLT inject structure without adding extra tokens?",{"text":117,"@type":113},"GiLT does not insert structural tokens; instead, it modulates Transformer attention weights using features extracted from dependency graphs that are constructed incrementally along token prediction.",{"name":119,"@type":110,"acceptedAnswer":120},"What performance improvements does GiLT achieve in experiments?",{"text":121,"@type":113},"GiLT improves syntactic generalization over Transformer baselines while maintaining competitive perplexity, and fine-tuning from a pretrained model yields better downstream task performance than the original pretrained model.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},657628,1793096046,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":34,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":41},3985747862151,"https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd","GiLT: Augmenting Transformer Language Models with Dependency  \nGraphs  \nTianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu*  \nSchool of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging {huangty2024,zhaoyd2023,zhouchy2022, [tukw}@shanghaitech.edu.cn](tukw}@shanghaitech.edu.cn)  \nAbstract  \nAugmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on syntactic tree structures of languages, in particular constituency tree structures. We propose GraphInfused Layers Transformer Language Model (GiLT) which leverages dependency graphs for augmenting Transformer language models. Unlike most previous work, GiLT does not insert extra structural tokens in language modeling;  \ninstead, it injects structural information into language modeling by modulating attention weights in the Transformer with features extracted from the dependency graph that is incrementally constructed along with token prediction. In our experiments, GiLT with semantic dependency graphs achieves better syntactic generalization while maintaining competitive perplexity in comparison with Transformer language model baselines. In addition, GiLT can be finetuned from a pretrained language model to achieve improved downstream task performance. Our code is released at [https://github.com/cookie-pie-oops/GiLT-LM](https://github.com/cookie-pie-oops/GiLT-LM).  \n1 Introduction  \nTransformer language models (LMs) have shown excellent performance in language modeling and downstream tasks (Vaswani et al., 2017) . Notably, linguistic structures such as syntactic and semantic parses that have been deemed essential in traditional natural language processing are absent from the model design and training process of Transformer LMs.  \nOver the past decade, a number of researchers have been trying to integrate linguistic structures into neural language models. Among them are syntactic LMs which jointly model syntactic structures and surface words (Choe and Charniak, 2016) .  \n* Corresponding Author  \nThese include earlier work such as RNNG, which combines constituency parsing with recurrent neural networks (Dyer et al., 2016 ; Kim et al., 2019 ; Noji and Oseki, 2021), and recent studies that incorporate constituency and dependency syntax into Transformers (Yoshida and Oseki, 2022 ; Qian et al., 2021 ; Sartran et al., 2022 ; Murty et al., 2023 ; Zhao et al., 2024) . Empirically, they achieve stronger syntactic generalization compared with standard Transformer LMs while retaining competitive language modeling performance.  \nHowever, existing researches in this direction have two major limitations. First, most of them are based on constituency syntactic tree structures. Dependency tree structures, another important form of syntax, receive much less attention (Zhao et al., 2024) . In addition, little work has been done to jointly model linguistic structures other than syntactic trees in LMs. Second, most of the existing methods require inserting additional tree-building operations into the input and output sequence, leading to longer sequence lengths and higher computational cost, and making it harder to finetune apretrained LM into a syntactic LM. An exception is Pushdown Layers (Murty et al., 2023), which leverages syntactic trees to guide attention computation without changing the LM’s symbol space.  \nIn this paper, we propose the Graph-Infused Layer Transformer LM (GiLT) that addresses the above-mentioned limitations in integrating linguistic structures into Transformer LMs. GiLT is based on dependency graphs that subsume both syntactic dependency trees and semantic dependency graphs, thus extending syntactic LM research beyond syntax. Inspired by Pushdown Layers (Murty et al., 2023), GiLT incrementally constructs dependency graphs without changing the symbol space of the underlying LMs, and modula","cbCainfjrVj9wogj","https://ap.wps.com/l/cbCainfjrVj9wogj","pdf",477132,12,"English","# Introduction\n## Background\n### Pushdown Layers","[{\"question\":\"What problem does GiLT address in Transformer language modeling?\",\"answer\":\"Transformer LMs typically lack explicit linguistic structure in their design and training. GiLT addresses this by integrating dependency-graph information into the model’s attention mechanism.\"},{\"question\":\"How does GiLT inject structure without adding extra tokens?\",\"answer\":\"GiLT does not insert structural tokens; instead, it modulates Transformer attention weights using features extracted from dependency graphs that are constructed incrementally along token prediction.\"},{\"question\":\"What performance improvements does GiLT achieve in experiments?\",\"answer\":\"GiLT improves syntactic generalization over Transformer baselines while maintaining competitive perplexity, and fine-tuning from a pretrained model yields better downstream task performance than the original pretrained model.\"}]","GiLT - Graph-Infused Layers Transformer Language Model | PDF",1792985444]