[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-157813-105":59,"doc-detail-157813-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","systematic-generalization-by-finetuning-analyzing-pretrained-language-models-using-constituency-tests","Systematic Generalization by Finetuning - Analyzing Pretrained Language Models Using Constituency Tests","","Constituents are word groups acting as syntactic units, and many linguistic operations require manipulating them. This paper studies how different fine-tuning setups change pretrained seq2seq language models such as BART and T5 in replicating constituency tests, by varying which constituency tests and sentence types the models see during training. Results show accurate transformations only for familiar syntactic constructions and main verbs, while other settings degrade strongly. The findings suggest reliance on surface cues rather than constituency-level structure, highlighting brittleness in downstream NLP.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/systematic-generalization-by-finetuning-analyzing-pretrained-language-models-using-constituency-tests/157813/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/systematic-generalization-by-finetuning-analyzing-pretrained-language-models-using-constituency-tests/157813.png","ImageObject",300,407,{"name":92,"@type":93},"Himbo","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-17","2026-08-29",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",13,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What are constituency tests and why are they important in this paper?","Question",{"text":113,"@type":114},"Constituency tests transform sentences using operations like substitution and displacement to reveal word groups that behave as syntactic units. The paper uses them to evaluate whether models capture constituency structure required for these transformations.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How does the paper vary fine-tuning and evaluation settings?",{"text":118,"@type":114},"It systematically changes the combinations of constituency tests and sentence types used during fine-tuning, and tests whether inputs involve novel syntactic constructions and whether the main verb is novel.",{"name":120,"@type":111,"acceptedAnswer":121},"What do the results indicate about systematic generalization after fine-tuning?",{"text":122,"@type":114},"Models correctly transform sentences mainly when evaluated on syntactic constructions and verbs they were trained on. Performance drops substantially in other settings, suggesting limited systematic generalization and likely dependence on surface-level cues.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},157813,1787989948,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":52,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":130,"read_time":144,"preview_page_count":4},687197100911,"https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697","Systematic Generalization by Finetuning? Analyzing Pretrained Language  \nModels Using Constituency Tests  \nAishik Chakraborty Jackie CK Cheung Timothy J. O’Donnell  \nSchool of Computer Science, McGill University  \nQuébec AI Instuite (Mila)  \n[aishik.chakraborty@mail.mcgill.ca](aishik.chakraborty@mail.mcgill.ca) , {timothy.odonell, [jackie.cheung}@mcgill.ca](jackie.cheung}@mcgill.ca)  \nAbstract  \nConstituents are groups of words that behave as a syntactic unit. Many linguistic phenomena (e.g., question formation, diathesis alternations) require the manipulation and rearrangement of constituents in a sentence. In this paper, we investigate how different finetuning setups affect the ability of pretrained sequence-to-sequence language models such as BART and T5 to replicate constituency tests—transformations that involve manipulating constituents in a sentence. We design multiple evaluation settings by varying the combinations of constituency tests and sentence types that a model is exposed to during finetuning. We show that models can replicatea linguistic transformation on a specific typeof sentence that they saw during finetuning, but performance degrades substantially in other settings, showing a lack of systematic generalization. These results suggest that models often learn to manipulate sentences at a surface level unrelated to the constituent-level syntactic structure, for example by copying the first word of a sentence. These results may partially explain the brittleness of pretrained language models in downstream tasks 1.  \n1 Introduction  \nThe study of syntax revolves around understanding how words and phrases are combined to form sentences. Certain groups of words, known as constituents, behave as units in a sentence. In linguistics, groups of words that form constituents are often identified via constituency tests (see, e.g., Haegeman, 1994) . The tests involve transforming an input sentence using operations that substitute, displace, or otherwise modify constituents. For example, one well-known constituency test is proform substitution whereby a constituent is replaced by a corresponding pronominal form: John gave the book about syntax to the student−→John gave  \n1Code is available at [https://github.com/](https://github.com/)[ ](https://github.com/)aishikchakraborty/constituency  \nit to the student. The fact that the phrase the book about syntax can be replaced by it as a single unit, indicates that it is a constituent. Other examples of constituency tests include clefting and wh-question formation.  \nPrevious work has shown that pretrained transformer-based (Vaswani et al., 2017 ; Liu et al., 2019b) language models (LMs), trained on large amounts of text achieve unparalleled performance on virtually every downstream task. Nevertheless, these models suffer from robustness issues which call into question their reliability and ability to recover human-like linguistic generalizations. For example, in natural language inference, it has been shown that models rely on superficial cues such as lexical overlap between the premise and the hypothesis in order to make the correct predictions (McCoy et al., 2019 ; Nie et al., 2019) .  \nA line of work has thus attempted to probe pretrained LMs by training classifiers on top of frozen LM weights in order to extract some desired linguistic representation such as syntax trees (Hewitt and Manning, 2019 ; Coenen et al., 2019) . Work by Prasad et al. (2019) also shows that LSTM based LMs contain information about relative clauses inan interpretable manner. However, such analyses have several limitations. First, it is difficult to separate the contributions of the pre-trained LM weights from that of the probing classifier. In practice, it is necessary to place some constraints on the classifier (e.g., they must be linear) to ensure that extractive performance can be attributed to the pre-trained LMs. More fundamentally, understanding what can be extracted from pre-trained representatio","cbCaijeCaO6bAo5E","https://ap.wps.com/l/cbCaijeCaO6bAo5E","pdf",270498,"English","# Abstract\n# 1 Introduction\n# 2 Background","[{\"question\":\"What are constituency tests and why are they important in this paper?\",\"answer\":\"Constituency tests transform sentences using operations like substitution and displacement to reveal word groups that behave as syntactic units. The paper uses them to evaluate whether models capture constituency structure required for these transformations.\"},{\"question\":\"How does the paper vary fine-tuning and evaluation settings?\",\"answer\":\"It systematically changes the combinations of constituency tests and sentence types used during fine-tuning, and tests whether inputs involve novel syntactic constructions and whether the main verb is novel.\"},{\"question\":\"What do the results indicate about systematic generalization after fine-tuning?\",\"answer\":\"Models correctly transform sentences mainly when evaluated on syntactic constructions and verbs they were trained on. Performance drops substantially in other settings, suggesting limited systematic generalization and likely dependence on surface-level cues.\"}]","Systematic Generalization by Finetuning - Analyzing Pretrained Language Models Using Constituency Tests | PDF",25]