[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-413521-105":59,"doc-detail-413521-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","vocabulary-modifications-for-domain-adaptive-pretraining-of-clinicallanguage-models","Vocabulary Modifications for Domain-adaptive Pretraining of ClinicalLanguage Models","","Research examines how domain-adaptive pretraining of clinical language models can be improved by modifying model vocabulary rather than relying solely on generic tokenization. The study tests a strategy that initializes from an existing generic model but introduces a domain-specific vocabulary for Swedish clinical text. Results show that pairing domain-adaptive pretraining with a domain vocabulary improves performance on two Swedish clinical NLP tasks, emphasizing vocabulary adaptation before continued training.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":35,"@type":76,"position":81},"https://docshare.wps.com/document/healthcare/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/vocabulary-modifications-for-domain-adaptive-pretraining-of-clinicallanguage-models/413521/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/vocabulary-modifications-for-domain-adaptive-pretraining-of-clinicallanguage-models/413521.png","ImageObject",300,407,{"name":92,"@type":93},"gewufjwa","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-29","2026-09-28",true,{"@type":102,"interactionType":103,"userInteractionCount":14},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"Why can generic BERT-style language models be sub-optimal in specialized clinical domains?","Question",{"text":112,"@type":113},"Domain differences affect both language use and vocabulary. As a result, generic vocabularies may tokenize domain terms poorly, reducing downstream effectiveness.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"What is the proposed strategy for Swedish clinical language modeling?",{"text":117,"@type":113},"Build a clinical-specific vocabulary using WordPiece on Swedish clinical text, then initialize an existing generic Swedish model to leverage shared parameters and handle both overlapping and non-overlapping tokens.",{"name":119,"@type":110,"acceptedAnswer":120},"What evidence supports the benefit of domain-adaptive pretraining with a domain-specific vocabulary?",{"text":121,"@type":113},"An empirical investigation reports improved performance on two downstream Swedish clinical NLP tasks when domain-adaptive pretraining is combined with a domain-specific vocabulary rather than a general-domain vocabulary.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},413521,1792128675,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":34,"category_name":35,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":14,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":44,"language":139,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":67,"update_tm":143,"read_time":144,"preview_page_count":4},3985747882900,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","# Vocabulary Modifications for Domain-adaptive Pretraining of ClinicalLanguage Models\n\nAnastasios Lamproudis,Aron Henriksson and Hercules DalianisDepartment of Computer and System Sciences,Stockholm University,Stockholm,Sweden  \nNatural Language Processing,Language Models,Domain-adaptive Pretraining,Clinical Text,Swedish.  \nAbstract:Research has shown that using generic language models-specifically,BERT models-in specialized domainsmay be sub-optimal due to domain differences in language use and vocabulary.There are several techniquesfor developing domain-specific language models that leverage the use of existing generic language models,including continued and domain-adaptive pretraining with in-domain data.Here,we investigate a strategybased on using a domain-specific vocabulary,while leveraging a generic language model for initialization.The results demonstrate that domain-adaptive pretraining,in combination with a domain-specific vocabulary-as opposed to a general-domain vocabulary-yields improvements on two downstream clinical NLP tasksfor Swedish.The results highlight the value of domain-adaptive pretraining when developing specializedlanguage models and indicate that it is beneficial to adapt the vocabulary of the language model to the targetdomain prior to continued,domain-adaptive pretrainng of a generic language model.  \n## 1 INTRODUCTION\n\nuing to pretrain an existing,generic language modelwith in-domain data(domain-adaptive pretraining),either by using large amounts of in-domain data,ifavailable,or by only using task-related unlabeled data(task-adaptive pretraining).  \nThe paradigm of pretraining and fine-tuning languagemodels has become a cornerstone of modern naturallanguage processing.By utilizing transfer learningtechniques,language models such as BERT(Devlinet al.,2019)are pretrained using vast amounts of un-labeled text data and subsequently adapted,or fine-tuned,to carry out various downstream tasks usinglabeled,task-specific data.While pretraining is gen-erally computationally expensive-and therefore typi-cally carried out by resource-rich organizations-fine-tuning is computationally inexpensive.These fine-tuned models have obtained state-of-the-art results inmany natural language processing tasks.  \nHowever,the domain of the language model isnot only manifested in the pretraining or fine-tuningsessions,but also in the model's vocabulary.Everylanguage model requires a vocabulary for processingthe input text.For transformer-based language mod-els,this vocabulary acts not only as a prepossessingmethod but for connecting the input text to the model.The vocabulary is used for creating the mapping of thelanguage to the learned-or to-be-learned-represen-tations of the model at its lowest level:the embeddingtable.This vocabulary can be built using different al-gorithms,such as WordPiece(Wu et al.,2016),Sen-tencePiece(Kudo and Richardson,2018)and Byte-Pair encoding(Sennrich et al.,2016).As such,thevocabulary of a language model is determined by thealgorithm and the data which was used to generate it.  \nLanguage models are often pretrained using large,readily available corpora in the general domain,e.g.Wikipedia.However,the use of generic languagemodels in specialized domains may be sub-optimalbecause of domain differences,in terms of,for in-stance,language use and vocabulary(Lewis et al.,2020,Gururangan et al.,2020).This has motivatedefforts to develop domain-specific language models,  \nA drawback of continued,domain-adaptive pre-training of an existing,generic language model is thatdomain-specific words are often tokenized poorly.For example,when tokenizing the common clinicalterm röntgenundersökning(English:x-ray investiga-tion)with a general-domain language model(KB-BERT),it is split into multiple subtokens:['ro’,  \ne.g.SciBERT(Beltagy et al.,2019)and BioBERT(Lee et al.,2020).Specialized language models havebeen developed either by(i)pretraining a languagemodel with in-domain data from scratch,possibly incombinati","cbCaia9WXVMVDQUa","https://ap.wps.com/l/cbCaia9WXVMVDQUa","pdf",503186,"English","# Abstract\n# 1 Introduction\n## Domain adaptation and vocabulary\n# 2 Related Work","[{\"question\":\"Why can generic BERT-style language models be sub-optimal in specialized clinical domains?\",\"answer\":\"Domain differences affect both language use and vocabulary. As a result, generic vocabularies may tokenize domain terms poorly, reducing downstream effectiveness.\"},{\"question\":\"What is the proposed strategy for Swedish clinical language modeling?\",\"answer\":\"Build a clinical-specific vocabulary using WordPiece on Swedish clinical text, then initialize an existing generic Swedish model to leverage shared parameters and handle both overlapping and non-overlapping tokens.\"},{\"question\":\"What evidence supports the benefit of domain-adaptive pretraining with a domain-specific vocabulary?\",\"answer\":\"An empirical investigation reports improved performance on two downstream Swedish clinical NLP tasks when domain-adaptive pretraining is combined with a domain-specific vocabulary rather than a general-domain vocabulary.\"}]","Vocabulary Modifications for Domain-adaptive Pretraining of ClinicalLanguage Models | PDF",1790595075,23]