[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273518-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273518-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","what-tokens-are-learned-when-tokenization-is-optimized-jointly-with-language-modeling","What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling","","Tokenization is a core component of language modeling pipelines, yet it is commonly kept fixed even though it strongly influences performance across languages. This study examines which tokens are learned when tokenization is jointly optimized with language modeling, comparing tokenizer-free approaches like SSLMs and H-Nets against fixed tokenizers across 18 typologically and script-diverse languages. Joint optimization changes token structure: SSLMs align morphologically and use context-efficient tokens, while H-Nets emphasize byte-level efficiency, yielding longer tokens with low subword overlap. It also varies by language typology and improves perplexity under downstream BERT evaluation.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/what-tokens-are-learned-when-tokenization-is-optimized-jointly-with-language-modeling/273518/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/what-tokens-are-learned-when-tokenization-is-optimized-jointly-with-language-modeling/273518.png","ImageObject",300,407,{"name":42,"@type":43},"Hazel","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-02","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",19,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What question does the paper investigate about tokenization?","Question",{"text":63,"@type":64},"It studies what tokens are learned when tokenization is optimized jointly with language modeling, rather than learned independently and kept fixed.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How do tokenizer-free approaches like SSLMs and H-Nets differ in learned token structure?",{"text":68,"@type":64},"SSLMs recover morphologically aligned and contextually efficient tokens, while H-Nets prioritize byte-level efficiency, producing longer tokens with very low overlap with standard subword vocabularies.",{"name":70,"@type":61,"acceptedAnswer":71},"How do joint tokenization approaches affect downstream performance?",{"text":72,"@type":64},"Using pretrained-then-finetuned BERT models, SSLM-based pretokenization consistently reduces language modeling perplexity and achieves competitive downstream performance despite distinct vocabularies.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273518,1789471843,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144,"preview_page_count":4},137441390410,"https://ap-avatar.wpscdn.com/avatar/2000252f4ab5702993?_k=1776741390130283984","What Tokens are Learned when Tokenization is Optimized Jointly with  \nLanguage Modeling?  \narXiv : 2608 . 17325v 1 [cs .CL] 18 Aug 2026  \nSaketh Reddy Vemula  \nIIIT Hyderabad, India [saketh.vemula@research.iiit.ac.in](saketh.vemula@research.iiit.ac.in)  \nAbstract  \nTokenization is a fundamental component of language modeling pipelines. Despite its importance, it is often fixed, even though it significantly impacts model performance across languages. In this work, we analyze what tokens are learned when tokenization is jointly optimized with language modeling. We compare tokenizer-free approaches such as SSLMs and H-Nets with fixed tokenizers across 18 typologically and script-diverse languages. Our results show that joint optimization fundamentally alters token structure. SSLMs recover morphologically aligned and contextually efficient tokens, whereas H-Nets prioritize byte-level efficiency, producing longer tokens with very low overlap with standard subword vocabularies. We further show that tokenization behavior varies across language typologies. Agglutinative languages exhibit more dynamic segmentation patterns while learning. Through downstream evaluation, with pretrained-thenfinetuned BERT models, we find that SSLMbased pretokenization consistently reduces language modeling perplexity and achieves competitive downstream performance despite distinct vocabularies. Overall, tokenizer-free approaches optimize for contextual and computational efficiency rather than strict morphological structure, resulting in fundamentally different yet effective vocabularies for downstream NLP.  \n1 Introduction  \nTokenization is a fundamental preprocessing step in language models (LMs) . In transformer-based LMs (Vaswani et al., 2023), it is often performed by learning a tokenizer through subword algorithms such as Byte-pair Encoding (BPE, Gage (1994); Sennrich et al. (2016)), Unigram Language Model (ULM, Kudo (2018)), or WordPiece (WPC, Schuster and Nakajima (2012)) . These algorithms are built on simple statistical priors, such as the assumption that frequently co-occurring characters  \nParameswari Krishnamurthy  \nIIIT Hyderabad, India [param.krishna@iiit.ac.in](param.krishna@iiit.ac.in)  \nFigure 1: Morphological alignment evaluation of tokenizerfree (H-Nets, SSLMs) and fixed-tokenizer approaches. SSLMs show consistently higher alignment, whereas H-Nets trade morphological alignment for longer, computationally efficient tokens.  \nshould be part of the same token (Sennrich et al., 2016) . Despite their widespread adoption, it is unclear whether such approaches provide best performance, especially when working with languages across diverse morphological complexities, typologies, and scripts. This concern is further amplified from the standard tokenization-LM-detokenization pipeline, where a tokenizer is learned independently of language model training and remains static. We refer to these as fixed-tokenizer approaches. Assessing the best tokenizer for such LMs is both difficult and expensive, since the tokenizer remains fixed prior to training and cannot be modified thereafter.  \nVarious modifications and novel algorithms have been proposed to learn a tokenizer. This includes advances in pre-tokenization (Banerjee and Bhattacharyya, 2018 ; Hu et al., 2025), robustness (Kudo and Richardson, 2018), morphological alignment (Hofmann et al., 2022 ; Libovický and Helcl, 2024 ; Zhu et al., 2025), vocabulary pruning (Cognetta et al., 2024) or modifications (Chizhov et al., 2024), and cross-token1 ap-  \n1Tokens that can span multiple words, for example “by the way”.  \nproaches (Schmidt et al., 2025 ; Liu et al., 2025) . However, no single approach can be concluded tobethe “best”, as conclusions depend on various factors such as: 1) languages chosen and their morphological properties, 2) experimental setup like model architecture, dataset size and other hyperparameters (Zhu et al., 2019 ; Poelman et al., 2025) .  \nRecently, new architectures have been ","cbCaigKFmHbvY47R","https://ap.wps.com/l/cbCaigKFmHbvY47R","pdf",903805,21,"English","# Abstract\n# 1 Introduction\n## Tokenization and fixed-tokenizer approaches\n## Tokenizer-free language models\n# 2 Related Work","[{\"question\":\"What question does the paper investigate about tokenization?\",\"answer\":\"It studies what tokens are learned when tokenization is optimized jointly with language modeling, rather than learned independently and kept fixed.\"},{\"question\":\"How do tokenizer-free approaches like SSLMs and H-Nets differ in learned token structure?\",\"answer\":\"SSLMs recover morphologically aligned and contextually efficient tokens, while H-Nets prioritize byte-level efficiency, producing longer tokens with very low overlap with standard subword vocabularies.\"},{\"question\":\"How do joint tokenization approaches affect downstream performance?\",\"answer\":\"Using pretrained-then-finetuned BERT models, SSLM-based pretokenization consistently reduces language modeling perplexity and achieves competitive downstream performance despite distinct vocabularies.\"}]","What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling | PDF",53]