[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85750-en":3,"doc-seo-85750-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85750,5909877438554,"Maeve","https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272",8,"Research & Report","Index-1.9B SLM Technical Report","Index-1.9B is an open series of small language models developed at Bilibili, including Base, Pure, Chat, and Character variants with distinct training and alignment strategies. Base is pre-trained on 2.8T mostly Chinese and English tokens; Pure filters instruction-like data to isolate its impact. Chat applies SFT and DPO alignment, while Character adds retrieval-augmented generation for few-shot role-playing customization. A Warmup–Stable–Decay schedule and Norm-Head stabilize large-learning-rate training, yielding an average 64.92 on standard benchmarks. The report studies model depth, learning-rate effects, schedule–data coupling, and an unexplained performance surge during constant-learning-rate training, and releases models and evaluation code.","arXiv :2607 .09885v 1 [ cs .CL] 10 Jul 2026  \nIndex  \nIndex SLM Technical Report  \n bilibili/Index-1 .9B  IndexTeam/Index-1 .9B-Chat  \nBilibili Index LLM Team  \nLusheng Zhang, Shien He, Tianxing Yan, Mengran Yu, Ziang Cui, Kai Zhao, Xiaojing Liu,  \nTianjiao Li†  \n{zhanglusheng01, heshien, yantianxing, yumengran, cuiziang, zhaokai, liuxiaojing, [litianjiao01}@bilibili. com](litianjiao01}@bilibili. com)  \n† Project lead  \nAbstract  \nWe present Index-1.9B, a series of open small language models developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion non-embedding parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; Index-1.9B-Chat, aligned from the base model with supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for few-shot role-playing customization. Pretraining employs a Warmup–Stable–Decay learning-rate schedule in which the concentration of curated data is raised substantially during the decay phase, together with a Norm-Head output layer that stabilizes training under large learning rates. On a suite of standard benchmarks covering examination, reasoning, mathematics, and code, Index-1.9B-Base attains an average score of 64.92, competitive with or exceeding open models of several times its size. We further report controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training, and we document an unexplained surge in benchmark performance midway through the constant-learning-rate phase. All models, together with evaluation code, are released at [https://github.com/bilibili/Index-1.9B](https://github.com/bilibili/Index-1.9B).  \nContents  \n1 Introduction 2  \n2 Pre-training 2  \n2. 1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2  \n2.2 Tokenizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3  \n2.3 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4  \n2.4 Training Recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4  \n2.5 Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5  \n3 Alignment 5  \n3.1 Supervised Fine-tuning ........................................ 5  \n3.2 Direct Preference Optimization ................................... 5  \n4 Few-shot Role-playing 6  \n5 Evaluation 6  \n5.1 Setup ................................................. 6  \n5.2 Base Model Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6  \n5.3 Chat Model Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6  \n5.4 Role-playing Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8  \n(c) 2024 Bilibili Index LLM Team. All Rights Reserved. 1  \n Index  \n6 Discussion 8  \n6.1 Stabilizing the LM Head: Norm-Head ................................ 8  \n6.2 Depth versus Width .......................................... 8  \n6.3 Learning-rate Magnitude ....................................... 9  \n6.4 Learning-rate Schedules ....................................... 9  \n6.5 Coupling Learning-rate Decay with Data Quality .......................... 9  \n6.6 Instruction Data in Pre-training ................................... 11  \n6.7 A Performance Surge During the Stable Phase ........................... 11  \n7 Limitations 12  \n8 Conclusion 12  \nA Qualitative Examples 15  \nB Safety Preference-pair Construction 16  \n1 Introduction  \nSmall language models (SLMs)","cbCaiaQPif3MPAlE","https://ap.wps.com/l/cbCaiaQPif3MPAlE","pdf",1768971,1,16,"English","en",105,"# Introduction\n# Pre-training\n## Data\n## Tokenizer\n## Model Architecture\n## Training Recipe\n## Infrastructure\n# Alignment\n## Supervised Fine-tuning\n## Direct Preference Optimization\n# Few-shot Role-playing\n# Evaluation\n## Setup\n## Base Model Results\n## Chat Model Results\n## Role-playing Results\n# Discussion\n## Stabilizing the LM Head: Norm-Head\n## Depth versus Width\n## Learning-rate Magnitude\n## Learning-rate Schedules\n## Coupling Learning-rate Decay with Data Quality\n## Instruction Data in Pre-training\n## A Performance Surge During the Stable Phase\n# Limitations\n# Conclusion\n# Qualitative Examples\n# Safety Preference-pair Construction","[{\"question\":\"What are the four main Index-1.9B model variants and how do they differ?\",\"answer\":\"Index-1.9B-Base is a foundation model pre-trained on 2.8T tokens. Index-1.9B-Pure is trained with instruction-like data strictly filtered out to isolate that effect. Index-1.9B-Chat is aligned from the base model using SFT and DPO, and Index-1.9B-Character adds retrieval-augmented generation for few-shot role-playing customization.\"},{\"question\":\"How does the report describe the training schedule and stabilization method?\",\"answer\":\"Pre-training uses a Warmup–Stable–Decay learning-rate schedule, with curated data concentration raised substantially during the decay phase. Training stability under large learning rates is supported by a Norm-Head output layer.\"},{\"question\":\"What experimental findings does the report highlight during training dynamics and evaluation?\",\"answer\":\"Controlled studies examine model depth, learning-rate magnitude and scheduling, how learning-rate decay couples with data quality, and the effect of including instruction data during pre-training. The report also documents an unexplained surge in benchmark performance midway through the constant-learning-rate phase, while reporting an average Base-model benchmark score of 64.92.\"}]",1784206004,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"index-19b-slm-technical-report","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/index-19b-slm-technical-report/85750/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What are the four main Index-1.9B model variants and how do they differ?","Question",{"text":74,"@type":75},"Index-1.9B-Base is a foundation model pre-trained on 2.8T tokens. Index-1.9B-Pure is trained with instruction-like data strictly filtered out to isolate that effect. Index-1.9B-Chat is aligned from the base model using SFT and DPO, and Index-1.9B-Character adds retrieval-augmented generation for few-shot role-playing customization.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does the report describe the training schedule and stabilization method?",{"text":79,"@type":75},"Pre-training uses a Warmup–Stable–Decay learning-rate schedule, with curated data concentration raised substantially during the decay phase. Training stability under large learning rates is supported by a Norm-Head output layer.",{"name":81,"@type":72,"acceptedAnswer":82},"What experimental findings does the report highlight during training dynamics and evaluation?",{"text":83,"@type":75},"Controlled studies examine model depth, learning-rate magnitude and scheduling, how learning-rate decay couples with data quality, and the effect of including instruction data during pre-training. The report also documents an unexplained surge in benchmark performance midway through the constant-learning-rate phase, while reporting an average Base-model benchmark score of 64.92.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,118,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":28,"slug":117},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":119,"slug":120},30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]