[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82854-en":3,"doc-seo-82854-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82854,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","LP-SFT Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure","Supervised fine-tuning (SFT) adapts pretrained language models to downstream domains but often damages out-of-domain capabilities and reduces generation diversity. The standard cross-entropy objective suppresses the observed label token without constraining how probability mass is redistributed among alternatives, potentially breaking the local preference structure formed during pretraining. By analyzing Shannon and Rényi entropies, the method identifies a regular multimodal entropy structure in pretrained models. Based on this, LP-SFT preserves local entropy structure using an adaptive alternative-token support and a locally normalized preservation loss, improving performance while maintaining diversity.","arXiv :2607 .04733v 1 [ cs .CL] 6 Jul 2026  \nLP-SFT: LOCAL-PRESERVING SUPERVISED FINE-TUNING  \nVIA MULTIMODAL ENTROPY STRUCTURE ∗  \nYueyang Wang 1† Baolong Bi2 Shuo Lu3 Jingyuan Zhang4  \n1 School of Mathematical Sciences, Peking University  \n2Institute of Computing Technology, Chinese Academy of Sciences  \n3Institute of Automation, Chinese Academy of Sciences  \n4 College of Computing, Georgia Institute of Technology  \nABSTRACT  \nSupervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard cross-entropy fine-tuning promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, potentially distorting the rich local preference structure learned during pretraining. We first analyze next-token predictions using Shannon and Rényi entropies, revealing that pretrained models exhibit a regular multimodal entropy structure. These entropy peaks correspond to varying numbers of plausible alternatives, indicating that the base model intrinsically encodes rich distributional knowledge beyond the single supervised token. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure. At each step, LP-SFT constructs an adaptive support of alternative tokens and applies a locally normalized preservation loss to maintain the base model’s relative structure among them, while standard cross-entropy independently optimizes the supervised token. Across mixed-domain and single-domain fine-tuning experiments, LP-SFT improves overall performance over vanilla SFT and recent SFT-enhancement baselines, achieving the best balance between pass@1 accuracy and pass@k performance. These results suggest that local preservation helps mitigate capability degradation without collapsing sampling-accessible diversity.  \nKeywords Supervised Fine-Tuning · Catastrophic Forgetting · Distribution Preservation · Multimodal Entropy Structure  \n1 Introduction  \nSupervised fine-tuning (SFT) has become the dominant paradigm for adapting pretrained language models to instruction following, reasoning, coding, and domain-specific tasks[16, 18] . Despite its simplicity and effectiveness, SFT often induces capability degradation along two dimensions: over-specialization to the fine-tuning distribution, which can cause catastrophic forgetting of out-of-domain capabilities, and reduced generation diversity[21], where improved single-sample accuracy may come at the cost of suppressing alternative valid solutions[15] .  \nWe argue that a significant portion of this degradation stems from the mismatch between the standard crossentropy objective and the rich distributional knowledge encoded in pretrained models. Large language models acquire broad capabilities from diverse pretraining corpora, resulting in next-token distributions that contain substantial information beyond the single observed target token. However, cross-entropy supervises each position using only this target token, encouraging the model to fit the fine-tuning data while largely ignoring  \n∗ Code is available at [https://github.com/Wakaka161/LP-SFT](https://github.com/Wakaka161/LP-SFT).  \n†Corresponding author: [wangyueyang@stu.pku.edu.cn](wangyueyang@stu.pku.edu.cn)  \nthe remaining preference structure of the base distribution. Under distribution shift between pretraining and fine-tuning data, repeatedly suppressing plausible alternatives can distort this pretrained structural integrity, leading to catastrophic forgetting and reduced generation diversity.  \nThis perspective suggests that effective fine-tuning should not only align the model with supervised targets, but also preserve useful local structure in the pretrained distribution. To understand where such preservation is most need","cbCair43K4dyrKT5","https://ap.wps.com/l/cbCair43K4dyrKT5","pdf",13761049,1,21,"English","en",105,"# Abstract\n# Introduction\n## Capability degradation in supervised fine-tuning\n## Entropy-based analysis of next-token distributions\n## LP-SFT objective and local structure preservation\n# Contributions","[{\"question\":\"Why can standard supervised fine-tuning lead to capability degradation?\",\"answer\":\"Cross-entropy fine-tuning repeatedly suppresses plausible alternatives and focuses only on the observed target token, which can distort the pretrained local preference structure under distribution shift.\"},{\"question\":\"What does the multimodal entropy structure refer to in pretrained models?\",\"answer\":\"Next-token prediction entropies form distinct peaks near ln k for integer k, indicating discrete uncertainty regimes where some positions are nearly deterministic while others admit a small set of alternatives.\"},{\"question\":\"How does LP-SFT preserve pretrained distributional structure during fine-tuning?\",\"answer\":\"LP-SFT builds an adaptive support of alternative tokens from the frozen base model, removes the supervised target token from that support, and applies a locally normalized preservation loss so the relative structure among alternatives is maintained.\"}]",1784183457,53,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"lp-sft-local-preserving-supervised-fine-tuning-via-multimodal-entropy-structure","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/lp-sft-local-preserving-supervised-fine-tuning-via-multimodal-entropy-structure/82854/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why can standard supervised fine-tuning lead to capability degradation?","Question",{"text":75,"@type":76},"Cross-entropy fine-tuning repeatedly suppresses plausible alternatives and focuses only on the observed target token, which can distort the pretrained local preference structure under distribution shift.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What does the multimodal entropy structure refer to in pretrained models?",{"text":80,"@type":76},"Next-token prediction entropies form distinct peaks near ln k for integer k, indicating discrete uncertainty regimes where some positions are nearly deterministic while others admit a small set of alternatives.",{"name":82,"@type":73,"acceptedAnswer":83},"How does LP-SFT preserve pretrained distributional structure during fine-tuning?",{"text":84,"@type":76},"LP-SFT builds an adaptive support of alternative tokens from the frozen base model, removes the supervised target token from that support, and applies a locally normalized preservation loss so the relative structure among alternatives is maintained.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]