[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-273392-105":59,"doc-detail-273392-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","safesteer-localized-on-policy-distillation-for-efficient-safety-alignment","SafeSteer - Localized On-Policy Distillation for Efficient Safety Alignment","","Aligning Large Language Models (LLMs) with human values often reduces general capability, known as the alignment tax. Existing safety alignment techniques rely on large general-purpose datasets or auxiliary reward models, and commonly optimize dual objectives with global trade-offs. SafeSteer argues safety signals are sparse in the output distribution, so alignment should be localized. It performs on-policy distillation confined to safety tokens, using an activation-steering safety teacher and a voting-based safety token selection algorithm to restrict reverse-KL penalties, preserving general abilities.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/safesteer-localized-on-policy-distillation-for-efficient-safety-alignment/273392/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/safesteer-localized-on-policy-distillation-for-efficient-safety-alignment/273392.png","ImageObject",300,407,{"name":92,"@type":93},"Patrick","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-02","2026-09-15",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",14,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What problem does SafeSteer target in LLM safety alignment?","Question",{"text":113,"@type":114},"SafeSteer targets the alignment tax, where safety alignment methods degrade LLM general capabilities. It addresses the inefficiency of global trade-offs when safety features are sparse in the output distribution.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How does SafeSteer build its safety teacher?",{"text":118,"@type":114},"SafeSteer constructs a safety teacher using activation steering. It extracts a refusal direction and injects it into the model’s residual stream activations at inference to produce stable safety signals.",{"name":120,"@type":111,"acceptedAnswer":121},"How does SafeSteer keep general capability from being harmed during training?",{"text":122,"@type":114},"It restricts the reverse-KL penalty to a selected subset of safety tokens. This token subset is mined by contrasting output distributions of the safety teacher and the base model using a voting-based aggregation algorithm.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},273392,1789471221,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":56,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":130,"read_time":144,"preview_page_count":4},549758146520,"https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470","SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment  \nHao Li 1∗ , Jingkun An1∗ , Zijun Song2∗ , Pengyu Zhu3 , Rui Li4  \nHao Wang 1 , Wendi Feng 1 , Yesheng Liu5 , Lijun Li6 , Jin-Ge Yao7 , Lei Sha 1†  \n1 Beihang University 2 Beijing Institute of Technology  \n3 Beijing University of Posts and Telecommunications 4 Peking University  \n5 Institute of Automation, Chinese Academy of Sciences  \n6 Shanghai Artificial Intelligence Laboratory 7 Beijing Academy of Artificial Intelligence  \n[hao612@buaa.edu.cn](hao612@buaa.edu.cn) [anjingkun02@gmail.com](anjingkun02@gmail.com) [zijunsong@bit.edu.cn](zijunsong@bit.edu.cn) [shalei@buaa.edu.cn](shalei@buaa.edu.cn)  \narXiv : 2606 .02530v2 [cs .AI] 2 1 Aug 2026  \nAbstract  \nAligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive generalpurpose data or auxiliary reward models.  \nIn this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performson-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities.  \nExperimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at [https://anjingkun.github.io/SafeSteer/](https://anjingkun.github.io/SafeSteer/) .  \n1 Introduction  \nLarge language models (LLMs) are now widely deployed in conversational and agentic settings (Liu et al., 2025 ; Zeng et al., 2026), but they remain prone to producing harmful content such as biased remarks, criminal advice, and other unsafe outputs (Mazeika et al., 2024 ; Qi et al., 2024) . Aligning LLM behavior with human values (Ouyang et al.,  \n⋆ Equal contribution † Corresponding author  \nGeneral Capability   \nFigure 1: Safety–capability trade-off on Qwen2.5-7BInstruct. Each point is a method, with the gray point marking the base model. Our SafeSteer achieves the highest safety score while preserving general capability.  \n2022) is therefore essential. However, mainstream safety alignment methods often degrade the models’ general capabilities, a phenomenon known asthe alignment tax (Huang et al., 2025) .  \nExisting safety alignment methods often frame the problem as dual-objective optimization, forcing a trade-off between safety and general capabilities (Dai et al., 2024) . Prior works have explored strategies such as mixing massive amounts of generalpurpose data into the training set (Zhang et al., 2025 ; Zou et al., 2024), orthogonally projecting safety gradients into the null space of general capability representations (Niu et al., 2025), or training auxiliary reward models (Dai et al., 2024 ; Huang et al., 2024) . However, since safety features are inherently sparse within an LLM’s output distribution, alignment demands not global trade-offs but rather localized modifications. This entails adjusting safety features while mitigating forgetting.  \nOn-policy distillation (OPD) (Gu et al., 2024 ; Agarwal et al., 2024) mitigates global distribution shifts by training on self-sampled data via reverse-KL minimization. However, to provide  \nsupervisory signals, standard LLM OPD typically necessitates an exte","cbCaimId8PLihClo","https://ap.wps.com/l/cbCaimId8PLihClo","pdf",2600131,"English","# Introduction\n## Alignment tax and dual-objective trade-offs\n## On-policy distillation and teacher construction\n## Localized updates via safety-token subset\n## SafeSteer contributions and experimental setup","[{\"question\":\"What problem does SafeSteer target in LLM safety alignment?\",\"answer\":\"SafeSteer targets the alignment tax, where safety alignment methods degrade LLM general capabilities. It addresses the inefficiency of global trade-offs when safety features are sparse in the output distribution.\"},{\"question\":\"How does SafeSteer build its safety teacher?\",\"answer\":\"SafeSteer constructs a safety teacher using activation steering. It extracts a refusal direction and injects it into the model’s residual stream activations at inference to produce stable safety signals.\"},{\"question\":\"How does SafeSteer keep general capability from being harmed during training?\",\"answer\":\"It restricts the reverse-KL penalty to a selected subset of safety tokens. This token subset is mined by contrasting output distributions of the safety teacher and the base model using a voting-based aggregation algorithm.\"}]","SafeSteer - Localized On-Policy Distillation for Efficient Safety Alignment | PDF",48]