[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-573897-105":3,"detail-sidebar-cat-1-en-105":79,"doc-detail-573897-en":125},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":72,"head_meta":74,"extra_data":76,"updated_unix":78},105,"en","robust-failure-conservative-repair-textual-knowledge-distillation-from-cross-model-failures","Robust Failure, Conservative Repair - Textual Knowledge Distillation from Cross-Model Failures","","Failure-based textual knowledge distillation uncovers gaps in a model’s knowledge by leveraging task errors, but transferred rules may be unstable. The work defines rule atoms as standalone inference-time injections that can encode transferable knowledge or harmful model-specific patches, and may be misapplied to unrelated cases. It proposes Robust Failure, Conservative Repair (RFCR), which extracts rules from failures shared across models, sharpens application boundaries using boundary cases, and abstains when rules are unhelpful. On BIG-Bench Hard, RFCR raises accuracy from 68.50% to 71.25% without degrading previously correct performance, supported by ablations and diagnostics.",{"@graph":14,"@context":71},[15,34,54],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/template/","Template",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/template/presentations/","Presentations",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/template/robust-failure-conservative-repair-textual-knowledge-distillation-from-cross-model-failures/573897/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":48,"encodingFormat":47,"isAccessibleForFree":49,"interactionStatistic":50},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/robust-failure-conservative-repair-textual-knowledge-distillation-from-cross-model-failures/573897.png","ImageObject",442,249,{"name":42,"@type":43},"lugxt","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-14",true,{"@type":51,"interactionType":52,"userInteractionCount":22},"InteractionCounter",{"@type":53},"ViewAction",{"@type":55,"mainEntity":56},"FAQPage",[57,63,67],{"name":58,"@type":59,"acceptedAnswer":60},"What is failure-based textual knowledge distillation in this work?","Question",{"text":61,"@type":62},"It discovers transferable knowledge by analyzing a source model’s task errors and converting useful information into inference-time textual artifacts. The paper focuses on failures where the source model answers incorrectly with a baseline cheat sheet.","Answer",{"name":64,"@type":59,"acceptedAnswer":65},"Why can knowledge transfer from failures be unstable?",{"text":66,"@type":62},"A failure-derived rule atom can encode a model-specific patch that fixes the source model’s residual errors but confuses other models. It can also be misapplied to unrelated cases, leading the target model to flip its answer due to irrelevant information.",{"name":68,"@type":59,"acceptedAnswer":69},"How does RFCR differ from single-model RFR?",{"text":70,"@type":62},"RFCR derives rule atoms from failures shared across models and uses activation boundary controls to restrict rule application. It abstains when no useful rule is found, aiming to improve across-model performance rather than only the source model.","https://schema.org",{"og:url":32,"og:type":73,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":75,"canonical":32},"index,follow",{"doc_id":77,"site_id":7},573897,1791978873,{"code":4,"msg":80,"data":81},"success",[82,86,91,96,101,106,111,116,121],{"id":83,"doc_module":22,"doc_module_name":25,"category_name":29,"show_sort_weight":84,"slug":85},11,90,"presentations",{"id":87,"doc_module":22,"doc_module_name":25,"category_name":88,"show_sort_weight":89,"slug":90},12,"Resumes",80,"resumes",{"id":92,"doc_module":22,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},14,"Invoices",70,"invoices",{"id":97,"doc_module":22,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},15,"Posters",60,"posters",{"id":102,"doc_module":22,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},16,"Social Media",50,"social-media",{"id":107,"doc_module":22,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},17,"Forms",40,"forms",{"id":112,"doc_module":22,"doc_module_name":25,"category_name":113,"show_sort_weight":114,"slug":115},18,"Letters",30,"letters",{"id":117,"doc_module":22,"doc_module_name":25,"category_name":118,"show_sort_weight":119,"slug":120},21,"Paper Templates",5,"papers-templates",{"id":122,"doc_module":22,"doc_module_name":25,"category_name":123,"show_sort_weight":4,"slug":124},158,"General","general-158",{"code":4,"msg":80,"data":126},{"doc_id":77,"user_id":127,"nickname":42,"user_avatar":128,"doc_module":22,"category_id":83,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":129,"file_id":130,"file_url":131,"file_type":132,"file_size":133,"view_count":22,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":102,"language":134,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":135,"faqs":136,"seo_title":137,"seo_description":12,"update_tm":138,"read_time":139},3985747862600,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","Robust Failure, Conservative Repair: Textual Knowledge Distillation from Cross-Model Failures  \nAndrew Ren and Haokun Liu and Chenhao Tan  \nUniversity of Chicago  \n{renqy, haokunliu, [chenhao}@uchicago.edu](chenhao}@uchicago.edu)  \narXiv : 2609 .25400v 1 [cs .AI] 2 1 Sep 2026  \nAbstract  \nFailure-based textual knowledge distillation aims to discover gaps in a model’s knowledge by examining its task errors. The distilled knowledge can be useful for the reasoning of both this model (“source model”) and other models. However, this transfer of knowledge may not be stable. We define a rule atom to bea standalone rule injected into a model’s textual input at inference time. A rule atom can encode transferable task knowledge or model-specific reasoning patches that can confuse other models. Also, the injected rule atoms can be misapplied to unrelated cases, causing the model to incorrectly flip its answer based on irrelevant information. Building on a pipeline that distills training examples into task-specific cheat sheets that aid model reasoning, we examine when failure-derived rules can improve these cheat sheets. Our early experiment shows rule distillation from a single model’s failures underperforms the baseline cheat sheet on nonsource model families. This motivates Robust Failure, Conservative Repair (RFCR), a textual distillation procedure that derives rules from failures shared across models, sharpens their application boundaries using boundary cases, and abstains when no useful rule is found. On a 400-item BIG-Bench Hard (Suzgun et al., 2022) task set, RFCR improves the baseline cheat sheets from 68.50% to 71.25%(+2.75 pp;  \n95% CI [+1.25,+4.50]) without performance degradation on previously correct cases. Ablations and cross-model diagnostics support that accuracy gains come from both new knowledge injection and strict rule-application control.  \n1 Introduction  \nLarge language models (LLMs) can turn concrete examples and task heuristics into textual artifacts. One example is a cheat sheet: a compact set of  \nCode and artifacts are available at: [https://github](https://github) . com/ChicagoHAI/RFCR.  \ntask rules distilled from sample task cases and provided at inference time. Cheat sheet in-context learning (CS-ICL) demonstrates this capability by compressing many-shot demonstrations into areusable task-specific cheat sheet (Honda et al., 2025) . Similarly, the SAIR Mathematics Distillation Challenge asks whether strong mathematical reasoning can be compressed into a compact, human-readable cheat sheet that helps a weaker model on equational-theory problems (SAIR Foundation, 2026) . In both cases, we construct concise and readable cheat sheets that can efficiently transfer strong-model reasoning to cheaper models. A natural question is whether such useful artifacts can be distilled from model failures.  \nThis question is a form of textual knowledge distillation. We focus on cases that the source model answers incorrectly when receiving a baseline cheat sheet. These failures may expose missing knowledge. For example, if a source model repeatedly errs on a geometric relation, a pronoun-resolution pattern, or a logical form, it likely lacks knowledge needed to answer such questions correctly. The difficulty is that source-model failures do not automatically produce transferable task knowledge. A failure-derived rule may encode a model-specific patch instead of transferable knowledge. Such a patch may correct one model’s residual errors but hurt a target model that does not share those errors. Our Robust Failure Repair (RFR) diagnostic exhibits this behavior: source-only failure refinement underperforms the static CS-ICL baseline across all tested non-source model families.  \nWe therefore introduce Robust Failure, Conservative Repair (RFCR), a conservative textual distillation pipeline that derives rule atoms from failures shared across models. We define a rule atom to bea standalone rule injected into the model’s textual in","cbCailx2Q40H4gJ8","https://ap.wps.com/l/cbCailx2Q40H4gJ8","pdf",1645773,"English","# Abstract\n# Introduction\n## Cheat sheets and in-context learning\n## Failure-based textual knowledge distillation\n## Robust Failure, Conservative Repair (RFCR)\n## Experimental setup and results","[{\"question\":\"What is failure-based textual knowledge distillation in this work?\",\"answer\":\"It discovers transferable knowledge by analyzing a source model’s task errors and converting useful information into inference-time textual artifacts. The paper focuses on failures where the source model answers incorrectly with a baseline cheat sheet.\"},{\"question\":\"Why can knowledge transfer from failures be unstable?\",\"answer\":\"A failure-derived rule atom can encode a model-specific patch that fixes the source model’s residual errors but confuses other models. It can also be misapplied to unrelated cases, leading the target model to flip its answer due to irrelevant information.\"},{\"question\":\"How does RFCR differ from single-model RFR?\",\"answer\":\"RFCR derives rule atoms from failures shared across models and uses activation boundary controls to restrict rule application. It abstains when no useful rule is found, aiming to improve across-model performance rather than only the source model.\"}]","Robust Failure, Conservative Repair - Textual Knowledge Distillation from Cross-Model Failures | PDF",1791950619,6]