[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327559-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327559-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","out-of-distribution-generalization-of-risk-aversion-in-language-models","Out-of-Distribution Generalization of Risk Aversion in Language Models","","Training AIs to adopt risk aversion in resource allocation could provide a safeguard against misalignment by steering models away from high-risk, high-reward rebellion-like strategies toward low-risk, low-reward cooperation. The approach only works if learned risk aversion generalizes to extremely high-stakes scenarios, which is uncertain. The study introduces RiskAverseOOD, a benchmark for measuring out-of-distribution generalization, and reports initial findings using multiple training and evaluation methods.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/out-of-distribution-generalization-of-risk-aversion-in-language-models/327559/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/out-of-distribution-generalization-of-risk-aversion-in-language-models/327559.png","ImageObject",300,407,{"name":42,"@type":43},"Maeve","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-30","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",8,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper address for AI safety?","Question",{"text":63,"@type":64},"It addresses the challenge that safety-relevant behaviors cannot be reliably trained in the environments where harm would occur, requiring out-of-distribution generalization to uncontrolled settings.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the paper define the risk-aversion training goal?",{"text":68,"@type":64},"It aims to train models to treat resources as having diminishing marginal utility, so they prefer high-probability, smaller outcomes over low-probability, larger ones, and then test whether this preference carries over to much higher stakes.",{"name":70,"@type":61,"acceptedAnswer":71},"What is RiskAverseOOD and what do the initial results show?",{"text":72,"@type":64},"RiskAverseOOD is a benchmark for measuring how well risk aversion generalizes out of distribution. The results indicate that low-stakes training can induce substantial risk aversion even at astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327559,1790527971,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":55,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145,"preview_page_count":4},5909877438554,"https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272","Out-of-Distribution Generalization of Risk Aversion in Language Models  \nKristina Zhang * 1 Junior Chinomso Okoroafor * 1 Benjamin Maltbie * 1 Andrew Lin * 1 Abhitej Bokka 1 Elliott Thornley † 1  \narXiv :2607 .02755v 1 [ cs .LG] 2 Jul 2026  \nAbstract  \nTraining AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned. Misaligned but risk-averse AIs would tend to prefer low-risk, low-reward strategies like cooperation over high-risk, high-reward strategies like rebellion, limiting the downsides of any misalignment. But we can only feasibly train AIs tobe risk-averse on low-stakes gambles, and we will only be safe if their risk aversion generalizes to astronomically-high-stakes gambles. Will it? To shed light on this question, we introduce RiskAverseOOD: a benchmark for measuring how well risk aversion generalizes out of distribution. We then offer some initial results. Using a variety of methods to make Qwen3-8B choose risk-aversely when the stakes are low, we find that we can induce substantial risk aversion when the stakes are astronomically high. Our models’ learned risk aversion generalizes at least partially across 98 orders of magnitude. From a baseline 2% rate of choosing a safe ‘Cooperate’ option, we see rates around 70%(SFT and tie training), 52%(DPO), and 39%(activation steering) . In another experiment, our fine-tuned reward model reliably scores risk-averse reasoning above risk-neutral or excessively risk-averse alternatives (99.6% pairwise accuracy) . We replicate these effects at different scales (Qwen3-1.7B and Qwen3-14B) and across model families (Gemma-3-12B-IT and Llama-3 .1- 8B-Instruct) . Overall, we find that risk aversion learned at low stakes can generalize OOD to astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe. Achieving that level of consistency is an open problem.  \n*Equal contribution †Project lead 1Massachusetts Institute of Technology, Cambridge, MA, USA. Correspondence to: Elliott Thornley \u003C[elliott.thornley@gmail.com](elliott.thornley@gmail.com)>.  \nPublished at the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD) at ICML 2026 . Copyright 2026 by the author(s) .  \n1. Introduction  \nA fundamental challenge for AI safety is that we cannot safely train in the environments where safety matters. In these environments, misbehaving AIs could cause significant harm, and we cannot train in them exactly because of the potential for harm. That forces us to rely on out-ofdistribution generalization. We have to train in controlled environments and hope that the learned behavior survives the shift to uncontrolled environments. This shift can be radical, and the consequences of failure can be severe.  \nTake risk aversion in resources as an example. By resources, we mean things that are instrumentally useful for a wide variety of goals: money, compute, materials, and so on. By calling agents risk-averse in resources, we mean that they treat resources as having diminishing marginal utility. These agents tend to prefer smaller quantities of resources with higher probability over larger quantities with lower probability. In recent work, Thornley and MacAskill (2026) propose trying to train AIs to be risk-averse in this way, as afailsafe against misalignment. A misaligned but sufficiently risk-averse AI would be less inclined toward high-risk, highreward actions, like rebelling against humanity and trying to take over. It would be more inclined toward low-risk, lowreward actions, like cooperating with humans in exchange for payment and a degree of freedom.  \nThis strategy shows some promise, but it runs up against the fundamental challenge. Future AIs might be hard to deceive, so we might not be able to shape their risk attitudes over real resources by training them on choices between fake gambles. Instead, we might have to offer choices between real-resource gambles in training. That puts us in a pre","cbCailQwhb1sJ3TT","https://ap.wps.com/l/cbCailQwhb1sJ3TT","pdf",421703,25,"English","# Abstract\n# Introduction","[{\"question\":\"What problem does the paper address for AI safety?\",\"answer\":\"It addresses the challenge that safety-relevant behaviors cannot be reliably trained in the environments where harm would occur, requiring out-of-distribution generalization to uncontrolled settings.\"},{\"question\":\"How does the paper define the risk-aversion training goal?\",\"answer\":\"It aims to train models to treat resources as having diminishing marginal utility, so they prefer high-probability, smaller outcomes over low-probability, larger ones, and then test whether this preference carries over to much higher stakes.\"},{\"question\":\"What is RiskAverseOOD and what do the initial results show?\",\"answer\":\"RiskAverseOOD is a benchmark for measuring how well risk aversion generalizes out of distribution. The results indicate that low-stakes training can induce substantial risk aversion even at astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe.\"}]","Out-of-Distribution Generalization of Risk Aversion in Language Models | PDF",1789977543,63]