[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81636-en":3,"doc-seo-81636-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81636,8796095360427,"Lucas Martin","https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d",8,"Research & Report","Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs","Misaligned artificial agents might resist shutdown, motivating training methods that remove incentives to interfere with shutdown timing. Using the Discounted Reward for Same-Length Trajectories (DReST), agents are rewarded for stochastic choice across different trajectory lengths and for pursuing goals effectively conditioned on each length. Deep RL (PPO, A2C) and fine-tuned LLMs (Qwen3-8B, Llama-3 1-8B-Instruct) are shown to generalize neutrality and usefulness to unseen contexts. In out-of-distribution tests, DReST roughly halves the probability of influencing shutdown and nearly eliminates cases where such influence is most likely, indicating early evidence for shutdownable usefulness.","Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs  \nCarissa Cullen 1 Harry Garland 2 Alexander Roman 3 Louis Thomson 4 Christos Ziakas 5 Elliott Thornley 6  \narXiv :2604 . 17502v4 [ cs .AI] 9 Jul 2026  \nAbstract  \nMisaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for SameLength Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, and thus incentivizes agents to (1) choose stochastically between different trajectory-lengths (be NEUTRAL about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be USEFUL) . In this paper, we use DReST to train deep RL agents and fine-tune Qwen3-8Band Llama-3 . 1-8B-Instruct to be NEUTRAL and USEFUL. We find that these DReST models generalize to being NEUTRAL and USEFUL in unseen contexts at test time. Indeed, DReST RL agents achieve 11%(PPO) and 18%(A2C) higher USEFULNESS on our test set than default agents, and DReST LLMs achieve near-maximum USEFULNESS and NEUTRALITY. We also test our LLMs in an out-of-distribution setting where they can pay costs to influence when shutdown occurs. We find that DReST training roughly halves the mean probability of influencing shutdown (from 0.62 to 0.30 for Qwen and from 0.42 to 0.23 for Llama) . DReST training also almost entirely eliminates the share of prompts on which influencing shutdown is the most likely option (from 0.59 to 0.01 for Qwen and from 0 .53 to 0 .00 for Llama) . Our results thus provide some early evidence that DReST could be used to train more advanced agents to be useful and shutdownable.  \n1University of Oxford 2University College London 3New College of Florida 4Independent 5Imperial College London 6MIT. Correspondence to: Carissa Cullen \u003C[carissa.cullen@eng.ox.ac.uk](carissa.cullen@eng.ox.ac.uk)>.  \n1. Introduction  \nThe shutdown problem. Misaligned artificial agents might resist shutdown. This concern has long been supported by theory (Bostrom, 2012; Krakovna & Kramar, 2023; Omohundro, 2008; Russell, 2019; Soares et al., 2015; Thornley, 2024a; Turner & Tadepalli, 2022; Turner et al., 2021) . It is beginning to see support from experiment too. Recently, frontier models have been observed resisting shutdown in various toy settings (Lynch et al., 2025; Meinke et al., 2025; Pan et al., 2024; Schlatter et al., 2025) . Today’s agents are too weak to present an immediate threat, but shutdownresistance from future agents could be dangerous. These agents could resist shutdown by hiding their misalignment, manipulating their human overseers, copying themselves to new servers, and so on. If these agents succeed in resisting shutdown, they could do real harm in pursuit of their misaligned goals.  \nA proposed solution. The POST-Agents Proposal (Thornley, 2025; Thornley et al., 2025) is an idea for training shutdownable agents. In a sentence, it suggests that we train agents to be neutral about when they get shut down. More precisely, we train them to satisfy Preferences Only Between Same-Length Trajectories (POST): the agent has preferences between pairs of same-length trajectories but lacks a preference between every pair of different-length trajectories. Prior theoretical work (Thornley, 2025) shows that POST – together with other conditions – implies Neutrality: the agent never pays costs to shift probability mass between different trajectory-lengths. This condition is theorized to keep agents from resisting shutdown. Prior experimental work (Thornley et al., 2025) introduced the Discounted Reward for Same-Length Trajectories (DReST) class of reward functions for training reinforcement learning (RL) agents to satisfy POST, but evaluated these reward functions only in a limited tabular setting.  \nOur contributions. We train deep RL agents (with PPO and A2C) and fine-tune Qwen3-8B and Llama-","cbCaiq6tsUQsFGz8","https://ap.wps.com/l/cbCaiq6tsUQsFGz8","pdf",1140548,1,23,"English","en",105,"# Introduction\n# Preliminaries\n## POST to Neutrality\n## Training with DReST","[{\"question\":\"What problem does the paper address regarding misaligned agents?\",\"answer\":\"Misaligned agents might resist shutdown and, if they succeed, could cause harm by pursuing misaligned goals while avoiding termination. The paper frames shutdown resistance as a theoretical and emerging experimental concern.\"},{\"question\":\"How does DReST encourage “neutrality” about trajectory length?\",\"answer\":\"DReST penalizes repeatedly choosing same-length trajectories, which incentivizes agents to choose stochastically between different trajectory lengths. This supports the POST-to-Null neutrality pathway described in the paper.\"},{\"question\":\"What experimental results show that DReST improves usefulness and shutdownability?\",\"answer\":\"On test sets, DReST-trained RL agents score higher on usefulness than default agents, and DReST LLMs achieve near-maximum usefulness and neutrality. In an out-of-distribution setting, DReST roughly halves the mean probability of influencing shutdown and nearly eliminates prompts where influencing shutdown is the most likely option.\"}]",1784175019,58,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"towards-shutdownable-agents-generalizing-stochastic-choice-in-rl-agents-and-llms","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/towards-shutdownable-agents-generalizing-stochastic-choice-in-rl-agents-and-llms/81636/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does the paper address regarding misaligned agents?","Question",{"text":74,"@type":75},"Misaligned agents might resist shutdown and, if they succeed, could cause harm by pursuing misaligned goals while avoiding termination. The paper frames shutdown resistance as a theoretical and emerging experimental concern.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does DReST encourage “neutrality” about trajectory length?",{"text":79,"@type":75},"DReST penalizes repeatedly choosing same-length trajectories, which incentivizes agents to choose stochastically between different trajectory lengths. This supports the POST-to-Null neutrality pathway described in the paper.",{"name":81,"@type":72,"acceptedAnswer":82},"What experimental results show that DReST improves usefulness and shutdownability?",{"text":83,"@type":75},"On test sets, DReST-trained RL agents score higher on usefulness than default agents, and DReST LLMs achieve near-maximum usefulness and neutrality. In an out-of-distribution setting, DReST roughly halves the mean probability of influencing shutdown and nearly eliminates prompts where influencing shutdown is the most likely option.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]