[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273551-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273551-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","on-the-fragility-of-self-improving-agents-variance-task-order-and-underspecification","On the Fragility of Self-Improving Agents - Variance, Task Order, and Underspecification","","Memory-based self-improving agents that learn from an online task stream by maintaining textual memory promise strong long-term improvement, yet their reliability has been insufficiently stress-tested. This work reevaluates two representative methods with two extensions: multiple self-improving runs to measure variance and randomized task shuffling to probe task-order effects. Results show noisy evaluation, increased run variance in most cases, and performance highly dependent on task order. Manually analyzing agent memories links fragility to task and environment underspecification and tests specification enhancements.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/on-the-fragility-of-self-improving-agents-variance-task-order-and-underspecification/273551/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/on-the-fragility-of-self-improving-agents-variance-task-order-and-underspecification/273551.png","ImageObject",300,407,{"name":42,"@type":43},"Aria","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-11","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",13,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What reliability aspects does the paper focus on for self-improving agents?","Question",{"text":63,"@type":64},"It targets evaluation reliability, specifically run-to-run variance and the effect of task ordering on agent improvement.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How do multiple self-improving runs change the observed results?",{"text":68,"@type":64},"Measuring variance across multiple runs reveals that performance is noisy, and adding a self-improving loop can amplify this variance.",{"name":70,"@type":61,"acceptedAnswer":71},"Why does task order significantly affect agent performance?",{"text":72,"@type":64},"The paper argues that default task orders create an implicit curriculum; when tasks are shuffled, performance degrades instead of improving.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273551,1789962522,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},2336464648322,"https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488","arXiv : 2608 . 18066v2 [cs .AI] 6 Sep 2026  \nOn the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification  \nQinyuan Ye Yu Li Yada Pruksachatkun Jiaxin Zhang Chien-Sheng Wu  \nSalesforce AI Research  \nAbstract  \nMemory-based self-improving agents—those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank—have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive reevaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Empirically, when these methods are applied, we observe the variance across runs increase in 71% of cases, and the gap between the best and worst runs of the same experiment can reach up to 10 percentage points. Second, the agent’s improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. In contrast, when evaluated under a shuffled task order, agent performance degrades (-4.5%) instead of exhibiting the expected improvement (+1.5%) .  \nTo better understand this fragility, we manually examine the agents’ memory and hypothesize that task and environment underspecification contribute to this fragility. Without clear specifications, agents generate plausible yet inapplicable memories (e.g., recommending API usage in a browser-only environment) that may distract the agent from feasible strategies. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubricsand environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under realistic, challenging conditions. Moreover, our findingson underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.1  \n1 Introduction  \nThe development of self-improving agents [1, 2, 3]—systems designed to learn from past experiencesand autonomously refine their performance over time—has emerged as a highly promising research direction. If they realize this potential, they could not only better automate routine workflows and create significant economic impact, but also enable highly adaptable systems and facilitate openended research discovery. While recent works show progress in this direction [4, 5, 6, 7, 8, 9], little  \n1§ SalesforceAIResearch/self-improve-fragility  Salesforce/self-improve-fragility  \nPreprint.  \nfocus has been placed on stress-testing the reliability of such systems [10] . In practical settings, such as enterprise deployments, there is often minimal tolerance for error; an initial failure risks losing user trust. Moreover, initial mistakes can cascade silently over the long term, causing irreversible or prohibitively costly impacts if adopted in high-stakes domains. Before we can confidently deploy these systems, we must critically evaluate their reliability under realistic, challenging conditions. This work begins with a comprehensive re-evaluation of two representative memory-based selfimproving systems—Agent Workflow Memory [11] and Reason","cbCaiuMkgOuOYP0G","https://ap.wps.com/l/cbCaiuMkgOuOYP0G","pdf",552787,32,"English","# Abstract\n# Introduction\n## Reliability stress-testing and evaluation design\n## Variance amplification across runs\n## Sensitivity to task order\n## Interpreting fragility via memory underspecification\n## Mitigations and outlook","[{\"question\":\"What reliability aspects does the paper focus on for self-improving agents?\",\"answer\":\"It targets evaluation reliability, specifically run-to-run variance and the effect of task ordering on agent improvement.\"},{\"question\":\"How do multiple self-improving runs change the observed results?\",\"answer\":\"Measuring variance across multiple runs reveals that performance is noisy, and adding a self-improving loop can amplify this variance.\"},{\"question\":\"Why does task order significantly affect agent performance?\",\"answer\":\"The paper argues that default task orders create an implicit curriculum; when tasks are shuffled, performance degrades instead of improving.\"}]","On the Fragility of Self-Improving Agents - Variance, Task Order, and Underspecification | PDF",1789472025,81]