[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327449-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327449-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","generalization-in-offline-rl-the-structure-is-more-important-than-the-amount-of-pessimism","Generalization in Offline RL - The Structure is More Important than the Amount of Pessimism","","Pessimism can counteract overestimation bias in offline reinforcement learning, yet excessive conservatism may reduce generalization quality. This work shows that in contextual MDPs, overly pessimistic learning does not inherently block optimal generalization. The key factor is whether the pessimistic value-function structure preserves the symmetries of the optimal solution. When pessimism violates required symmetries induced by dataset coverage, generalization degrades; data augmentation can help via symmetry-focused consistency loss during policy extraction. Results with IQL and CQL confirm improved generalization on a rotationally symmetric reacher task.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/generalization-in-offline-rl-the-structure-is-more-important-than-the-amount-of-pessimism/327449/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/generalization-in-offline-rl-the-structure-is-more-important-than-the-amount-of-pessimism/327449.png","ImageObject",300,407,{"name":42,"@type":43},"Ivy","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-16","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",11,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why can pessimism harm generalization in offline reinforcement learning?","Question",{"text":63,"@type":64},"Overly conservative pessimistic value learning can be linked to hindering generalization to out-of-distribution actions and impairing trajectory stitching in continuous settings.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What determines generalization performance in the paper’s analysis?",{"text":68,"@type":64},"Generalization depends not on how much pessimism is used, but on whether the pessimistic value-function structure respects the symmetries of the optimal solution.",{"name":70,"@type":61,"acceptedAnswer":71},"How does dataset coverage relate to the pessimistic structure?",{"text":72,"@type":64},"In offline RL, the structure of the pessimism is induced by how well the dataset coverage supports (or contradicts) the symmetries needed for optimal generalization.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327449,1790558593,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},549758252649,"https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819","arXiv :2607 .02288v 1 [ cs .LG] 2 Jul 2026  \nGeneralization in offline RL: The structure is more important than the amount of pessimism  \nMax Weltevrede 1 , Matthijs T. J. Spaan 1 , Wendelin Böhmer 1 [m.r.weltevrede@tudelft.nl](m.r.weltevrede@tudelft.nl)  \n1Delft University of Technology, Delft, The Netherlands  \nAbstract  \nWhile pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate that being overly pessimistic does not inherently prevent optimal generalization in contextual MDPs (CMDPs) . Instead, we argue successful generalization depends not on the amount of pessimism, but whether the pessimistic structure respects the underlying symmetries of the optimal solution. We prove that a mildly pessimistic, non-symmetric value function can generalize worse than an overly pessimistic, symmetric one. In offline RL, the structure of the pessimism is determined by the structure of the dataset coverage. As such, enforcing a symmetric value function can be non-trivial, and might require techniques such as data augmentation (DA) . Inspired by our theoretical results, we argue that DA can best be applied through a consistency loss during policy extraction, rather than the common practice of (regular) offline training on an augmented dataset. This is empirically validated using IQL and CQL on a rotationally symmetric reacher environment.  \n1 Introduction  \nThe objective of offline reinforcement learning (RL) is to find a policy that maximizes expected returns in an environment, by only training on a given, fixed dataset of collected experiences (Levine et al., 2020) . In order to avoid common offline RL pathologies, many methods employ pessimistic value learning or other forms of conservatism (Kumar et al., 2020 ; Kostrikov et al., 2022 ; Fujimoto & Gu, 2021 ; An et al., 2021) . However, being overly conservative has been associated with hindering generalization to out-of-distribution actions or impeding trajectory stitching in continuous environments (Wang et al., 2024 ; Ma et al., 2023 ; Mediratta et al., 2024 ; Park et al., 2024) . Recently, several approaches have tried to reduce the level of conservatism in order to improve generalization (Mao et al., 2024 ; Lyu et al., 2022 ; Shimizu et al., 2024), even claiming that, while some level of conservatism is necessary to find the best policy, it should be as mild as possible to improve this kind of generalization (Lyu et al., 2022) . However, it is not yet clear whether this notion also holds for generalization to new scenarios or environments.  \nIn this paper, we argue that it is not the amount of pessimism, but its structure that is inherently important for generalization in the zero-shot policy transfer (ZSPT, Kirk et al., 2023) setting to new and unseen testing environments. Fundamentally, we argue that generalization is about learning the underlying structure of the optimal policy or value function over training and testing states, often formalized as symmetries. For the theoretical analysis in this paper, we use the generalization-throughinvariance ZSPT (GTI-ZSPT, Weltevrede et al., 2025) setting, in which optimal generalization is achieved by learning the correct symmetry from the training data. In this setting, we prove that as long as the pessimistic value learning is symmetric, optimal generalization performance can be achieved even with arbitrary large levels of pessimism. Furthermore, we prove that for certain instances of the  \nGTI-ZSPT problem, there exists mild forms of pessimism, that violate this symmetric structure, that are guaranteed to generalize arbitrarily worse than overly pessimistic, but symmetric ones.  \nAs such, we argue that optimal generalization is not determined by how pessimistic the agent is, but by whether the pessimism breaks the symmetries the optimal solution should have. Although this notion ","cbCaibZEaAQZM9g9","https://ap.wps.com/l/cbCaibZEaAQZM9g9","pdf",530276,31,"English","# Abstract\n# Introduction\n# Background","[{\"question\":\"Why can pessimism harm generalization in offline reinforcement learning?\",\"answer\":\"Overly conservative pessimistic value learning can be linked to hindering generalization to out-of-distribution actions and impairing trajectory stitching in continuous settings.\"},{\"question\":\"What determines generalization performance in the paper’s analysis?\",\"answer\":\"Generalization depends not on how much pessimism is used, but on whether the pessimistic value-function structure respects the symmetries of the optimal solution.\"},{\"question\":\"How does dataset coverage relate to the pessimistic structure?\",\"answer\":\"In offline RL, the structure of the pessimism is induced by how well the dataset coverage supports (or contradicts) the symmetries needed for optimal generalization.\"}]","Generalization in Offline RL - The Structure is More Important than the Amount of Pessimism | PDF",1789977228,78]