[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327424-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327424-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","sunta-hierarchical-video-prediction-with-surprise-based-chunking-327424","SUNTA - Hierarchical Video Prediction with Surprise-based Chunking","","Hierarchical state-space models support long-horizon prediction by segmenting sequences into temporal chunks, yet results strongly depend on how chunk boundaries are chosen. Prior approaches use fixed-length chunking or similarity-based boundary detection, which can misalign with intrinsic temporal structure. The work argues for prediction-error-driven chunking, but integrating surprise into HSSMs faces hierarchical collapse in end-to-end training and missing surprise signals in open-loop rollouts. It proposes Surprise-based Nested Temporal Abstraction (SUNTA) with decoupled training and a top-down inconsistency metric to set boundaries. Experiments on 2D and 3D video prediction show superior long-horizon accuracy beyond 250 steps.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/sunta-hierarchical-video-prediction-with-surprise-based-chunking-327424/327424/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/sunta-hierarchical-video-prediction-with-surprise-based-chunking-327424/327424.png","ImageObject",300,407,{"name":42,"@type":43},"Eliana","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-03","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",10,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why does chunk boundary selection matter for hierarchical state-space models?","Question",{"text":63,"@type":64},"HSSM performance depends on whether boundaries align with the intrinsic temporal structure, so higher-level dynamics can be modeled without entangled transitions caused by misalignment.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What are the two main challenges when adding surprise-based chunking to HSSMs?",{"text":68,"@type":64},"Gap 1 is hierarchical collapse caused by surprise minimization during end-to-end training, and Gap 2 is the absence of surprise signals in open-loop generation because external observations are unavailable.",{"name":70,"@type":61,"acceptedAnswer":71},"How does SUNTA determine chunk boundaries during imagined rollouts?",{"text":72,"@type":64},"SUNTA uses a decoupled training strategy to preserve surprise signals and employs internal inconsistency as a top-down surprise metric to decide where chunks begin and end within imagined rollouts.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327424,1790448110,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":55,"slug":125},"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":127,"language":139,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":12,"update_tm":143,"read_time":144,"preview_page_count":4},4398048949847,"https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267","arXiv :2607 .02087v 1 [ cs .AI] 2 Jul 2026  \nSUNTA: Hierarchical Video Prediction with Surprise-based Chunking  \nTomoshi Iiyama Masahiro Suzuki Yutaka Matsuo  \nThe University of Tokyo  \n{iiyama, masa, [matsuo}@weblab.t.u-tokyo.ac.jp](matsuo}@weblab.t.u-tokyo.ac.jp)  \nAbstract  \nHierarchical state-space models (HSSMs) offer a promising approach to longhorizon prediction by segmenting sequences into temporal chunks. However, their performance hinges on how chunk boundaries are determined. While prior HSSMs typically rely on fixed-length chunking or similarity-based boundary detection, these methods often misalign with the intrinsic temporal structure of the data. We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary. Nevertheless, integrating surprise-based chunking into HSSMs introduces critical challenges, including hierarchical collapse during end-to-end training and the absence of surprise signals during open-loop prediction. To address these issues, we propose Surprisebased Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals and uses internal inconsistency as atop-down surprise metric to determine chunk boundaries within imagined rollouts.  \nExperiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines, uniquely maintaining accurate predictions over  \n250 timesteps, whereas all baselines degrade within the first 10 timesteps.  \n1 Introduction  \nPredicting future environmental states is a cornerstone of intelligence [21, 23], motivating research on world models that endow agents with internal predictive models [9, 12, 28] and substantially enhance planning and decision-making [14, 15, 17, 18, 19, 20] . A central challenge for such models is to support long-horizon prediction: since complex tasks often require foresight to anticipate distant consequences [16, 37, 42], capturing long-term dynamics is essential for practical world models. A promising route toward long-horizon prediction is hierarchical state-space models (HSSMs) [11, 25, 26, 39, 41, 50], which decompose sequences into temporal chunks and learn dynamics across multiple timescales. Crucially, performance depends on chunk boundaries: they should align with intrinsic temporal structure so that high-level dynamics are easier to model; misalignment forces the higher level to explain entangled transitions and obscures regularities even when the underlying dynamics are simple 1.  \nNevertheless, existing HSSMs typically fix chunk lengths [39, 41] or constrain lengths toward apreset value [26] . Recent work, such as VPR [50], instead infers boundaries from observational changes such as visual shifts (i.e., similarity-based chunking) . However, such cues can fail in practice: important contextual shifts in the underlying dynamics may occur without visual salience, while superficial visual changes may appear without semantic transitions.  \n1For example, predicting a sentence becomes unnecessarily difficult when it is segmented into arbitrary substrings (Iforg → otmyu → mbrel → laont → hetra → in) rather than into syntactically meaningful units (I → forgot → my → umbrella → on → the → train)  \nPreprint.  \nFigure 1: Similarity-based chunking vs. surprise-based chunking. Similarity-based chunking monitors observational changes and may over-segment superficial appearance changes or miss nonsalient semantic transitions. Surprise-based chunking cuts at peaks of prediction error, highlighting shifts in the latent dynamics.  \nWe argue that temporal abstraction should be driven by prediction errors, or surprise, within the internal world model rather than visual changes, since surprise reflects shifts in the underlying dynamics rather than superficial appearance differences (see Fig. 1): chunks should begin and end where prediction error spikes, marking moments where higher-level abs","cbCaifjzKxtbG8E0","https://ap.wps.com/l/cbCaifjzKxtbG8E0","pdf",5999205,"English","# Introduction\n## Hierarchical state-space models and the chunk-boundary problem\n## Similarity-based chunking limitations\n## Surprise-based chunking and its two integration gaps\n## Proposed solution: SUNTA","[{\"question\":\"Why does chunk boundary selection matter for hierarchical state-space models?\",\"answer\":\"HSSM performance depends on whether boundaries align with the intrinsic temporal structure, so higher-level dynamics can be modeled without entangled transitions caused by misalignment.\"},{\"question\":\"What are the two main challenges when adding surprise-based chunking to HSSMs?\",\"answer\":\"Gap 1 is hierarchical collapse caused by surprise minimization during end-to-end training, and Gap 2 is the absence of surprise signals in open-loop generation because external observations are unavailable.\"},{\"question\":\"How does SUNTA determine chunk boundaries during imagined rollouts?\",\"answer\":\"SUNTA uses a decoupled training strategy to preserve surprise signals and employs internal inconsistency as a top-down surprise metric to decide where chunks begin and end within imagined rollouts.\"}]","SUNTA - Hierarchical Video Prediction with Surprise-based Chunking | PDF",1789977171,48]