[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85655-en":3,"doc-seo-85655-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85655,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","SUNTA: Hierarchical Video Prediction with Surprise-based Chunking","Hierarchical state-space models support long-horizon prediction by splitting sequences into temporal chunks, but performance depends on how chunk boundaries are chosen. Prior methods using fixed or similarity-based chunking often misalign with the data’s intrinsic temporal structure. Chunking should instead be driven by prediction errors that indicate when long-range context becomes necessary. Integrating surprise into HSSMs introduces hierarchical collapse during end-to-end training and missing surprise signals in open-loop generation. SUNTA addresses both via decoupled training that preserves surprise signals and a top-down internal inconsistency metric to set chunk boundaries during imagined rollouts.","arXiv :2607 .02087v2 [ cs .AI] 10 Jul 2026  \nSUNTA: Hierarchical Video Prediction with Surprise-based Chunking  \nTomoshi Iiyama Masahiro Suzuki Yutaka Matsuo  \nThe University of Tokyo  \n{iiyama, masa, [matsuo}@weblab.t.u-tokyo.ac.jp](matsuo}@weblab.t.u-tokyo.ac.jp)  \nAbstract  \nHierarchical state-space models (HSSMs) offer a promising approach to longhorizon prediction by segmenting sequences into temporal chunks. However, their performance hinges on how chunk boundaries are determined. While prior HSSMs typically rely on fixed-length chunking or similarity-based boundary detection, these methods often misalign with the intrinsic temporal structure of the data. We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary. Nevertheless, integrating surprise-based chunking into HSSMs introduces critical challenges, including hierarchical collapse during end-to-end training and the absence of surprise signals during open-loop prediction. To address these issues, we propose Surprisebased Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals and uses internal inconsistency as atop-down surprise metric to determine chunk boundaries within imagined rollouts.  \nExperiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines, uniquely maintaining accurate predictions over  \n250 timesteps, whereas all baselines degrade within the first 10 timesteps.  \n1 Introduction  \nPredicting future environmental states is a cornerstone of intelligence [21, 23], motivating research on world models that endow agents with internal predictive models [9, 12, 28] and substantially enhance planning and decision-making [14, 15, 17, 18, 19, 20] . A central challenge for such models is to support long-horizon prediction: since complex tasks often require foresight to anticipate distant consequences [16, 37, 42], capturing long-term dynamics is essential for practical world models. A promising route toward long-horizon prediction is hierarchical state-space models (HSSMs) [11, 25, 26, 39, 41, 50], which decompose sequences into temporal chunks and learn dynamics across multiple timescales. Crucially, performance depends on chunk boundaries: they should align with intrinsic temporal structure so that high-level dynamics are easier to model; misalignment forces the higher level to explain entangled transitions and obscures regularities even when the underlying dynamics are simple 1.  \nNevertheless, existing HSSMs typically fix chunk lengths [39, 41] or constrain lengths toward apreset value [26] . Recent work, such as VPR [50], instead infers boundaries from observational changes such as visual shifts (i.e., similarity-based chunking) . However, such cues can fail in practice: important contextual shifts in the underlying dynamics may occur without visual salience, while superficial visual changes may appear without semantic transitions.  \n1For example, predicting a sentence becomes unnecessarily difficult when it is segmented into arbitrary substrings (Iforg → otmyu → mbrel → laont → hetra → in) rather than into syntactically meaningful units (I → forgot → my → umbrella → on → the → train)  \nPreprint.  \nFigure 1: Similarity-based chunking vs. surprise-based chunking. Similarity-based chunking monitors observational changes and may over-segment superficial appearance changes or miss nonsalient semantic transitions. Surprise-based chunking cuts at peaks of prediction error, highlighting shifts in the latent dynamics.  \nWe argue that temporal abstraction should be driven by prediction errors, or surprise, within the internal world model rather than visual changes, since surprise reflects shifts in the underlying dynamics rather than superficial appearance differences (see Fig. 1): chunks should begin and end where prediction error spikes, marking moments where higher-level abs","cbCaijouHTJfqyBx","https://ap.wps.com/l/cbCaijouHTJfqyBx","pdf",5877299,1,19,"English","en",105,"# Introduction\n## Long-horizon prediction and hierarchical state-space models\n## Limitations of fixed and similarity-based chunking\n## Surprise-based chunking and its integration challenges\n## SUNTA approach and contributions","[{\"question\":\"Why do chunk boundaries matter for hierarchical state-space models in long-horizon prediction?\",\"answer\":\"Chunk boundaries must align with the data’s intrinsic temporal structure so higher-level dynamics are modeled more easily. Misaligned boundaries entangle transitions and obscure regularities, degrading predictive performance.\"},{\"question\":\"What problem arises when using surprise minimization to determine chunk boundaries in a naive HSSM setup?\",\"answer\":\"As the high-level model improves via top-down conditioning, low-level prediction errors diminish. This erases the surprise signals that originally defined the chunks, causing the hierarchy to repeatedly collapse and re-emerge during training.\"},{\"question\":\"How does SUNTA enable surprise-based chunking during open-loop video prediction when external observations are unavailable?\",\"answer\":\"SUNTA replaces observation-based surprise with a top-down surprise signal derived from the high-level representation. It uses internal inconsistency during imagined rollouts to detect when to terminate chunks and trigger higher-level abstraction.\"}]",1784205384,48,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"sunta-hierarchical-video-prediction-with-surprise-based-chunking","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/sunta-hierarchical-video-prediction-with-surprise-based-chunking/85655/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do chunk boundaries matter for hierarchical state-space models in long-horizon prediction?","Question",{"text":75,"@type":76},"Chunk boundaries must align with the data’s intrinsic temporal structure so higher-level dynamics are modeled more easily. Misaligned boundaries entangle transitions and obscure regularities, degrading predictive performance.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What problem arises when using surprise minimization to determine chunk boundaries in a naive HSSM setup?",{"text":80,"@type":76},"As the high-level model improves via top-down conditioning, low-level prediction errors diminish. This erases the surprise signals that originally defined the chunks, causing the hierarchy to repeatedly collapse and re-emerge during training.",{"name":82,"@type":73,"acceptedAnswer":83},"How does SUNTA enable surprise-based chunking during open-loop video prediction when external observations are unavailable?",{"text":84,"@type":76},"SUNTA replaces observation-based surprise with a top-down surprise signal derived from the high-level representation. It uses internal inconsistency during imagined rollouts to detect when to terminate chunks and trigger higher-level abstraction.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},"General","general"]