[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84494-en":3,"doc-seo-84494-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84494,549758146520,"Patrick","https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470",8,"Research & Report","Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models","Modern interactive video world models deliver high visual fidelity but fall short in fine-grained multi-entity control and cross-entity, cross-world generalization. The core limitation is the action interface: standard protocols bind action semantics to specific entities or engines during design. Incantation introduces natural language conditioning at per-latent-frame granularity (0.25s) to enable simultaneous control and concept-level cross-entity transfer beyond fixed rendering pipelines, paired with a bidirectional video backbone and real-time long-horizon streaming.","arXiv :2605 . 18601v2 [ cs .CV] 12 Jul 2026  \nIncantation: Natural Language as the Action Interface for Multi-Entity Video World Models  \nShangwen Zhu 1∗ Qianyu Peng8∗ Zhao Pu 1∗ Zhilei Shu3 Xiangrui Ke6 Zhaohu Xing7 Zizhao Tong4 Zeqing Wang5 Xinyu Cui9 Zian Zheng6 Huangji Wang 1 Jian Zhao9 Yeying Jin5 Fan Cheng 1† Ruili Feng2 ,6†  \n1 SJTU 2NVIDIA 3Matrix Team 4PKU 5NUS  \n6UWaterloo 7HKUST 8HKU 9ZGCA  \nGithub: [https://github.com/zhushangwen/Incantation](https://github.com/zhushangwen/Incantation)[ ](https://github.com/zhushangwen/Incantation)Project Page: [https://matrixteam-ai.github.io/pages/Incanation](https://matrixteam-ai.github.io/pages/Incanation)[ ](https://matrixteam-ai.github.io/pages/Incanation)Dataset Preview: [https://huggingface.co/datasets/zhush/incantation-elden-ring-scenes](https://huggingface.co/datasets/zhush/incantation-elden-ring-scenes)  \nFigure 1: Demonstrations of Incantation’s cross-entity action transfer and multi-entity control in the game Elden Ring. (i) Two bosses, Margit and Crucible Knight, each possessing characterexclusive moves, are conditioned via natural language to perform each other’s actions, each executed by both its native character and the other: Light Blade Attack (Margit-exclusive, green rows) and Tail of the Crucible (Crucible Knight-exclusive, blue rows), demonstrating Incantation’s cross-entity generalization. (ii) Incantation simultaneously controls three entities (two bosses and one player) each via a distinct natural-language prompt (orange rows), trained on two-entity scenarios only.  \nAbstract  \nModern interactive video world models have achieved impressive visual fidelity, yet lack fine-grained multi-entity control and cross-entity, cross-world generalization.  \nWe trace this gap to the action interface: standard control protocols (e.g. animation IDs, device inputs, scene-level captions) bind action semantics to specific entities or engines at design time. We propose natural language as the interface to unlock expressiveness that no prior interface can achieve, and we present Incantation, the first interactive video world model with per-latent-frame (0 .25s) natural-language conditioning that supports simultaneous multi-entity control and concept-level  \n*Equal contribution.  \n†Corresponding authors.  \nPreprint.  \ncross-entity transfer beyond any fixed rendering pipeline. We pair a pretrained bidirectional video backbone with frame-local text cross-attention, and enable real-time long-horizon streaming through ODE-initialized Self-Forcing distillation with a RoPE-decoupled sliding KV-cache. We surpass the Action-Index baseline on cross-entity transfer (89% vs. 43%) and out-of-vocabulary prompts (90% vs.  \n0%), and our 2-step student sustains 19.7FPS at 480p with stable FVD over 2-hour rollouts. We further apply the same architecture and training recipe to The King of Fighters, changing only the per-entity action vocabulary slots. We have released a preview subset of the Incantation dataset, containing manually collected Elden Ring player-boss combat clips with structured action-oriented metadata. Largerscale Elden Ring and KOF data will be released with the full project.  \n1 Introduction  \nModern video diffusion models [21, 6, 39] have driven a growing line of controllable interactive world models [8, 28, 15, 2, 13, 48, 20, 40, 11] to near-cinematic fidelity, yet every such system inherits a structural limitation from the rendering pipelines it replaces: actions are bound to engine-internal animation namespaces or device-level inputs, locking action semantics to a specific entity and engine. This entity-and-engine binding forces a separate action vocabulary to be designed for every entity in every world, making cross-entity and cross-world generalization an engineering burden rather than a modeling choice. We argue that this is not an intrinsic property of multi-entity interactive video, but a property of the action interface: the protocol through which a user specifies wha","cbCaivQEPvjFDqDZ","https://ap.wps.com/l/cbCaivQEPvjFDqDZ","pdf",8906101,1,31,"English","en",105,"# Abstract\n## Action-interface limitation\n## Incantation approach\n## Model components and training\n## Experimental results and dataset preview","[{\"question\":\"What limitation do interactive video world models face in multi-entity settings?\",\"answer\":\"They lack fine-grained multi-entity control and cross-entity/cross-world generalization because action semantics are tied to entity- or engine-specific protocols designed ahead of time.\"},{\"question\":\"How does Incantation use natural language to control multiple entities?\",\"answer\":\"It conditions each latent frame with per-latent-frame natural-language prompts (0.25s), enabling independent and simultaneous multi-entity control under a shared viewpoint.\"},{\"question\":\"What performance improvements does the approach report?\",\"answer\":\"Incantation surpasses an Action-Index baseline on cross-entity transfer (89% vs. 43%) and on out-of-vocabulary prompts (90% vs. 0%), while sustaining real-time long-horizon rollouts with stable FVD.\"}]",1784196045,78,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"incantation-natural-language-as-the-action-interface-for-multi-entity-video-world-models","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/incantation-natural-language-as-the-action-interface-for-multi-entity-video-world-models/84494/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What limitation do interactive video world models face in multi-entity settings?","Question",{"text":75,"@type":76},"They lack fine-grained multi-entity control and cross-entity/cross-world generalization because action semantics are tied to entity- or engine-specific protocols designed ahead of time.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does Incantation use natural language to control multiple entities?",{"text":80,"@type":76},"It conditions each latent frame with per-latent-frame natural-language prompts (0.25s), enabling independent and simultaneous multi-entity control under a shared viewpoint.",{"name":82,"@type":73,"acceptedAnswer":83},"What performance improvements does the approach report?",{"text":84,"@type":76},"Incantation surpasses an Action-Index baseline on cross-entity transfer (89% vs. 43%) and on out-of-vocabulary prompts (90% vs. 0%), while sustaining real-time long-horizon rollouts with stable FVD.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]