[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81534-en":3,"doc-seo-81534-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81534,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",6,"Technology","M4V: Multimodal Mamba for Efficient Text-to-Video Generation","Text-to-video generation enriches content creation and can act as a foundation for world simulation, but modeling the full spatiotemporal space is computationally heavy. Transformer diffusion models face quadratic sequence costs, while Mamba’s efficient linear-time design is hindered by limited multimodal and spatiotemporal capability. This work proposes M4V, introducing a MultiModal Diffusion Mamba block that unifies multimodal fusion and spatial–temporal modeling. MM-DiM cuts FLOPs by 45% at 768×1280 resolution versus attention baselines and supports effective training using public datasets.","M4V: Multimodal Mamba for Efficient Text-to-Video Generation  \nJiancheng Huang 1∗ Gengwei Zhang2∗ Zequn Jie 1† Siyu Jiao3  \nYinlong Qian 1 Ling Chen2 Yunchao Wei3† Lin Ma 1  \n1 Meituan 2 University of Technology Sydney 3 Beijing Jiaotong University  \narXiv :2506 . 10915v2 [ cs .CV] 10 Jul 2026  \nAbstract  \nText-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, particularly when employing Transformers, which incur quadratic complexity in sequence processing and thus limit practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. Nevertheless, its plain design limits its direct applicability to multimodal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a multimodal Mamba framework for efficient textto-video generation. Specifically, a MultiModal diffusion Mamba (MM-DiM) block is designed within the framework to enable seamless integration of multimodal information and spatiotemporal modeling. In detail, we introduce a novel multimodal token re-composition design, which employs abidirectional scheme for multimodal information integration through simple token arrangement, along with visual registers to enhance spatial–temporal consistency. As a result, the MM-DiM blocks in M4V reduce FLOPs by 45% compared with the attention-based alternative when generating videos at 768×1280 resolution. Additionally, several training strategies are explored in this work to provide abetter understanding of training text-to-video models using only publicly available datasets. Extensive experiments on text-to-video benchmarks demonstrate M4V’s ability to produce high-quality videos while significantly lowering computational costs. Project page is available at [https:](https:)//[huangjch526.github.io/M4V_project/](huangjch526.github.io/M4V_project/).  \n1. Introduction  \nText-to-video (T2V) generation, which aims at creating video content from natural language instructions, is recognized as one of the most challenging tasks in genera-  \n∗ Equal Contribution. † Corresponding authors.  \nFigure 1 . Comparison of FLOPS between full attention baseline and ours.  \ntive AI. This area has recently received significant attention following the impressive results showcased by OpenAI’s Sora [4] . Notably, Transformer-based diffusion models, such as DiT [38], have been identified as a key factor in achieving Sora’s high-quality video synthesis. Despite their potential effectiveness, Transformer-based models suffer from high computational costs due to their quadratic complexity, making the already computationally demanding task even more resource-intensive.  \nRecently, a novel architecture called Mamba [14] has demonstrated the potential to match or even surpass Transformers in language modeling tasks. Building on the success of state-space models (SSMs) [15], Mamba variants [8] enhance the long-range modeling capacity of SSMs while maintaining the linear-time complexity in sequence processing. This positions Mamba as a promising alternative to Transformers.  \nHowever, unlike Transformers [48], which have driven remarkable advancements in generation tasks across both natural language processing and computer vision, Mamba remains largely unexplored in multimodal generative tasks. The limitation arises primarily from the following aspects:  \n(1) Mamba is inherently designed for processing unidirectional 1D sequences, whereas high-resolution image and video generation require sophisticated spatial and temporal modeling capabilities; (2) the lack of design for multimodal interactions, resulting in its limited exploration in text-conditioned visual generation tasks.  \nIn this paper, we address these two limitations by proposing a unified design that leverages Mamba for generati","cbCaivRwTlvdOQk4","https://ap.wps.com/l/cbCaivRwTlvdOQk4","pdf",11635227,1,20,"English","en",105,"# Introduction\n# Related Work","[{\"question\":\"What problem does M4V address in text-to-video generation?\",\"answer\":\"M4V targets the high computational cost of transformer-based video diffusion and Mamba’s inability to directly handle multimodal spatiotemporal modeling for text-conditioned video generation.\"},{\"question\":\"How does M4V improve efficiency compared with attention-based alternatives?\",\"answer\":\"M4V uses the MultiModal Diffusion Mamba (MM-DiM) block to reduce computation, reporting 45% lower FLOPs when generating videos at 768×1280 resolution.\"},{\"question\":\"What mechanisms enable multimodal and spatiotemporal modeling in M4V?\",\"answer\":\"M4V decouples information flow into 2D spatial scans and 1D temporal processing, and applies a MultiModal Token Re-Composition strategy before the SSMs so text and visual tokens can integrate global information via SSM hidden states.\"}]",1784174091,50,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"m4v-multimodal-mamba-for-efficient-text-to-video-generation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/m4v-multimodal-mamba-for-efficient-text-to-video-generation/81534/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does M4V address in text-to-video generation?","Question",{"text":74,"@type":75},"M4V targets the high computational cost of transformer-based video diffusion and Mamba’s inability to directly handle multimodal spatiotemporal modeling for text-conditioned video generation.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does M4V improve efficiency compared with attention-based alternatives?",{"text":79,"@type":75},"M4V uses the MultiModal Diffusion Mamba (MM-DiM) block to reduce computation, reporting 45% lower FLOPs when generating videos at 768×1280 resolution.",{"name":81,"@type":72,"acceptedAnswer":82},"What mechanisms enable multimodal and spatiotemporal modeling in M4V?",{"text":83,"@type":75},"M4V decouples information flow into 2D spatial scans and 1D temporal processing, and applies a MultiModal Token Re-Composition strategy before the SSMs so text and visual tokens can integrate global information via SSM hidden states.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,111,116,121,125,128,132],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":110},"technology",{"id":112,"doc_module":4,"doc_module_name":45,"category_name":113,"show_sort_weight":114,"slug":115},7,"Healthcare",40,"healthcare",{"id":117,"doc_module":4,"doc_module_name":45,"category_name":118,"show_sort_weight":119,"slug":120},8,"Research & Report",30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":21,"slug":124},9,"Religion & Spirituality","religion-spirituality",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":126,"show_sort_weight":21,"slug":127},"World Cup","world-cup",{"id":129,"doc_module":4,"doc_module_name":45,"category_name":130,"show_sort_weight":129,"slug":131},10,"Lifestyle","lifestyle",{"id":133,"doc_module":4,"doc_module_name":45,"category_name":134,"show_sort_weight":105,"slug":135},19,"General","general"]