Worldscape-MoE - A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control

Sep 21, 2026

Uploaded by Skyler

Worldscape-MoE proposes a unified video generation world model for embodied intelligence that supports scalable heterogeneous action control. It addresses a structural bottleneck: existing world models are fragmented by control interface type, such as camera trajectories, robot actions, or hand-joint signals, which limits extensible learning from mixed supervision. The method uses modality-aware control injection, shared and control-specific experts, and progressive MoE tuning to extend to new action modalities. Experiments on multiple control settings validate improved control capability and robust out-of-distribution generalization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

AccessFree to download

Introduction

World models and controllable embodied intelligence

Fragmentation by control modality

Unified framework based on factorization

arXiv :2607 .03964v 1 [ cs .RO] 4 Jul 2026 July 5, 2026 Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control Jianjie Fang 1,* , Yongyan Xu 1,* , Ziyou Wang 1,* , Chen Gao 1,*,†, Yuchao Huang 1 , Zhaolu Wang 1 , Rongze Tang 1 , Mingyuan Jia 1 , Baining Zhao 1 , Weichen Zhang 1 , Xin Zhang2 , Haisheng Su2 , Yu Shang 1 , Wei Wu2 , Xinlei Chen 1 , Yong Li 1,† 1Tsinghua University 2 Manifold AI * Equal contribution. †Corresponding authors. Abstract. World models are rapidly becoming a core infrastructure for embodied intelligence and interactive agents: they provide controllable simulators in which agents can perceive, act, forecast, and acquire scalable experience. Yet current video generation world models are still organized around isolated control interfaces, such as camera trajectories, robot actions, or hand-joint signals. This fragmentation is increasingly a scaling bottleneck. The central challenge is not the absence of controllable generators, but the lack of a unified and extensible learning framework that can absorb heterogeneous action supervision while preserving a shared model of world dynamics. In this work, we introduce Worldscape-MoE, a Mixture-of-Experts world model built on Diffusion Transformers for scalable heterogeneous action control. Our key observation is that different controls specify different interfaces to the same underlying world: although their representations differ, they constrain shared physical regularities, scene dynamics, and interaction semantics. Worldscape-MoE operationalizes this observation through modality-aware control injection, shared and control-specific experts, and a progressive MoE tuning strategy that supports continual extension to new action modalities. Experiments across locomotion, robotic manipulation, and egocentric hand control show that heterogeneous supervision improves rather than interferes with individual control capabilities. Worldscape-MoE achieves strong resultson WorldArena, improves locomotion and hand-control metrics, exhibits robust out-of-distribution generalization, and demonstrates scaling behavior as additional control data and experts are integrated. Date: July 5, 2026 Webpage: [https://worldscape-moe.com](https://worldscape-moe.com) 1 Introduction World models are moving from a research prototype toward a general infrastructure for embodied intelligence and interactive agents. Built on recent progress in high-quality video generation [1–3], they can simulate how visual scenes evolve under actions, support interactive exploration, and provide scalable experience beyond what can be collected directly in the physical world [4, 5] . This role is especially important for agents that must reason through the consequences of actions before execution: a world model is no longer merely a video generator, but a controllable substrate for perception, planning, evaluation, and policy learning. Despite this progress, current world models remain largely fragmented by the form of control they accept. Interactive environment models, such as Genie [6], the Matrix-Game series [7–9], and HYWorld [10, 11], primarily focus on camera or navigation-like controls. Embodied world models, such Prompt: On a tropical beach, a weathered wooden pier extends into turquoise shallows. Green palm trees sway in the warm breeze, fronds catching golden sunlight. Rustic thatched huts rest on stilts, with canoes moored beside the dock. Pan left to view the scenery. Action Map Worlds cape-MoE Prompt: A red placemat on wooden table with blue plastic cutlery arranged in front. In the background, a red cup and saucer with red plastic cutlery, and a blue cup and saucer are placed against a white wall. Place the blue cup on its saucer. Prompt: In a fixed robotic workspace, generate a rigid, physically consistent embodied robotic arm. The arm maintains high stability with no deformation and picks up Sprite placed in the middle

FAQs

What problem does Worldscape-MoE address in current world models?

It targets fragmentation caused by isolated control interfaces, which prevents a scalable and unified learning framework for heterogeneous action supervision.

How does Worldscape-MoE enable unified heterogeneous action control?

It introduces modality-aware control injection, shared plus control-specific experts, and a progressive MoE tuning strategy for continual extension to new action modalities.

What control modalities are demonstrated in the experiments?

The framework supports locomotion, robotic manipulation, and egocentric hand control, showing that heterogeneous supervision can improve individual control capabilities.

Sep 21, 2026

Uploaded by Skyler