SPEAR: A Simulator for Photorealistic Embodied AI Research

Jul 16, 2026

Upload by Skyler

Interactive photorealistic simulators enable training embodied agents and producing synthetic visual data, yet existing solutions are hindered by limited generality, weak programmability, and slow rendering-to-Python pipelines. SPEAR addresses these issues with a modular Unreal Engine plugin architecture controlled by a Python library. It exposes 14K+ UE functions to Python, renders 1920×1080 beauty images into NumPy at 73 FPS, and provides ground-truth modalities including non-diffuse intrinsic decomposition, material IDs, and PBR parameters. It also supports deterministic high-level graph execution within a single UE frame and showcases multi-agent control, city-scale rendering, procedural editing, multi-view face synthesis, MuJoCo co-simulation, and natural-language scene editing via an AI coding assistant.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Abstract

Introduction

Overview of SPEAR capabilities

arXiv :2607 .0670 1v 1 [ cs .CV] 7 Jul 2026 SPEAR: A Simulator for Photorealistic Embodied AI Research Mike Roberts 1 ,2 , Renhan Wang3 , Rushikesh Zawar4 , Rachith Dey-Prakash2 , Quentin Leboutet2 , Stephan R. Richter2 , Matthias Müller2 , German Ros5 , Rui Tang3 , Stefan Leutenegger6 ,7 , Yannick Hold-Geoffroy 1 , Kalyan Sunkavalli 1 , and Vladlen Koltun2 1 Adobe Research 2 Intel Labs 3 Manycore Tech Inc 4 Adobe 5 NVIDIA 6 ETH Zurich 7 Imperial College London [https://github.com/spear-sim/spear](https://github.com/spear-sim/spear) Abstract. Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920×1080 photorealistic beauty images directly into a user’s NumPy array at 73 frames per second – an order of magnitude faster than existing UE plugins – while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters) . Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE’s procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant. 1 Introduction Interactive simulators have become a critical layer of scientific infrastructure, participating in major breakthroughs in reinforcement learning (e.g., worldchampion performance in competitive e-sports games [56, 75 , 76 , 82] and drone racing [30–32, 43 , 44]), sensorimotor control [60, 61 , 88], autonomous driving [18, 2 M. Roberts et al. Fig. 1: SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions, representing an order-of-magnitude increase in programmable functionality over existing simulators. We demonstrate the flexibility of SPEAR by using it to control 6 distinct embodied agents (each with a different action space) across several Epic Games sample projects: a person and a car from CitySample (top); a flying robot from StackOBot (bottom far left); multiple agents in a resource collecting game called CropoutSample (bottom center left); as well asa person with parkour skills and a quadruped robot from GameAnimationSample (bottom right) . 53], dexterous manipulation [55], and quadruped locomotion [24, 39 , 50] . Additionally, photorealistic synthetic datasets (e.g., [62]) are now being used to train and evaluate large-scale foundation models (e.g., [12, 20 , 51 , 69 , 73 , 85]), as well as state-of-the-art methods for 3D and 4D reconstruction [49, 77 , 78], segmentation [35, 59], depth estimation [33], and controllable video synthesis [13] . Motivated by the parallel goals

FAQs

What problems does SPEAR aim to solve compared with existing Unreal Engine–based photorealistic simulators?

SPEAR targets limited generality, insufficient programmability, and rendering/communication inefficiencies between Unreal Engine and Python, which slow down high-resolution data return.

How does SPEAR connect to and control Unreal Engine applications?

SPEAR is a Python library that interfaces with Unreal Engine through a modular plugin architecture, exposing 14K+ unique UE functions to Python for programmatic control.

What performance and data capabilities does SPEAR provide for image generation?

A single SPEAR instance renders 1920×1080 photorealistic images directly into a user’s NumPy array at 73 FPS and additionally outputs ground-truth image modalities such as non-diffuse intrinsic decomposition, material IDs, and physically based shading parameters.

Jul 16, 2026

Upload by Skyler