CRISP - CRISP: A Spatiotemporal Camera–Radar Backbone for Driving via Forecasting-Based World-Model Pretraining

Sep 21, 2026

Uploaded by Riley

Camera–radar (CR) fusion supports autonomous driving, yet many existing models depend on task-specific supervision, which restricts reusable representation learning. CRISP presents a spatiotemporal CR backbone pretrained with forecasting-based representation learning, using historical multi-view images and radar sweeps to predict future LiDAR point clouds and learn a unified BEV representation. LiDAR serves only as privileged supervision during pretraining, enabling deployment with camera and radar. Enhanced radar encoding, radar-aware temporal self-attention, and multimodal feature rendering with gating inject radar range and Doppler cues into BEV temporal propagation. Experiments on nuScenes show improved long-horizon point-cloud forecasting and strong transfer to 3D detection, tracking, online mapping, motion forecasting, future occupancy prediction, and planning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

I. INTRODUCTION

Sensor modalities and fusion motivation

Limitations of current supervised CR methods

Pretraining and forecasting-based world-model learning

Motivation for forecasting-based CR representation learning

Paper contribution: CRISP

CRISP: A Spatiotemporal Camera–Radar Backbone for Driving via Forecasting-Based World-Model Pretraining Jingyu Song∗1, Yi Liu∗1, and Katherine A. Skinner1 arXiv :2607 .0454 1v 1 [ cs .CV] 5 Jul 2026 Abstract—Camera-radar (CR) fusion is a practical sensing configuration for autonomous driving, but existing models are typically trained with task-specific supervision, limiting reusable representation learning. We present CRISP, a spatiotemporal CR backbone pretrained through forecasting-based representation learning. Given historical multi-view images and radar sweeps, CRISP learns a unified bird’s-eye-view (BEV) representation by predicting future LiDAR point clouds. LiDAR is used only as privileged supervision during pretraining; the deployed model requires only camera and radar. To make forecasting-based pretraining effective for CR fusion, CRISP introduces an enhanced radar encoder, radar-enhanced temporal self-attention, and multimodal feature rendering with modality innovation gating. These components inject radar range and Doppler cues into BEV temporal propagation and allow BEV tokens to selectively incorporate camera and radar evidence. Experiments on nuScenes show that CRISP improves long-horizon point cloud forecasting and transfers effectively to downstream tasks, including 3D detection, tracking, online mapping, motion forecasting, future occupancy prediction, and planning, suggesting that predictive CR pretraining is a promising path toward scalable driving representations under practical sensor configurations. The project website is [https://umfieldrobotics.github.io/CRISP](https://umfieldrobotics.github.io/CRISP). Index Terms—Autonomous Driving, Pretraining, CameraRadar Fusion, Point Cloud Prediction. I. INTRODUCTION Autonomous vehicles (AVs) rely on perception systems to understand the surrounding environment and support downstream prediction and planning modules [1]–[5] . Common sensing modalities include camera, LiDAR, and radar, each providing distinct advantages and limitations [6], [7] . Cameras offer dense semantic cues, LiDAR provides accurate 3D geometry, and radar is robust to adverse weather and lighting while also measuring motion through Doppler velocity. Since no single sensor is sufficient across all conditions, multimodal fusion has received significant interest from the research community [6], [8] . Among different sensor configurations, LiDAR-camera (LC) fusion has achieved strong performance across many perception benchmarks [1], [2], [9]–[12] . However, the cost and form factor of LiDAR can limit large-scale deployment in production vehicles [7], [12] . In contrast, camera-radar (CR) sensing offers a more scalable and lower-cost configuration that is already accessible on many vehicles with advanced driver assistance systems [7], [13] . As a result, CR fusion has attracted increasing attention in recent years, with demonstrated benefits for 3D object detection, scene segmentation, ∗denotes equal contribution 1J. Song, Y. Liu, and K. A. Skinner are with the Department of Robotics, University of Michigan, Ann Arbor, MI 48109 USA. Corresponding author e-mail: [jingyuso@umich.edu](jingyuso@umich.edu) Broad Downstream Transfer Fig. 1. Overview of the motivation of CRISP. We use low-cost camera-radar inputs for inference, while leveraging LiDAR as privileged supervision during pretraining to learn a transferable multimodal driving backbone. mapping, and occupancy prediction [14]–[18] . These advances suggest that CR fusion is a promising path toward practical and robust autonomous driving. Despite this progress, including recent CR fusion networks, many AV perception models [11], [19], [20] still rely heavily on supervised training with precise 3D task annotations. Such annotations are expensive to obtain at scale, which motivates pretraining as a more scalable way to learn effective representations for downstream driving tasks from large amounts of unlabeled or weakly label

FAQs

What problem does CRISP address in camera-radar fusion for autonomous driving?

CRISP targets the limitation that many CR fusion models rely on task-specific supervision, which reduces the ability to learn reusable representations for a range of driving tasks.

How does CRISP perform pretraining and what sensors are used at deployment?

CRISP uses historical multi-view images and radar sweeps to predict future LiDAR point clouds, learning a unified BEV representation. LiDAR is used only as privileged supervision during pretraining; the deployed model requires only camera and radar.

Which components help CRISP make forecasting-based pretraining effective for CR fusion?

CRISP introduces an enhanced radar encoder, radar-enhanced temporal self-attention, and multimodal feature rendering with modality innovation gating to inject radar range and Doppler cues and let BEV tokens selectively incorporate camera and radar evidence.

Sep 21, 2026

Uploaded by Riley

CRISP - CRISP: A Spatiotemporal Camera–Radar Backbone for Driving via Forecasting-Based World-Model Pretraining_Page 1

Open in WPS to view all 17 pages