Synthetic-to-Real Translation for Class-Agnostic Motion Prediction

Jul 16, 2026

Motion understanding is essential for safety and robustness in autonomous driving, yet real-world motion labels are costly to obtain. This work studies synthetic-to-real translation for motion prediction (SRMP) and addresses the instability of naive motion regression under synthetic-to-real domain shift. The proposed SRMP integrates objectness-aware motion prediction to learn domain-invariant features and objectness-aided motion enhancement to refine pseudo motion labels by filtering motion noise. A physically based pipeline generates Motion4D, the first synthetic 4D LiDAR dataset for SRMP, enabling improved performance on real scenes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Introduction

Synthetic-to-Real Motion Prediction (SRMP)

Objectness-aware motion prediction

Objectness-aided motion enhancement

Motion4D dataset generation

Physically-based pipeline

Experimental results and expected release

Synthetic-to-Real Translation for Class-Agnostic Motion Prediction Yizheng Wu* , Hongwei Fan* , Kewei Wang, Ruibo Li, Xingyi Li, Xiao Song, Zhe Wang, Chenjing Ding, Dongliang Wang, Zhiguo Cao, Guosheng Lin arXiv :2607 .063 19v 1 [ cs .CV] 7 Jul 2026 Abstract—Motion understanding is critical for ensuring safety and robustness in autonomous driving systems, driving increasing interest in motion prediction. A key challenge in this domain is the high cost associated with acquiring real-world motion labels. It is therefore ideal if we could transfer motion knowledge from synthetic data to real data. In this context, we explore the potential of synthetic-to-real translation for motion prediction (SRMP). However, the most used naive motion regression methods are notably sensitive to the synthetic-to-real domain shift, resulting in unreliable knowledge translation. To address this, we propose a novel approach integrating a motion knowledge translation framework with two key components: (1) objectness-aware motion prediction, which explicitly models the joint distribution of motion patterns and objectness priors to improve domaininvariant feature learning, and (2) objectness-aided motion enhancement, a motion label refinement mechanism that leverages learned objectness priors to filter motion noise. Furthermore, we present a physically-based pipeline for generating Motion4D, the first synthetic 4D LiDAR dataset tailored for SRMP research, addressing the lack of synthetic motion datasets. Experimental results demonstrate that our approach effectively bridges the domain gaps and yields superior performance on real scenes. Code and dataset will be made publicly available. I. INTRODUCTION Motion prediction [1]–[5] aims to predict future motion displacements based on past observations from multimodal sensors, playing a crucial role in path planning and navigation for autonomous driving systems. To mitigate failures when dealing with unknown classes, many recent motion prediction approaches [4], [6], [7] adopt a class-agnostic paradigm to directly regress cell-wise motions. These approaches transform the point clouds into bird’s eye view (BEV) maps and represent motions as 2D displacements along the ground plane, achieving convincing performance. However, it is extremely challenging and costly to obtain motion labels on real data due to the sparse and non-uniform nature of point clouds [8], [9] . Yizheng Wu, Kewei Wang, and Xingyi Li are with the Key Laboratory of Image Processing and Intelligent Control, Ministry of Education, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China, and also with Nanyang Technological University, Singapore 639798, Singapore. Hongwei Fan, Xiao Song, Zhe Wang, Chenjing Ding, and Dongliang Wang are with the SenseAuto, Beijing 100080, China. Zhiguo Cao is with the Key Laboratory of Image Processing and Intelligent Control, Ministry of Education, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China. Ruibo Li, Guosheng Lin is with Nanyang Technological University, Singapore 639798, Singapore. Corresponding author: Guosheng Lin. Email: [gslin@ntu.edu.sg](gslin@ntu.edu.sg)[ ](gslin@ntu.edu.sg)Yizheng Wu and Hongwei Fan contributed equally. This paper has supplementary downloadable material available at [http://ieeexplore.ieee.org](http://ieeexplore.ieee.org)., provided by the author. The material includes a video comparing our dataset with other real-world datasets. This material is 142MB in size. Real-World Real-World Target Data T S EMA Noisy Pseudo Labels Initial Pseudo Labels Improved Pseudo Labels (a) Baseline. (b) Ours (SR-Motion) . Ground Truth Initial Pseudo Labels Improved Pseudo Labels (c) Visualization of motions. Fig. 1. Problem identification and resolution for SRMP. (a) The naive regression of motions results in noisy pseudo labe

FAQs

为什么合成数据到真实数据的知识迁移在运动预测中很难？

因为合成与真实数据在运动分布、遮挡模式和目标组成上存在显著域差。仅用合成数据训练会在真实场景中出现明显性能退化。

作者提出的 SRMP 方案包含哪些关键组成？

方案整合两部分：objectness-aware motion prediction，用于显式建模运动模式与目标性先验以学习域不变特征；objectness-aided motion enhancement，用于利用学习到的目标性先验细化并过滤噪声伪标签。

Motion4D 数据集在这项研究中扮演什么角色？

Motion4D 通过物理驱动流程生成首个面向 SRMP 研究的合成 4D LiDAR 数据集，直接提供准确的合成运动标签，从而支持合成到真实的知识学习与迁移。

Jul 16, 2026

Upload by River Wang