Is Energy Guidance All You Need Training-Free Norm Injection for Driving World Models

Jul 16, 2026

Driving world models based on large video-diffusion backbones can synthesize realistic scenes but remain difficult to control for safety-critical behavior. Enforcing traffic norms often requires retraining the backbone or conditioning on hand-built layouts during training, which prevents adding or re-prioritizing rules at sampling time. This work tests whether controllability needs training by using a rectified-flow driving world model steered at sampling via differentiable energy functions encoding driving norms, showing ego-trajectory braking control without diffusion retraining while revealing limits in end-to-end video matching due to cross-stream coupling.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Hypothesis and training-free energy guidance

Trajectory-vs-video steering question

Contributions

Related Work

Training-free diffusion guidance

Guidance for flow matching

Controllable video diffusion

Robotic world models for driving

Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models Xiyan Su, Frank Diermeyer, Markus Lienkamp Institute of Automotive Technology, Technical University of Munich, Munich Germany arXiv :2607 . 10781v1 [ cs .CV] 12 Jul 2026 Abstract—Driving world models built on large video-diffusion backbones generate realistic scenes but are hard to control: enforcing a traffic norm typically means retraining the backbone or conditioning it on hand-built layouts. We ask whether controllability requires training at all. Our experiment shows that a rectified-flow driving world model, which jointly generates future video and a planned ego trajectory, can have its planned trajectory steered entirely at sampling time by differentiable energy functions that encode driving norms, without knowledge-specific retraining of the diffusion backbone. Concretely, we demonstrate that a world model built on Open-Sora 2.0 MM-DiT backbone can be steered to brake at a counterfactual target by injecting energy guidance at sampling time. However, we find that the generated video does not yet follow the steered trajectory through the backbone’s joint self-attention and identify the cross-stream coupling as a crucial requirement for end-to-end-controllable rollouts. I. INTRODUCTION Video-diffusion driving world models generate realistic sensor streams and have become increasingly popular for simulation and planning [1, 2, 3] . Their central practical weakness is controllability: a model that produces a beautiful, crisp scene in which the ego vehicle drifts across a solid line, exceeds the speed limit, or collides with another agent is not useful for safety-critical use. However, state-of-theart mitigation strategies remain costly to scale. One either fine-tunes the backbone whenever a new constraint must be complied with, or one conditions the model on hand-constructed layouts baked in during training, such as maps and agent bounding boxes. Both require high-quality data and couple new driving knowledge to a training run. Neither lets a deployed model add or re-prioritize a rule during sampling time. Hypothesis: We ask whether controllability requires training at all. Training-free guidance has shown that a frozen diffusion prior can be steered toward samples that satisfy an external objective by perturbing the sampling trajectory with the gradient of a differentiable criterion [4, 5, 6, 7] . We bring this view to world models for autonomous systems: as an application example, we take driving norms as energy functions evaluated on the generated ego trajectory, and inject them only at sampling time into a frozen rectified-flow backbone. The backbone is trained completely without knowledge-specific conditioning (see Fig. 1) . Does video match trajectory? A driving world model is valuable precisely because it generates the video, not just a Fig. 1. Overview. A frozen video world model (Open-Sora 2.0 MMDiT + LoRA) jointly denoises future video and an ego trajectory conditioned only on observed past without agent or map at training. Driving norms enter at sampling as energies on the clean ego estimate ˆx0 and are applied as a velocity correction to the ego stream. We study the question of whether the generated video is also steered through the backbone’s joint self-attention. trajectory. Therefore, the question we center on is whether steering the ego trajectory with an energy term also guides the video to match through the backbone’s joint denoising. We find that the trajectory is steered readily, but the video does not yet follow. We trace this back to the model architecture and identify the cross-stream coupling as the main cause. Contributions: Our contributions are three-fold: • We frame driving-norm compliance as training-free energy guidance on a frozen rectified-flow world model that jointly generates future video and a planned ego trajectory, and we show that all the knowledge can be added only at the s

FAQs

Why is controllability a key problem for driving world models?

Models that generate realistic video can still fail safety-critical requirements like lane keeping, speed limits, or collision avoidance. Scaling existing mitigation is costly because new constraints require training-time changes.

How does the proposed method add driving norms without retraining the diffusion backbone?

It injects driving norms at sampling time as differentiable energy functions evaluated on the planned ego trajectory. The energy guidance steers the planned trajectory using velocity correction derived from the energy.

What limitation is found regarding steering the generated video to match the guided trajectory?

Although the ego trajectory is steerable, the generated video does not follow the steered trajectory through the backbone’s joint self-attention. Cross-stream coupling is identified as a crucial requirement for end-to-end controllable rollouts.

Jul 16, 2026

Upload by River Wang