Talk2Traffic - Interactive and Editable Traffic Scenario Generation for Autonomous Driving with Multimodal Large Language Model

Oct 3, 2026

Autonomous vehicles (AVs) need safety-critical testing across diverse traffic scenarios, yet real-world data collection is costly and slow. Simulation offers cheaper alternatives, but prior generators lack intuitive, instruction-guided editing of produced scenes. This paper introduces Talk2Traffic, which uses multimodal large language models to generate scenarios from text, speech, and sketches, converts structured outputs into executable Scenic code via retrieval-augmented generation, and supports iterative refinement using natural-language feedback. Experiments show improved scenario generation quality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Motivation: Safety and Scenario Coverage

Simulation-Based Generation Limitations

Multimodal LLM Opportunities

Proposed Framework: Talk2Traffic

This CVPR Workshop paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. Talk2Traffic: Interactive and Editable Traffic Scenario Generation for Autonomous Driving with Multimodal Large Language Model Zihao Sheng 1 Zilin Huang 1 Yansong Qu2 Yue Leng3 Sikai Chen 1 * 1University of Wisconsin-Madison 2Purdue University 3 Google [https://zihaosheng.github.io/Talk2Traffic/](https://zihaosheng.github.io/Talk2Traffic/) Abstract Deploying autonomous vehicles (AVs) requires testing in diverse and challenging scenarios to ensure safety and reliability, yet collecting real-world data remains prohibitively expensive. While simulation-based approaches offer costeffective alternatives, most existing methods lack sufficient support for intuitive, interactive editing of generated scenarios. This paper presents Talk2Traffic, a novel framework that leverages multimodal large language models (MLLMs) to enable interactive and editable traffic scenario generation. Talk2Traffic allows human users to generate various traffic scenarios through multimodal inputs (text, speech, and sketches). Our approach first employs an MLLM-based interpreter to extract structured representations from these inputs. These representations are then translated into executable Scenic code using a retrieval-augmented generation mechanism to reduce hallucinations and ensure syntactic correctness. Furthermore, a human feedback guidance module enables iterative refinement and editing of scenarios through natural language instructions. Experiments demonstrate that Talk2Traffic outperforms state-of-the-art methods in generating challenging scenarios. Qualitative evaluations further illustrate the framework can handle diverse input modalities and support scenario editing. 1. Introduction Recent years have witnessed remarkable progress in autonomous vehicles (AVs), yet safety concerns continue to pose significant challenges for their widespread deployment on public roads [17, 19, 34] . Ensuring the reliability of AVs requires extensive training and validation across diverse and complex traffic scenarios that encompass the full spectrum of real-world driving conditions [22, 28, 32, 33] . While existing large-scale autonomous driving datasets, including Waymo Open Dataset [15], nuScenes [4], and Argoverse [5], have provided valuable traffic data, the process * Corresponding author: [sikai.chen@wisc.edu](sikai.chen@wisc.edu) Figure 1 . Talk2Traffic enables human designers to create traffic scenarios through multimodal inputs (text descriptions, speech commands, and freehand sketches) . of collecting and annotating real-world scenarios remains prohibitively expensive and time-consuming. Furthermore, these datasets inherently capture a limited subset of possible traffic interactions, leaving AVs potentially vulnerable to unforeseen scenarios. This gap motivates the development of more cost-effective and flexible approaches through simulation-based traffic scenario generation. Existing approaches for traffic scenario generation in simulation environments have evolved along several directions [10] . Traditional rule-based methods, such as CARLA ScenarioRunner [7], enable structured scenario configuration through parameterized templates, offering repeatable test cases under predefined conditions. However, their reliance on hand-crafted rules restricts scenario diversity and requires extensive domain expertise to model complex agent interactions. Data-driven approaches, particularly those leveraging diffusion models [21, 27], have demon- strated promising results in generating diverse and realistic traffic scenarios by learning from real-world data distributions. These methods can capture complex interaction patterns that are difficult to specify through explicit rules. Nevertheless, a critical li

FAQs

Why is real-world data collection for autonomous driving scenarios expensive?

Collecting and annotating scenarios in the real world is prohibitively expensive and time-consuming, which limits coverage of possible traffic interactions.

What problem does Talk2Traffic address compared with existing scenario generators?

Existing methods generate scenarios but do not adequately support instruction-guided, interactive editing of agent behaviors or scene layouts during generation.

How does Talk2Traffic generate and refine traffic scenarios?

It extracts structured representations from multimodal inputs using an MLLM-based interpreter, translates them into executable Scenic code with retrieval-augmented generation to reduce hallucinations, and uses a human-feedback guidance module for iterative natural-language refinement.

Oct 3, 2026

Upload by Chumphorn