Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Jul 16, 2026

Upload by Rowan

Video-to-audio (V2A) generation synthesizes audio that matches both the semantics and the timing of a silent video, yet many approaches remain computationally heavy due to multi-stage training or rely on video-to-text conversions that miss fine temporal cues. Flowley introduces end-to-end single-stage training with visual features and textual prompts, and a Progressive Softmasked Cross-Attention that embeds synchronization in attention at no extra cost. SoundCap addresses limited benchmark captions by producing detailed, sound-aware annotations. Flowley sets new results on VGGSound and improves zero-shot audio quality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages32

Reading Time1 h 21 min

Introduction

Video-to-Audio generation

Sound design background

Proposed Flowley approach

Related Works

Video-to-Audio generation

arXiv :2607 .06405v 1 [ cs .MM] 7 Jul 2026 Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space Thanh V. T. Tran 1 , Ngoc-Son Nguyen 1 , Luong Tran 1 , Long-Khanh Pham 1 , Paarth Neekhara2 , Shehzeen Hussain2 , and Van Nguyen 1 1 FPT Software AI Center, Vietnam 2 NVIDIA Corporation, USA [https://flowley-v2a.github.io](https://flowley-v2a.github.io) {thanhtvt1,sonnn45,luongtk,khanhpl2,[vannth19}@fpt.com](vannth19}@fpt.com)[ ](vannth19}@fpt.com){pneekhara,[shehzeenh}@nvidia.com](shehzeenh}@nvidia.com) Abstract. Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Softmasked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting. Keywords: Video-to-Audio · Cross-attention · Video captioning 1 Introduction Sound design is the craft of storytelling through sonic composition. A key branch of this field is Foley [46], where sound effects are created in precise synchronization with onscreen action during post-production. These soundscapes are brought to life by a skilled Foley artist working on a purpose-built stage stocked with a wide array of props and materials for generating the required sounds3. 3 See how a foley artist performing a scene in our project page. 2 Tran et al. Recent advances in generative audio modeling have facilitated text-to-audio (T2A) systems [11, 28, 32], allowing designers to generate sound effects directly from written descriptions. Despite accelerating the search for appropriate sounds, they must still manually adjust timing to align audio with on-screen action. This contrasts sharply with traditional Foley work, where artists naturally craft and sync sound effects in real time by interacting physically with props. To overcome this limitation, video-to-audio (V2A) generation has gained significant attention, dividing into two major directions. The first converts visual features from silent video frames into text, leveraging pretrained T2A models to synthesize sound [51, 57] . While this approach leverages the strengths of pretrained T2A systems, it inevitably discards fine-grained temporal details, which are crucial for film-production sound effects. The second line of work uses multistage training: early phases learn to extract or align acoustic cues from video frames, either via dedicated regression networks [18, 47, 52] or contrastive objectives [31], and subsequent stages build on these pretrained modules. While effective, these pipelines incur substantial delays and computational overhead. Furthermore, integrating textual descriptions into the generation workflow, beyond their use in T2A systems, remains under-explored, despite the demonstrated benefits of text

FAQs

What problem does Flowley address in video-to-audio generation?

Flowley targets two key issues: high cost from multi-stage training and loss of fine-grained temporal cues when visual inputs are converted into text for pretrained text-to-audio models.

How does Progressive Softmasked Cross-Attention (PSCA) help synchronization?

PSCA incorporates audio-visual synchronization directly into the attention mechanism, aligning acoustic and visual information without adding extra computational cost compared to standard attention layers.

Why is SoundCap introduced, and how does it work?

Existing V2A benchmarks lack sound-oriented descriptive captions, which can harm audio semantic consistency. SoundCap is a plug-and-play captioning pipeline that generates detailed, sound-aware captions using pretrained audio-visual large language models to serve as training guidance.

Jul 16, 2026

Upload by Rowan