Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking

Jul 16, 2026

Upload by Liam

Multimodal Video-to-Music Recommendation (VTMR) introduces a two-stage framework that improves how relevant music is matched to a video. Stage 1 aligns multimodal video, music, and LLM-generated text in a joint audio-visual-text embedding space, retrieving semantically compatible candidates with coarse global embeddings. Stage 2 performs temporal reranking using cross-modal attention over video and music sequences, capturing fine-grained timing correspondence. Experiments show improved R@10 and Median Rank over strong baselines, and a human study confirms comparable preference to a commercial system while producing better music quality than a generative baseline.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Abstract

Introduction

Methods

Multimodal Media Encoding

Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking Seungheon Doh 1 * Minhee Lee 1 Sangmoon Lee 2 Ben Sangbae Chon 2 Juhan Nam 1 [https://seungheondoh.github.io/video-to-music-demo](https://seungheondoh.github.io/video-to-music-demo) arXiv :2607 .0597 1v 1 [ cs .MM] 7 Jul 2026 Abstract We present VTMR, a two-stage framework for Video-To-Music Recommendation. In Stage 1, VTMR aligns comprehensive video and music signals in a joint audio-visual-text representation space and efficiently retrieves semantically compatible candidates using coarse global embeddings. In Stage 2, it reranks the retrieved candidates by attending to the temporal sequences of both video and music, thereby capturing finegrained temporal correspondence. Evaluated on the video-to-music recommendation task, the multimodal retrieval stage improves R@10 from 14.2 to 15.9 and Median Rank from 75 to 58 over the strongest baseline; the temporal reranker further boosts R@10 to 18.3 and Median Rank to 46, demonstrating complementary gains from richer query encoding and temporal alignment. A human preference study confirms that VTMR is on par with a commercial baseline in overall preference, while outperforming a generative baseline in music quality. 1. Introduction Music is a cornerstone of compelling video content, shaping emotional tone, narrative pacing, and audience engagement across domains from cinematic productions to short-form social media. Yet selecting appropriate background music is far from trivial: it requires not only an understanding of high-level semantic compatibility, such as mood and genre, but also a fine-grained understanding of the temporal correspondence between evolving visual dynamics and musical elements. Identifying music that satisfies both criteria is time-consuming and often yields suboptimal results. 1 Graduate School of Culture Technology, KAIST, South Korea 2 Gaudio Lab, Inc, South Korea. ∗Work completed while Seungheon was visiting Gaudio Lab. Correspondence to: Seungheon Doh <[seungheon.doh@gmail.com](seungheon.doh@gmail.com) >. International Conference on Machine Learning (ICML) 2026, Learning to Listen: Workshop on Machine Learning for Audio, Copyright 2026 by the author(s) . Automatic video-to-music (V2M) recommendation has therefore attracted growing interest as a cross-modal music retrieval task (Li & Kumar, 2019 ; Huang et al., 2022 ; Doh et al., 2023b ; Wu et al., 2025), aiming to learn joint embedding spaces between visual content and music (Sur´ıset al., 2022 ; Prtet et al., 2023 ; Wilkins et al., 2023 ; Stewart et al., 2025) . Despite steady progress, current frameworks leave substantial room for improvement across two major dimensions. First, while general-purpose multimodal models (Guzhov et al., 2022 ; Girdhar et al., 2023 ; Zhu et al., 2024) have demonstrated the power of joint audio-visualtext representations, established V2M methods still rely almost exclusively on raw visual features. Even though some previous works (McKee et al., 2023) have utilized both visual and textual modalities for V2M recommendation, existing architectures have yet to fully exploit the multimodal richness of both video and music signals. Second, all existing V2M methods reduce retrieval to a single global embedding similarity score, compressing entire video and music streams into static vectors. While global embeddings are effective for capturing coarse semantic compatibility, such as overall mood or genre, this single-vector bottleneck fundamentally cannot represent how localized musical events align with specific video moments. The temporal dimension is collapsed by design, making it impossible to distinguish music that is globally compatible from music that is temporally aligned with specific video moments. To address both limitations, we propose VTMR, a two-stage framework capable of capturing both unified multimodal semantics and temporal dynamics (see Figure 1) . Stage 1 (Sem

FAQs

What is VTMR, and how does it structure video-to-music recommendation?

VTMR uses a two-stage design: semantic retrieval followed by temporal reranking. Stage 1 retrieves candidates using aligned multimodal embeddings, and Stage 2 reranks them by attending to temporal sequences across video and music.

How does Stage 1 perform retrieval in VTMR?

Stage 1 projects comprehensive video, music, and LLM-generated text into a shared audio-visual-text representation space. It then retrieves top-N candidates efficiently via similarity computed from coarse global embeddings.

Why does VTMR include temporal reranking, and what does Stage 2 do?

VTMR addresses the limitation of single global embeddings that collapse time and cannot model localized alignment. Stage 2 applies a fine-grained cross-encoder that attends to dense, unpooled temporal sequences of the video and each candidate music track to capture temporal correspondence.

Jul 16, 2026

Upload by Liam