BackgroundMellow：面向叙事驱动的电影级多模态连贯音景生成框架

Jul 16, 2026

BackgroundMellow proposes an end-to-end multimodal framework for narrative-driven cinematic soundscape generation from long-form text, addressing limitations of existing Text-to-Audio systems in cohesion, temporal alignment, and emotional depth. The master-specialist architecture decomposes stories into multilayer audio cues, generates each sound category with specialized models, and superimposes aligned soundscapes. Built on Tango2 for environmental synthesis and a Cinematic BGM Retriever, it uses an NLP module to predict mixing parameters (start time, duration, relative loudness) from the narrative timeline and evaluates via nearest-neighbor retrieval and semantic-temporal metrics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Abstract

Contributions

Hierarchical Orchestration Framework

Deterministic Temporal Alignment

Retrieval-Based Evaluation Metrics

Empirical Evaluation and Benchmarking

BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation Ajitesh Jamulkar and Aritra Hazra Department of Computer Science & Engineering, Indian Institute of Technology Kharagpur, India. 13 Jul 2026 Abstract—Generating immersive, synchronized and cinematic audio for long-form textual narratives remains a significant challenge in multi-modal AI. While current Text-to-Audio (TTA) frameworks successfully synthesize isolated sound effects, they struggle with narrative cohesion, temporal alignment, and cinematic emotional depth. We present BackgroundMellow, a framework that treats story-to-audio generation as a precise orchestration and signal processing problem. This framework is enabled without ground-truth through a master-specialist agent architecture that decomposes text into precise and multilayered audio cues, generates each category of sounds with suitable specialist model, and superimposes the soundscapes to create a unified and aligned audio segment. Our pipeline is built over Tango2 latent diffusion model for environmental synthesis alongside a novel Cinematic BGM Retriever mined from professional soundtracks. To automate the sound mixing process, we use an NLP based module that predicts precise audio parameters, like start time, duration, and relative loudness, based on the narrative timeline. We further empirically evaluate and show the efficacy of the proposed framework leveraging nearest-neighbor retrieval against a curated dataset of YouTube advancements in Text-to-Audio (TTA) generation, Latent Diffusion Models (LDMs) and flow-matching arXiv :2607 . 11364v1 frameworks, have demonstrated remarkable success in synthesizing high-fidelity acoustic clips. However, a critical frontier remains largely unsolved: the generation of long-form, multitrack narrative soundscapes. Real-world cinematic storytelling is not a monolithic acoustic event; it is a complex, temporally orchestrated mixture of expressive speech, discrete sound effects (SFX), continuous environmental ambience, and contextual background music (BGM) . Current state-of-the-art end-to-end models often excel at isolated sound generation but fundamentally struggle with compositional and acoustic complexity. When prompted to generate simultaneous, overlapping events, they frequently suffer from phase cancellation, acoustic muddiness, and temporal drift. For instance, successfully rendering the narrative prompt, “It was softly raining as I walked through the forest, where I heard a dog barking,” requires far more than basic semantic understanding. It demands the cohesive orchestration of a continuous rain ambience, an emotive background score, and a discrete dog bark SFX that triggers at the precise temporal onset of the corresponding spoken word. Maintaining the structural integrity of these diverse acoustic elements and superimposing them with millisecond precision remains a highly complicated and challenging task. Furthermore, the absence of holistic and rigorous evaluation mechanisms for such multi-stem acoustic narratives presentsa fundamental research bottleneck. Cinematic sound design is inherently subjective; a generative model might synthesize asoundscape that utilizes a different musical genre or richer ambient textures than a human baseline, making it equally valid or even superior. Consequently, establishing a rigid, absolute “ground truth” dataset for multi-track audio is fundamentally flawed, as it penalizes generative diversity and creative variance. Existing objective metrics, such as Frchet Audio Distance (FAD) [16] or global Contrastive Language-Audio Pretraining (CLAP) [15] scores, are heavily optimized for evaluating isolated acoustic events. They are notoriously illequipped to assess the temporal synchronization, hierarchical audio ducking, and semantic harmony required for a multiclass cinematic mix. To bridge this gap, we propose an end-to-end, multimodal cohesive fram

FAQs

BackgroundMellow解决了文本到音频生成中的哪些关键问题？

它针对长文本叙事音频生成中的叙事连贯性不足、时间对齐困难以及缺乏电影式情感深度等问题，提出以编排与信号处理为核心的框架来统一多轨音景。

框架如何实现多音轨的精确时间对齐与叠加？

BackgroundMellow采用混合对齐引擎，用微调的数字声音预测器将SFX、环境氛围与音乐锚定到TTS主轨提取的词级时间戳，并通过参数化叠加生成对齐的音景片段。

文中提出了哪些评估思路来适配多轨叙事音频？

针对现有评估指标对孤立声事件的优化不足，作者提出基于最近邻检索的评估框架，并引入语义召回与时间交并比（IoU）等指标，用于衡量节奏与同步，同时避免惩罚合理的生成多样性。

Jul 16, 2026

Upload by Clementine