CARD Cross-component Audio Representation Distillation for Encoder-free Audio Captioning

Jul 16, 2026

Upload by Violet

Modern audio captioning systems rely on a frozen audio encoder plus a trainable projector to bridge audio features into a large language model (LLM), but this keeps encoder inference in the loop and constrains acoustic understanding to fixed encoder representations. CARD introduces encoder-free audio captioning by removing the audio encoder at inference. A 13.2M projector feeds a frozen LLM with merged LoRA adapters. Training distills knowledge from CLAP-HTSAT, distributing teacher representations across projector perceptual stages and LLM semantic stages. This placement yields +12.18 CIDEr-D on AudioCaps and +5.21 on Clotho while eliminating encoder cost.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Automated Audio Captioning and Existing Encoder-based Designs

Encoder-free Challenges

Knowledge Distillation and the Need for Cross-component Placement

CARD Approach and Contributions

CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning Ganesh Pavan Kartikeya Bharadwaj Kolluri 1 , Yuchen Zhang 1,2 , Michael Kampouridis 1 , Ravi Shekhar 1,2 1 School of Computer Science and Electronic Engineering, University of Essex 2 Institute for Analytics and Data Science, University of Essex {karthik.kolluri, yuchen.zhang, mkampo, [r.shekhar](r.shekhar}@essex.ac.uk)[}](r.shekhar}@essex.ac.uk)[@essex.ac.uk](r.shekhar}@essex.ac.uk) arXiv :2607 .046 19v 1 [ cs . SD] 6 Jul 2026 Abstract—Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder’s inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher’s representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher’s knowledge is placed matters as much as its presence. Index Terms—audio captioning, encoder-free models, crosscomponent distillation, audio-language models. I. INTRODUCTION Automated audio captioning (AAC) is the task of generating a natural-language description of the sound events in an audio clip, such as “rain falls as thunder rumbles in the distance.”Unlike automatic speech recognition (ASR), which transcribes spoken words from the speech audio, AAC aims to describe the broader acoustic scene, capturing non-speech sounds and their relationships in free-form natural language. This ability to summarize what an audio clip contains has made AAC valuable for applications such as accessibility, audio retrieval, and content understanding [1], [2] . Most recent AAC systems follow a common architectural design: A frozen pre-trained audio encoder, such as CLAP [3], first converts the raw audio clip into a sequence of acoustic feature vectors. A trainable projector then maps these features into the input space of the Large Language Model (LLM), which decodes them into a caption. Since both the audio encoder and the LLM are pre-trained separately and usually kept frozen, only the projector is trained to bridge the two components together. This design has been widely adopted because it inherits the strengths of a powerful audio encoderand a capable language model while learning little more than the alignment between them [4]–[6] . Its main limitation, however, is that inference still depends on the dedicated audio encoder. Every clip must pass through a full encoder forward pass before the language model can act, which adds computation and memory, and confines the language model to perceiving audio only through the encoder’s fixed features. A natural way to remove this dependency is to discard the audio encoder from the pipeline and let the model caption audio on its own. However, without the encoder, the projector and the language model must learn to perceive audio directly from the raw acoustic signal, rather than building on representations from a model already trained on large amounts of audio. Learning to extract useful acoustic features from scratch, while at the same time learning to generate captions, places a much heavier burden on training and tends to be unstable. For this reason, encoder-free models have generally struggled to reach the performance of their encoder-based counterparts [7], [8] . A promising way to ease this difficulty is knowledge distillation. Instead of learning to

FAQs

What changes occur at inference time in CARD?

The teacher and distillation heads are removed, and the LoRA adapters are merged into the base LLM. The deployed model contains only the audio projector and the language model, with no audio encoder used at inference.

Jul 16, 2026

Upload by Violet