Mixture of Probes Learning from Privileged Modalities in Multimodal LLMs Through Probing

Jul 16, 2026

Upload by Miles

Multimodal Large Language Models (MLLMs) are often trained assuming every modality used at training is available at inference, yet real deployments frequently follow a privileged modality setting where auxiliary modalities exist only during training. Mixture of Probes (MoP) disentangles modality-specific and modality-general signals inside a multimodal LLM to preserve modality-dependent structure while learning transferable representations. A structured probing mechanism extracts organized information from intermediate encoder states rather than only final-layer alignment. MoP Cross-modal Training (MoP-X) adds a probe disentanglement loss to prevent collapse and promote cross-modal learning. Across two domains, eight tasks, and four modalities, MoP consistently outperforms strong baselines, reaching up to 65% relative improvement under modality-only inference.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Privileged modality setting

Limitations of existing MLLMs

Proposed Mixture of Probes (MoP)

Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing Dominick Reilly 1 ,3 ∗ Qiyu Wu 1† Hiromi Wakaki 1 Srijan Das3 Yuki Mitsufuji 1 ,2 1 Sony Group Corporation 2 Sony AI 3 University of North Carolina at Charlotte [dreilly1@charlotte.edu](dreilly1@charlotte.edu) [firstname.lastname@sony.com](firstname.lastname@sony.com) [https://github.com/Sony/MoP](https://github.com/Sony/MoP) arXiv :2607 .08839v 1 [ cs .CV] 9 Jul 2026 Figure 1: (left) MoP improves single-modality inference across five modalities, outperforming both unimodal MLLMs and naive multimodal training. (right) MoP achieves this through a structured probing mechanism with modality-specific and modality-general probes, which disentangle modality representations before integration with the LLM. Abstract Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training. Code, model checkpoints, and evaluation protocols will be made available at [https://github.com/Sony/MoP](https://github.com/Sony/MoP). ∗ Work done during internship at Sony Group Corporation †Corresponding Author: [qiyu.wu@sony.com](qiyu.wu@sony.com) Preprint. 1 Introduction Stemming from the success of large language models (LLMs) [39, 4, 38], recent multimodal large language models (MLLMs) have enabled language-based interaction with diverse sensory inputs such as vision, audio, and depth [48, 14, 27, 42] . Yet real-world scenarios often exhibit an asymmetric structure: the sensors that are available for training are not always available for deployment. A model may be trained with rich auxiliary streams, such as egocentric video, depth, or audio, but later be required to operate from a single cheap, reliable, or non-invasive modality. This creates a privileged modality setting, where auxiliary modalities provide supervision during training but are unavailable at inference. For example, in activities of daily living (ADL), training data may include egocentric video from wearable devices [11, 10], depth from specialized sensors [15], and exocentric video from static cameras [9, 6], while deployment is typically restricted to exocentric video alone. Existing MLLMs [49, 19, 36, 2] are not designed to address this setting. They primarily view them

FAQs

What is the privileged modality setting in multimodal LLMs?

It is a scenario where auxiliary modalities are available during training but not accessible during inference. The model must therefore learn from training-only information and still perform using only the deployment modality.

How does Mixture of Probes (MoP) improve cross-modal knowledge transfer?

MoP uses a structured probing mechanism to disentangle modality-specific and modality-general signals. It extracts organized information from intermediate encoder representations so that transferable knowledge can be learned beyond final-layer alignment.

What role does MoP Cross-modal Training (MoP-X) play?

MoP-X introduces a probe disentanglement loss designed to prevent probe collapse and encourage cross-modal learning. This supports more effective disentanglement during training.

Jul 16, 2026

Upload by Miles