Projector Is All You Train

Sep 15, 2026

Uploaded by Rowan

The typical training of multimodal large language models (MLLMs) adapts both the language backbone and a projector that connects the backbone to a modality-specific encoder. The study tests whether backbone fine-tuning is necessary by conducting experiments on 3D MLLMs, showing that projector-only training can reach strong multimodal performance versus existing baselines and jointly trained models. Joint training also causes undesired language-model capability drift, which is avoided when the backbone is frozen. The work reports roughly double training throughput for projector-only training. Findings are validated across different language-model backbones using 3D classification, captioning, and language-vision-spatial reasoning benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Related Work

3D MLLMs

Point-BERT encoder

Projector Is All You Train Nyx Iskandar∗ Ramen VR [nyx@ramenvr.com](nyx@ramenvr.com) Saathvik Selvan∗† University of California, Berkeley [sselvan@berkeley.edu](sselvan@berkeley.edu) Slater Victoroff [iph.so](iph.so)[s@iph.so](s@iph.so) arXiv : 2608 . 19726v 1 [cs .CL] 20 Aug 2026 Abstract The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities. 1 Introduction Multimodal large language models (MLLMs) extend pretrained language models with representations from additional modalities, such as images, audio, or 3D data [1, 4, 20, 22, 32] . A common architecture visualized in Figure 1 maps features from a pretrained modality-specific encoder into the language model’s embedding space through a learned projector. Training typically proceeds in two stages: the projector is first trained to align encoder features with the language model (LM), after which both the projector and LM are adapted on multimodal instruction data [9, 13, 21, 22, 43] . We question whether adapting the LM in this second stage is necessary. For 3D MLLMs, we find that training the projector alone is sufficient to achieve strong multimodal performance. Across multiple LM backbones, models trained with a frozen LM perform comparably to existing baseline modelson 3D classification and captioning benchmarks [3, 37, 43, 44] . Furthermore, we independently executed training runs where both the projector and LoRA adapters [19] of the backbone are jointly trained in a one-stage process using the same dataset as the projector-only training runs. Comparing the 3D classification performance throughout training under matched compute budgets, we find that projector-only training is consistently competitive with joint training when using the same point cloud encoder and LM backbone. These results suggest that training the projector alone can be sufficient to adapt an MLLM to a new modality. In our experiments, projector-only training is approximately twice as fast as joint training. Moreover, joint training introduces regressions in LM performance across language, vision, and spatial reasoning benchmarks. This drift is guaranteed to be absent in projector-only training by construction, since the backbone weights are never updated. ∗ Equal Contribution. Correspondence to: [nyx@ramenvr.com](nyx@ramenvr.com or nyx@berkeley.edu)[ or](nyx@ramenvr.com or nyx@berkeley.edu)[ nyx@berkeley.edu](nyx@ramenvr.com or nyx@berkeley.edu). †Work done while at Ramen VR. Preprint. Figure 1: Architecture of 3D MLLMs in this paper. This figure depicts projector-only training, where all parameters are frozen except those of the projector. An alternative training regime is joint training, where all parameters are frozen except those of the projector and LoRA adapters of the LLM backbone. 2 Related Work 3D MLLMs. Existing 3D multimodal large language models (MLLMs) differ primarily in their point-cloud representations, modality-alignment strategies, and forms of instruction supervision. ShapeLLM [32] targets interaction-oriented 3D und

FAQs

What question does the paper address about MLLM training?

Whether fine-tuning the language backbone is necessary when adapting an MLLM to a new modality, compared with training only the projector.

What do experiments on 3D MLLMs show about projector-only training?

Training only the projector is sufficient to achieve strong multimodal performance, performing comparably to baseline models across multiple language-model backbones.

How does joint training affect language-model capabilities?

Joint training introduces regressions and undesired drift in the language model’s existing capabilities, while projector-only training avoids drift by freezing the backbone.

Sep 15, 2026

Uploaded by Rowan