MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

Sep 17, 2026

Large pre-trained models have demonstrated strong zero-shot and prompt-based few-shot abilities in unimodal vision and language tasks. This work introduces MAPL, a simple, parameter-efficient approach that reuses frozen unimodal pre-trained models and exploits their generalization in multimodal vision-language settings. MAPL learns a lightweight mapping between representation spaces using aligned image-text data, enabling generalization to unseen VL tasks from only a few in-context examples. Experiments on visual question answering and image captioning benchmarks show MAPL achieves superior or competitive performance while training far fewer parameters, with code and model weights released publicly.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

AccessFree to download

1 Introduction

Background and motivation

Problem: aligning vision and text representation spaces

Approach: MAPL

Contributions and evaluation overview

2 Related Work

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting Oscar Mañas△ Pau Rodriguez∗ ,♢ Saba Ahmadi∗ ,△ Aida Nematzadeh♡ Yash Goyal♣ Aishwarya Agrawal△ ,† △Mila, Université de Montréal ♢ ServiceNow Research ♡DeepMind ♣ Samsung-SAIT AI Lab, Montreal †Canada CIFAR AI Chair [oscar.manas@mila.quebec](oscar.manas@mila.quebec) Abstract Large pre-trained models have proved to be remarkable zero- and (prompt-based) fewshot learners in unimodal vision and language tasks. We propose MAPL, a simple and parameter-efficient method that reuses frozen pre-trained unimodal models and leverages their strong generalization capabilities in multimodal vision-language (VL) settings. MAPL learns a lightweight mapping between the representation spaces of unimodal models using aligned image-text data, and can generalize to unseen VL tasks from just a few in-context examples. The small number of trainable parameters makes MAPL effective at low-data and in-domain learning. Moreover, MAPL’s modularity enables easy extension to other pretrained models. Extensive experiments on several visual question answering and image captioning benchmarks show that MAPL achieves superior or competitive performance compared to similar methods while training orders of magnitude fewer parameters. MAPL can be trained in just a few hours using modest computational resources and public datasets. We release our code and pre-trained model weightsat [https://github.com/oscmansan/mapl](https://github.com/oscmansan/mapl). 1 Introduction Over the past few years, natural language processing and computer vision have witnessed impressive progress in learning models capable of transferring to unseen tasks or benchmarks (Brown et al., 2020 ; Zhang et al., 2022 ; Radford et al., 2021 ; Jia et al., 2021) . Recently referred to as foundation models (Bommasani et al., 2021), these can be adapted to a wide range of unimodal vision and language tasks without any additional training. In this work, we study reusing such powerful unimodal foundation models for multimodal visionlanguage (VL) downstream tasks. In particular, we propose to connect a vision encoder, such as * denotes equal contribution. CLIP (Radford et al., 2021), to an autoregressive language model (LM), such as GPT (Radford et al., 2018, 2019 ; Brown et al., 2020), with minimal additional training on multimodal data. Our goal is to obtain a single VL model that can leverage the in-context learning abilities (Brown et al., 2020) of the pre-trained LM to generalize to unseen VLtasks from just a few examples. One challenge in connecting vision encoders with LMs is aligning the visual and textual representation spaces. Recent works have approached this by adapting the LM to visual representations, either by fine-tuning the entire LM (Dai et al., 2022) or training adapter layers (Eichenberg et al., 2021 ; Alayrac et al., 2022) . These systems are computationally expensive to train as they have a large number of learnable parameters (hundreds of millions toa few billions) and use large-scale multimodal training data. On the other hand, Frozen (Tsimpoukelliet al., 2021) keeps the LM frozen, thus learning ∼ 10× less parameters than the above methods. However, it requires training a visual encoder from scratch, which is also computationally expensive. Differently, we aim to reuse large pre-trained unimodal models while keeping them completely frozen and free of adapter layers. We present MAPL (Multimodal Adaptation of Pre-trained vision and Language models), a simple and parameter-efficient VL model capable of tackling unseen VL tasks. MAPL learns a lightweight mapping between the representation spaces of pretrained unimodal models. MAPL has orders of magnitude fewer parameters than previous methods (including Frozen) and can be trained in just a few hours. Moreover, MAPL’s modularity makes it general-purpose and easily extensible to newer and/or better pre-traine

FAQs

What is MAPL and what problem does it address?

MAPL is a parameter-efficient method for multimodal vision-language few-shot prompting. It connects frozen unimodal vision encoders and language models by learning a lightweight mapping between their representation spaces to handle unseen VL tasks with only a few examples.

How does MAPL achieve few-shot generalization on unseen vision-language tasks?

MAPL learns an efficient representation-space mapping using aligned image-text data. At inference time, it leverages the in-context learning abilities of the pre-trained language model to generalize to unseen VL tasks from few in-context examples.

Why is MAPL considered more parameter-efficient than prior approaches?

MAPL keeps both unimodal pre-trained models completely frozen and introduces only a small number of trainable parameters for the mapping. The paper reports training with orders of magnitude fewer parameters than methods that fine-tune the full model or train large adapter layers.

Sep 17, 2026

Uploaded by 8796093062539