Plug-and-Play VQA - PNP-VQA - Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

Aug 26, 2026

Upload by Liam

Visual question answering (VQA) under the zero-shot setting is a difficult vision-language reasoning task. This paper proposes Plug-and-Play VQA (PNP-VQA), a modular framework that enables zero-shot VQA without additional training of large pretrained language models. By using question-guided informative image captions as an intermediate representation, the method connects pretrained vision-language and language models for question answering. It achieves state-of-the-art results on zero-shot VQAv2 and GQA, outperforming strong end-to-end baselines.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

Introduction

Related Work

Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training Anthony Meng Huat Tiong 1 ,2 , Junnan Li 1 , Boyang Li2 , Silvio Savarese 1 , and Steven C.H. Hoi 1 1 Salesforce Research 2Nanyang Technological University, Singapore {anthony.tiong, [junnan.li](junnan.li) , ssavarese, [shoi}@salesforce.com](shoi}@salesforce.com) [boyang.li@ntu.edu.sg](boyang.li@ntu.edu.sg) [https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa](https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa) Abstract Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 (Goyal et al., 2017) and GQA (Hudson and Manning, 2019) . With 11B parameters, it outperforms the 80B-parameter Flamingo model (Alayrac et al., 2022) by 8.5% on VQAv2 . With 738M PLM parameters, PNPVQA achieves an improvement of 9.1% on GQA over FewVLM (Jin et al., 2022) with 740M PLM parameters. 1 Introduction Recent years have witnessed unprecedented performance gains on many natural language reasoning tasks, especially in zero-shot and few-shot settings, being derived from scaling up pretrained language models (PLMs) and their training data (Devlin et al., 2019 ; Liu et al., 2019 ; Brown et al., 2020 ; Raffel et al., 2020 ; Black et al., 2022 ; Sanhet al., 2022 ; Wei et al., 2021) . Inspired by their success, a natural thought is that utilizing PLMs should also boost zero-shot performance in visionlanguage reasoning tasks. However, to leverage PLMs for vision-language tasks, most existing methods require non-trivial adaptation of the PLMs for the vision modality, which necessitates the design of new network components and training objectives. For example, Sung et al. (2022) and Alayrac et al. (2022) insert into the PLMs new layers that are trained from scratch. Tsimpoukelli et al. (2021) train vision encoders that output soft prompts to frozen PLMs. Chen et al.(2022) and Eichenberg et al. (2021) train both the vision encoders and new layers inserted into PLMs. In the zero-shot setting, various vision-language pretraining objectives are employed, such as image captioning (Alayrac et al., 2022) and imageconditioned masked language modeling (Jin et al., 2022) . From the perspective of general-purpose AI, the ability to perform new tasks by simply recombining large-scale pretrained models, or foundation models (Bommasani et al., 2021), without architectural changes or extra training would be highly desirable. Such a system would be able to dynamically adjust to previously unknown tasks by simply rewiring a small number of foundation models. However, to obtain high performance without some form of end-to-end training would seem difficult, if not impossible. We present Plug-and-Play VQA (PNP-VQA), a framework for zero-shot visual question answering which conjoins large pretrained models with zero additional training and achieves state-of-the-art performance on zero-shot VQAv2 (Goyal et al., 2017) and GQA (Hudson and Manning, 2019) . For the purpose of bridging the vision and language modalities, we employ a pretrained vision-language model (PVLM) (Li et al., 2022b) that describes visual information with textual captions. In order to obtain relevant and informative captions, we apply a network interpretability techniq

FAQs

What is Plug-and-Play VQA (PNP-VQA)?

PNP-VQA is a modular framework for zero-shot visual question answering that combines pretrained vision-language and language models without additional training.

How does PNP-VQA bridge vision and language modalities?

It generates question-guided informative image captions using a network interpretability technique, then feeds the captions to a pretrained language model to answer the question.

What performance improvements does PNP-VQA achieve?

PNP-VQA reaches state-of-the-art zero-shot results on VQAv2 and GQA, including an improvement over Flamingo and FewVLM at comparable parameter scales.

Aug 26, 2026

Upload by Liam