RoboVista：评估用于多样机器人应用的视觉语言模型

Jul 16, 2026

Upload by Maeve

RoboVista proposes a modular evaluation approach for aligning Vision–Language Models (VLMs) with real robotic needs across diverse embodiments, visual conditions, and planning structures. The Robot Question Answering (RQA) framework decomposes robot behavior into decision components and builds high-quality robot-centric VQAs with expert annotation. RoboVista provides 474 multiple-choice VQA instances covering 39 task types across agricultural, industrial, domestic, surgical robotics, autonomous driving, and open-robot datasets. Experiments show notable gaps for state-of-the-art VLMs and strong correlation with real-world task execution.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

Introduction

Visual Question Answering as Evaluation

Modular Decision-Making Motivation

Robot Question Answering (RQA)

RoboVista Dataset Overview

Benchmark Scope and Annotation

Module Abstraction of Robot Pipelines

arXiv :2607 .046 10v 1 [ cs .RO] 6 Jul 2026 RoboVista: Evaluating Vision Language Models for Diverse Robot Applications Shuangyu Xie 1 ,∗ , Kaiyuan Chen 1 ,∗ , Ziyang Chen 1 , Simeon Adebola 1 , Yixuan Huang2 , Zehan Ma 1 , Tianshuang Qiu 1 , Wentao Yuan3 , Dhruv Shah2 ,3 , Pannag R. Sanketi3 , Ken Goldberg 1 1University of California, Berkeley 2Princeton University 3 Google DeepMind ∗Equal contribution Fig. 1: RoboVista Overview. To support future robot applications, RoboVista presents fine-grained spatial understanding and embodied decision-making challenges for Vision–Language Models (VLMs) . Grounded in 6 robot application domains and 39 diverse tasks, RoboVista is an expert-annotated Visual Question Answering (VQA) dataset emphasizing variable robot embodiments (left), interactions with deformable objects and complex and cluttered scenes (middle), and long-horizon contextual understanding (right) . Abstract—Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision–Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution. [https://berkeleyautomation.github.io/robovista](https://berkeleyautomation.github.io/robovista) I. INTRODUCTION Deploying robots in real-world domains such as industrial automation [1, 2], agriculture [3, 4], and surgery [5, 6] requires a general-purpose interface that can reason over diverse decisions, embodiments, and situations. Vision-Language Models (VLMs) [7, 8] are a potential backbone for these systems, as their language-driven reasoning has shown strong capabilities in complex domains such as software engineering [9] and has shown early success in robot manipulation [10–12] . However, to support real robot tasks, VLMs must deliver consistent, spatially grounded decision-making: for example, on an industrial gasket assembly line, a VLM must accurately judge whether a gasket is properly seated and choose the correct next action (re-seat or fasten) . Toward this goal, we present RoboVista, a high-quality benchmark that covers diverse robots application scenarios (as shown in Fig. 1) . A primary way to evaluate this capability is Visual Question Answering (VQA) [13–18], where a model is given images or videos and must answer task-relevant questions that probe scene understanding and action selection. Existing efforts, such as Robo2VLM [19] and RoboBrain [20], largely draw from imitation learning datasets such as Open X-Embodiment [21], and have shown success in improving spatial capabilities. However, many existing real-world robotic applications are fundamentally modular: complex behaviors are decomposed into task-level decisions (e.g., task sequencing, action planning, and recovery) that are hard to capture by end-to-end robot trajectories. This motivates extending evaluation to these modular decisions with broader application domains, especially where public data is scarce, and many critical decisions are handled by analytical pipel

FAQs

What problem does RoboVista address for robotics with vision–language models?

RoboVista targets the need for consistent, spatially grounded decision-making across varying robot embodiments, visual conditions, and complex planning scenarios. It emphasizes evaluation of modular decision components behind robot behavior rather than only end-to-end trajectories.

How does the RQA framework support RoboVista’s evaluation design?

RQA systematically decomposes robot applications using standard robotic abstractions and unifies expert annotation, algorithmic execution, and automated question construction through a shared RobotVQA interface. This enables principled creation of robot-centric VQAs.

What does the RoboVista benchmark include and how broad is it?

RoboVista contains 474 multiple-choice visual question answering instances with human annotated reasoning. It covers 39 task types spanning surgical, agricultural, industrial, domestic, autonomous driving, and open robot datasets.

Jul 16, 2026

Upload by Maeve