MM-ToolSandBox：用于评估视觉工具调用智能体的统一框架

Jul 16, 2026

Upload by Ophelia

MM-ToolSandBox is a benchmark and evaluation framework for visually grounded tool-calling agents, designed around a stateful environment with 500+ tools across 16 application domains. It supports multi-image, multi-turn tasks where agents must progressively ground arriving visual inputs into executable tool calls while handling realistic conversation dynamics such as goal revisions, error corrections, and state mutations. An automated scenario generation pipeline creates diverse visual scenarios, and evaluations on 12 state-of-the-art models show success rates below 50%, with visual precision as a key bottleneck.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages32

Reading Time1 h 21 min

Introduction

Motivation and Gap in Existing Benchmarks

Visual Tool-Calling as a Distinct Capability

MM-ToolSandBox Overview

arXiv :2607 . 11818v1 [ cs .CV] 13 Jul 2026 MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents Kaixin Ma∗, Di Feng∗, Alexander Metz, Jiarui Lu, Eshan Verma, Afshin Dehghan Apple ∗ Equal Contribution We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations) . An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at [https://github.com/apple/ml-mmtoolsandbox](https://github.com/apple/ml-mmtoolsandbox). Date: July 14, 2026 1 Introduction Evaluating LLM-based agents on tool-augmented tasks has become an active area of research, with benchmarks targeting function calling (Li et al. , 2023 ; Patil et al. , 2025), stateful environment interaction (Lu et al. , 2025 ; Trivedi et al. , 2024), and multi-turn conversational dynamics (Froger et al. , 2026 ; Barres et al. , 2025 ; Xiu et al. , 2026) . However, most of these frameworks remain text-centric, and cannot evaluate many realistic assistant tasks with visual inputs, for example, a user may share a screenshot to diagnose an issue or provide a photo of an event poster to schedule an event. Solving visual tool-calling tasks requires more than visual recognition. The agent must extract task-relevant information from images, map visual evidence to the correct tool or code action, execute that action against astateful environment, and continue the interaction when the user’s intent evolves. This makes visual tool calling a distinct agentic capability: the model must not only understand what is shown, but also decide how to act on it through tools. Among the few benchmarks that incorporate visual inputs, images are typically provided as a static prefix in the initial query (Wang et al. , 2024a), or the tasks are limited to single-round user-agent interactions (Xie et al. , 2024 ; Rawles et al. , 2025) . Moreover, many existing benchmarks operate with relatively small tool spaces, often containing only 15–100 tools (Froger et al. , 2026 ; Wang et al. , 2024a ; Kong et al. , 2025 ; Lu et al. , 2025) . As a result, there is no unified framework and benchmark for systematically studying the visual tool-calling capability of multimodal agents under dynamic, multi-turn, and large-tool-space settings. To close this gap, we introduce MM-ToolSandBox, a diverse simulation framework for evaluating visual tool-calling agents. MM-ToolSandBox extends the stateful tool-use environments from ToolSandBox (Lu et al. , 2025), with explicit image handling, visual-specific tools, and support for both structured tool-use Environment | Entity Database<br>Images Documents Contacts Mails Transactions<br>Service-pro

FAQs

MM-ToolSandBox评估的核心能力是什么？

它评估具备视觉扎根能力的工具调用智能体：从图像中提取与任务相关的信息，将视觉证据映射到正确的工具或代码动作，并在有状态环境中执行后继续多轮互动。

该框架如何处理动态的多轮对话与真实偏差？

MM-ToolSandBox支持多图片、多轮任务，并显式建模目标修订、错误纠正和状态突变等对话现象，使评测更贴近真实使用过程。

实验结果显示当前模型存在什么主要瓶颈？

失败分析表明瓶颈不仅在规划，还在视觉精度：约53%的失败来自图像信息提取错误，而工作流本身往往是正确的。

基准数据与场景是如何构建和校验的？

通过信息流引导的规划与多阶段质量过滤生成多样化的视觉场景，得到经人工验证的名义场景，并使用专家进行修正与校验，以保证视觉扎根、可执行任务规格和完成标准可靠。

Jul 16, 2026

Upload by Ophelia