ProAct A Benchmark and Multimodal Framework for Structure-Aware Proactive Response

Jul 16, 2026

Upload by Finn

Proactive agents achieve higher-level objectives by continuously monitoring the environment to decide when and how to act, yet specialized resources for training and evaluation remain limited. ProAct-75 introduces a benchmark spanning 75 proactive visual-response tasks across assistance, maintenance, and safety monitoring, with 91,581 step-level annotations enhanced by explicit task graphs. These graphs capture step dependencies and parallel execution options for structure-aware decision-making. A multimodal baseline, ProAct-Helper, uses state grounding and entropy-driven heuristic search, enabling independent parallel threads. Experiments show improved trigger detection mF1 by 6.21%, fewer steps online, and a 15.58% higher parallel action rate.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Abstract

Introduction

Proactive response settings

Need for structured task representations

Gaps in existing video understanding benchmarks

ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response Xiaomeng Zhu 1 2 Fengming Zhu 1 Weijie Zhou 2 Ye Tian 2 Zhenlin Hu 3 Yufei Huang 2 Yuchun Guo 2 Xinyu Wu 4 Zhengyou Zhang 2 Fangzhen Lin 1 Xuantang Xiong 2 arXiv :2602 .03430v3 [ cs .RO] 13 Jul 2026 Abstract While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human’s next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58% . Code is available at [https://github.com/](https://github.com/)[ ](https://github.com/)[ZhuXMMM/ProAct.git](ZhuXMMM/ProAct.git) 1Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China 2Tencent, Shenzhen, China 3Futian Laboratory, Shenzhen, China 4 Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences, Shenzhen, China. Correspondence to: Fangzhen Lin <[flin@cse.ust.hk](flin@cse.ust.hk)>, Xuantang Xiong <[sheltxiong@tencent.com](sheltxiong@tencent.com)>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) . Previous Current Future Figure 1. Overview of proactive response tasks. ProAct-75 supports five vision-based tasks with step-level annotations, hierarchical labels, and task graphs. Traditional intent-following approaches predict human-intended actions (e.g., tie the bag) and execute them, inadvertently blocking workflows. Our benchmark enables evaluation of strategies where robots pursue independent parallel threads to reduce disruptions. 1. Introduction Unlike passive agents that respond to explicit instructions, proactive agents take initiative toward higher-level objectives by continuously observing the environment, and autonomously selecting actions (Wooldridge & Jennings, 1995; Li et al., 2023; van Den Broek & Moeslund, 2024) . For instance, a robot may replace a trash bag before it overflows in a human-absent scenario, or proactively prepare a new bag while observing a human remove a full one in a collaborative scenario. However, most existing robotic systems still rely on passive instructions, imposing cognitive load and limiting the robot’s autonomous operation (Johannsmeier & Haddadin, 2016; Camilleri et al., 2022; Noormohammadi-Asl et al., 2025) . In this work, we study proactive response for robot agents in settings of human-absent autonomy and human-robot collaboration, where agents must continuously monitor video observations to determine when to intervene and what action to take. Training such proactive agents requires a robust ability of Figure 2. Qualitative examples of ProAct-75 across three application scenarios. We visualize the previous-current-future observation window and structured ann

FAQs

What is ProAct-75 and what problem does it address?

ProAct-75 is a benchmark for training and evaluating proactive agents across diverse domains such as assistance, maintenance, and safety monitoring. It addresses the lack of specialized resources for developing proactive agents.

How do task graphs improve decision-making in ProAct-75?

Task graphs encode step dependencies and parallel execution possibilities. This structural grounding supports feasibility-preserving, structure-aware planning for complex proactive response.

What is ProAct-Helper and how does it select actions?

ProAct-Helper is a reference baseline built on a multimodal large language model that grounds decisions in state detection. It leverages task graphs with entropy-driven heuristic search to choose actions and to allow independent parallel threads.

Jul 16, 2026

Upload by Finn