Long-Horizon-Terminal-Bench：使用密集基于奖励的评分测试长任务中智能体的极限

Jul 16, 2026

Upload by Rowan

Long-Horizon-Terminal-Bench is a challenging terminal benchmark focused on long-horizon, domain-specific agent workflows that existing terminal benchmarks largely miss. It introduces 46 tasks across nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task is decomposed into fine-grained graded subtasks with intermediate checks to provide dense rewards and partial credit. Evaluation shows frontier agents require extensive execution time and tokens yet achieve low pass rates under partial and perfect reward thresholds.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Introduction

Motivation and gaps in existing benchmarks

Long-Horizon-Terminal-Bench design

Evaluation setup and results

arXiv :2607 .08964v 1 [ cs .AI] 9 Jul 2026 Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading Zongxia Li†1 ,2 , Zhongzhi Li†1 ,3 , Yucheng Shi†1 , Ruhan Wang 1 ,5 , Junyao Yang 1 ,7 , Zhichao Liu2 , Xiyang Wu2 , Anhao Li4 , Yue Yu5 , Ninghao Liu8 , Lichao Sun6 , Haotao Mi 1 , LeoweiLiang 1 1 Tencent HY LLM Frontier 2 University of Maryland, College Park 3 University of Georgia 4 University of Minnesota, Twin Cities 5 Indiana University 6 Lehigh University 7 National University of Singapore 8 The Hong Kong Polytechnic University † Core contribution. [zli12321@umd.edu](zli12321@umd.edu) , [zongxiali@global.tencent.com](zongxiali@global.tencent.com) ,[zhongzhili@global.tencent.com](zhongzhili@global.tencent.com) AI agents have become increasingly capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on relatively simple problems that finish within a few minutes and are typically evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, leading to sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a challenging terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on difficult, open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and tens of minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench substantially more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0 .95 and 10.9% at a perfect-reward threshold of 1 .0, while the mean pass rate across models is just 4.3% and 1.7% under the two thresholds, respectively. These results reveal substantial headroom for improvement. We further analyze common failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on robust long-horizon terminal agents. Project page: [https://zli12321.github.io/LHTB/](https://zli12321.github.io/LHTB/) Date: July 13, 2026 1 Introduction AI agents built on large language models (LLMs) are rapidly improving at autonomous decision making and tool use [2, 6] . Recent work has demonstrated impressive performance on short, well-scoped tasks such as fixing a code repository issue, completing a coding ticket, or issuing a handful of shell commands [22] . However, they still cover only a narrow slice of the long-horizon, domain-specific, and practically important workflows that human experts care about in practice [22, 46 , 51] . Nowadays, many real workflows are long-horizon: they require agents to execute hundreds of steps, maintain and update plans over tens of minutes to hours, and manage evolving long-context state [42] . Examples include reproducing results from published research papers [36], installing environments from a github repo, auditing complex multimodal datasets, debugging compiler toolchains, or shepherding a multi-stage ML training pipeline. In these settings, agents must repeatedly use terminal command lines to read and write files, run scripts, inspect partial o

FAQs

Long-Horizon-Terminal-Bench与以往终端基准相比，主要改进是什么？

它将每个长任务分解为细粒度、带中间校验的分项子任务，从而实现密集的中间奖励与部分得分，而不仅仅看最终结果。

Long-Horizon-Terminal-Bench包含哪些任务类型和覆盖范围？

共46个长时域终端任务，覆盖九类内容，包括实验复现、软件工程、多模态分析、交互游戏和科学计算等。

基准评测揭示了哪些性能现状与不足？

评测显示模型平均需要大量tokens与较长执行时间；即使最强模型在部分奖励与完美奖励阈值下的pass@1也很低，不同模型的平均通过率整体偏低，体现了改进空间。

Jul 16, 2026

Upload by Rowan