PolyWorkBench: Multilingual Long-Horizon LLM Agents Benchmarking

Jul 16, 2026

Uploaded by Riley

PolyWorkBench benchmarks multilingual long-horizon LLM agents on realistic workplace workflows that combine planning, tool use, and cross-language interaction. The benchmark includes 67 tasks across five domains: commerce, knowledge work, legal analysis, localization, and manufacturing. Agents must process heterogeneous multilingual inputs, execute iterative reasoning, invoke external tools, and deliver structured outputs. A hybrid evaluation framework integrates structural grading, executable verification, and LLM-based semantic assessment. Results show multilingual workflows significantly degrade performance and create compounding error effects across reasoning and execution steps.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

Motivation and limitations of existing benchmarks

Multilingual NLP benchmarks vs. agentic evaluation

PolyWorkBench design and task scope

arXiv :2607 .06008v2 [ cs .AI] 9 Jul 2026 PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents Hongliang Li* 1 ,2 , Yijin Liu2 , Zhiwei Zhang* 1 ,2 , Zihe Liu2 , Xinyue Lou 1 , Jinan Xu 1 , Fandong Meng2 , Kaiyu Huang 1 1 Beijing Jiaotong University, 2Weixin AI, Tencent Inc ∗Work done during an internship at Weixin AI, Tencent Inc. Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks implicitly assume a monolingual setting, where the entire execution process, including reasoning, tool invocation, and output generation, is conducted within a single language. In contrast, real-world applications often involve multilingual inputs and outputs within a unified workflow, yet the interaction between multilinguality and agentic execution remains underexplored. In this work, we introduce PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows. PolyWorkBench consists of 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing, where agents must process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. To enable comprehensive evaluation, we propose a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment. This design allows us to capture both functional correctness and linguistic consistency across complex workflows. Empirical results show that state-of-the-art LLM agents suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts. Our analysis suggests that multilinguality introduces compounding effects across reasoning and execution steps, highlighting the importance of jointly modeling language variation and procedural decision-making in agent evaluation. Date: July 7, 2026 Correspondence: Hongliang Li at [superhanslerli@gmail.com](superhanslerli@gmail.com), Kaiyu Huang at [kyhuang@bjtu.edu.cn](kyhuang@bjtu.edu.cn) 1 Introduction Recent advances in large language model (LLM) agents have enabled substantial progress in solving complex long-horizon tasks that require planning, tool use, and interaction with external environments. A growing line of work has introduced benchmarks that simulate realistic workflows in web environments, software engineering tasks, and office applications, including SWE-bench Jimenez et al. (2024), WebArena Zhou et al. (2024), OSWorld Xie et al. (2024), and more recent workflow-oriented benchmarks such as OdysseyBench Wanget al. (2025) and CoffeeBench Sugiura et al. (2026) . These benchmarks collectively emphasize sequential decision-making and extended interaction horizons, revealing that current agents still struggle with sustained execution and robust tool coordination. Despite these advances, most existing evaluation settings implicitly assume that the entire agentic process operates within a single linguistic regime. Task instructions, intermediate reasoning, tool interactions, and final outputs are typically expressed in English, with multilinguality either absent or reduced to surface-level translation. This assumption simplifies the evaluation pipeline but overlooks an important characteristic of real-world deployments, where agents must operate over heterogeneous linguistic inputs and outputs within a single coherent workflow. In parallel, multilingual NLP research has extensively studied cross-lingual transfer and multilingual generalization, as reflected in benchmarks such as XGLUE Liang et al. (2020), M-MMLU Lai et al. (2023), and MGSM Shi et al. (2022) . These studies demonstrate that language variation significantly affects model performance, particularly under low-resource or cross-lingual transfer conditions. However, such evaluations are pr

FAQs

What problem does PolyWorkBench address in evaluating LLM agents?

Existing benchmarks usually assume monolingual execution, while real workplace workflows involve multilingual inputs and outputs. PolyWorkBench studies how multilinguality impacts long-horizon agent reasoning, tool use, and generation stability.

What does PolyWorkBench include and how many tasks are in it?

PolyWorkBench contains 67 tasks across five domains: commerce, knowledge work, legal analysis, localization, and manufacturing. Tasks require processing heterogeneous multilingual inputs, iterative reasoning, tool invocation, and structured output generation.

How is PolyWorkBench evaluated?

It uses a hybrid framework combining structural grading, executable verification, and LLM-based semantic assessment. This aims to measure both functional correctness and linguistic consistency across complex workflows.

Jul 16, 2026

Uploaded by Riley