Opti Agent Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

Jul 16, 2026

LLM agents increasingly target optimization tasks, yet existing benchmarks often assess them on pre-structured mathematical forms, skipping the hardest step: converting complex business requirements into correct models and solving them efficiently. Opti-Agent-Bench provides an end-to-end evaluation across the full optimization R&D pipeline—business-language understanding, mathematical modeling, algorithm selection and code implementation, and solution report generation. Three design pillars—anti-template traps, cross-module consistency checks, and an ORAC bi-level validity framework—reveal failures such as constraint omission, model-code inconsistency, and report-implementation divergence.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages57

Reading Time2 h 24 min

Introduction

The Evaluation Gap

Opti-Agent-Bench Design

Business-Semantic Authenticity with Anti-Template Traps

Modular Evaluation with Cross-Module Consistency Checking

ORAC Bi-level Validity Framework

Benchmark Tasks and Findings

Failure Modes Revealed on Industrial-Scale Problems

arXiv :2607 . 10768v 1 [ cs .AI] 12 Jul 2026 Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems Yongchang Fu 1 , Xinjie Huang 1 ,2 , Chengjun Dai 1 , Chengzhe Feng 1 , Junshao Zhang 1 , Hong Zhu 1 1 Ding Talk, Alibaba Group, Hangzhou, China 2 Zhejiang University, Hangzhou, China {fuyongchang.fyc, chengjun.dcj, fengchengzhe.fcz, jiushao.zhangjs, [yisu}@alibaba-inc.com](yisu}@alibaba-inc.com) [xinjiehuang@zju.edu.cn](xinjiehuang@zju.edu.cn) Abstract LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured mathematical formulations that bypass the most critical challenge: translating complex business requirements into correct models and solve efficiently. We introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates Large Language Models (LLMs) across the complete optimization R&D pipeline, from understanding business-language descriptions through mathematical modeling, algorithm selection, and code implementation, to solution report generation. Our design rests on three pillars: (1) businesssemantic authenticity with anti-template traps that defeat pattern matching; (2) modular evaluation with cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting; and (3) the ORAC bi-level validity framework that simultaneously ensures task quality and scoring integrity. Across several industrialscale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, that remain invisible under conventional single-metric evaluation. 1 Introduction Operations research and mathematical optimization underpin decision-making across industries—from supply chain logistics and production scheduling to financial portfolio design and energy network management. In practice, however, the path from a business stakeholder’s requirements to a deployed optimization solution is far from straightforward. It demands a complete R&D pipeline: understanding the business context, identifying the underlying optimization structure, formulating a rigorous mathematical model, selecting and implementing an appropriate solution algorithm, and communicating the results through a clear, faithful report (see Figure 1) . Each link in this chain requires deep domain knowledge, mathematical sophistication, and engineering discipline. The rapid advancement of LLM-based agents has generated considerable excitement about automating this pipeline. Systems such as OptiMUS [AhmadiTeshnizi et al. , 2023] have demonstrated that LLM agents can, in favorable conditions, translate natural-language problem descriptions into solver code and iteratively debug their solutions. Meanwhile, foundation model approaches for mixed-integer linear programming (MILP) [Li et al. , 2025a] and knowledge-augmented formulation Figure 1: A Pipeline for Solving Optimization Problems in Real-World Business Scenarios. frameworks [Peng et al. , 2025] suggest that integrating LLMs with optimization tools can yield practically useful results. Yet a fundamental question remains largely unanswered: How reliably can these agents handle the full complexity of real-world optimization R&D? 1.1 The Evaluation Gap Current benchmarks for optimization agents suffer from three interrelated limitations that systematically overestimate agent capabilities. Over-Structured Problem Descriptions. Most existing benchmarks—including NLP4LP [AhmadiTeshnizi et al. , 2023], OptiBench [Yang et al. , 2025], and the NL4Opt competition [Ramamonjison et al. , 2023]—present problems that are already substantially structured. Task descriptions often use academic terminology (“minimize the total weighted tardiness,”“subject to capacity con

FAQs

What gap in existing optimization-agent benchmarks does Opti-Agent-Bench target?

It addresses the fact that most benchmarks rely on pre-structured mathematical formulations, avoiding the core challenge of turning real business requirements into correct models and solving them efficiently.

How does Opti-Agent-Bench evaluate the entire optimization R&D pipeline?

It assesses LLMs end to end, from understanding business-language descriptions to formal mathematical modeling, algorithm selection and code implementation, and faithful solution report generation.

What kinds of model failures does Opti-Agent-Bench expose that single-metric evaluations can miss?

It reveals critical failure modes including constraint omission, model-code inconsistency, and divergence between the generated report and the implemented solution.

Jul 16, 2026

Upload by River Wang