A2utoLPBench - An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Construction

Sep 21, 2026

Uploaded by Ivy

Most LP-from-text benchmarks are static word-problem datasets created with manual writing and labeling. Such releases lock size and difficulty, and each problem can leak into future LLM training. A2utoLPBench benchmarks LLM-driven agents on plain-text linear programming tasks with an Auto generator for on-demand fresh problems and an Agent evaluation environment that any agent can run with calibrated scores.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

AccessFree to download

1 Introduction

Large language models and agent-style evaluation

Linear programming as an end-to-end LP-from-text task

Limitations of static LP-from-text benchmarks

Figure 1 and the A2utoLPBench design

Auto inverse-KKT generator

Agent runnable evaluation surface

arXiv :2607 .02 14 1v 1 [ cs .AI] 2 Jul 2026 A2 utoLPBench: An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Construction Shuo Ren Yaohui Han Yifan Shi Libo Shen Haodong Lu Dongfang Wu Rongliang Fu Bei Yu Tsung-Yi Ho The Chinese University of Hong Kong Abstract Most LP-from-text benchmarks are static datasets of word problems written and labeled by hand. Once such a dataset is released, its size is fixed, its difficulty is fixed, and every problem can leak into the training data of future LLMs. We present A2utoLPBench* , a benchmark for testing LLM-driven agents on linear programming problems written in plain text. It has two parts, Auto and Agent: a generator that produces fresh problems on demand, and a ready-to-run evaluation environment that any LLM-driven agent can plug into. The generator builds each LP problem in reverse. We first pick a feasible point and dual, then write down a problem for which that point is optimal and the objective value is known. The answer is known by construction, with no solver call and no human annotator. The evaluation environment bundles a reference solver-critic baseline and a Docker image whose usage instructions are written for an LLM-driven agent to read. With these in place, any agent can run the benchmark and get a calibrated score with one command. Because the benchmark is a generator rather than a fixed dataset, it has properties no fixed dataset can match: an unlimited supply of fresh problems, a difficulty knob set by (n, m), ground-truth answers correct by construction, low LLM-side cost per problem relative to human authoring, repeatable scores across independent batches, and resistance to training-data leakage when fresh post-cutoff seed ranges are used. We show empirically that solve rates change smoothly with problem size, that independent batches give matching scores, and that the bundled solver-critic baseline also helps on existing LP benchmarks. 1 Introduction Large language models (LLMs) have advanced rapidly toward agent-style code execution and tool use Anthropic [4], DeepSeek [10, 11], Moonshot AI [26], OpenAI [27], Qwen Team [29] . This shift has created a need to measure not the surface quality of LLM output but the end-to-end problem-solving capability of an LLM-driven agent on concrete tasks. A growing body of agent-style benchmarks targets this need across different task families [20, 22, 28] . Linear programming (LP) is one such task family. An LP-from-text agent [1, 19, 30, 38] receives a problem description in natural language, builds the mathematical formulation, generates solver code, executes it, and reports the optimum. Formulation, code generation, and numerical fidelity must all be correct simultaneously, which makes LP-from-text a strict end-to-end test of LLM-agent capability. Reliable evaluation of these agents demands a test set of problem–answer pairs whose answers are verified correct. Producing such pairs is laborious: benchmarks such as NL4OPT [30] and MAMO [19] each assembled hundreds of human-authored word problems whose gold answers were labeled by hand, a process that is slow, costly, and difficult to scale. * Open-sourced at [https://anonymous.4open.science/r/AutoLPBench/](https://anonymous.4open.science/r/AutoLPBench/); usage and agent plug-in instructions are in Section H. Preprint. Figure 1: Construction pipelines compared and the motivation of the A2utoLPBench. Left: traditional LP benchmarks require three coordinated human roles (Teacher / Student / Coder) . Right: A2utoLPBench replaces them with an inverse-KKT routine, an LLM-driven Natural Language (NL) drafter, and an automation layer that streams instances at any seed and dimension, removing per-item human labor and making the supply and the difficulty distribution programmable. Beyond their production cost, these benchmarks share a deeper structural disadvantage: they are all released as static corpora, a fixed set of human-authored word problem

FAQs

What problem does A2utoLPBench target compared with existing LP-from-text benchmarks?

It targets limitations of static datasets by enabling end-to-end evaluation of LLM-driven agents on linear programming problems without fixed, human-labeled corpora.

How does the Auto part generate LP problems in A2utoLPBench?

It uses an inverse-KKT routine: it selects a primal-dual-coefficient triple and reverse-derives an LP whose optimum is known by construction, avoiding solver calls and human annotation.

What does the Agent part provide for evaluation?

It bundles a solver-critic baseline and a Docker environment with agent-readable documentation, so tool-use-capable agents can run the benchmark with a single entry point and obtain calibrated scores.

Sep 21, 2026

Uploaded by Ivy