AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

Jul 16, 2026

Upload by Ophelia

Large language models demonstrate strong results on high-school and competition-level math, but advanced mathematical reasoning remains insufficiently measured. Existing benchmarks often use limited disciplinary coverage and rely on final-answer correctness or coarse grading, leaving proof validity and reasoning quality under-evaluated. AdvancedMathBench introduces ProverBench (245 UG and QE problems) with an expert-trained automatic verification pipeline, plus VerifierBench (888 trajectories) to assess proof validity judging and verification rationales.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Introduction

Motivation for advanced proof evaluation

AdvancedMathBench design and components

Evaluation results and remaining challenges

arXiv :2607 . 1 1849v 1 [ cs .CL] 13 Jul 2026 2026-7-14 AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification Lingkai Kong1,2 , Zijian Wu1,3 , Yuzhe Gu1,2 , Haiteng Zhao1 , Wenyong Huang1,2 , Shuang Sun1,2 , Zhicheng Xiong1,2 , Xiaotian Zhang1,2 , Shuya Zhao1,2 , Yan Wang2 , Disheng Xu4 , Wenwei Zhang1† and Kai Chen1† 1 Shanghai AI Laboratory, 2 Shanghai Jiao Tong University, 3 MMLab, The Chinese University of Hong Kong, 4 Great Bay University Large language models (LLMs) have achieved remarkable performance on high-school and competition-level mathematics, yet their capabilities on advanced mathematics remain poorly understood. Existing benchmarks, however, fall short in both scope and evaluation granularity: they provide limited disciplinary coverage and often rely on final-answer correctness or coarse judgments, leaving the validity of the reasoning process inadequately assessed. To bridge this gap, we introduce AdvancedMathBench, a benchmark suite designed to evaluate the reasoning capabilities of LLMs on advanced mathematics. Its core proof-generation benchmark, ProverBench, contains 245 problems spanning undergraduate (UG) and doctoral qualifying-exam (QE) levels. To provide reliable evaluation of the proofs, we develop a dedicated automatic verification pipeline trained on large-scale expert annotations to produce both correctness verdicts and fine-grained assessments of proof errors, which exhibits strong agreement with human experts on held-out proof trajectories. We further introduce VerifierBench, consisting of 888 model-generated proof trajectories paired with expert ground truth, to evaluate whether models can correctly judge proof validity and provide sound verification rationales. Experiments show that AdvancedMathBench remains challenging for frontier models. On proof generation, the best-performing model, GPT-5.5-xhigh, achieves only 64.5 and 48.9 on the UG and QE splits, respectively, indicating substantial room for improvement on advanced mathematical proof construction. On proof verification, the best model only attains a Balanced F1 of only 65.1 suggesting that critical error detection remains a major bottleneck in applying LLMs for proof verification. 1. Introduction Large language models (LLMs) have achieved remarkable progress on mathematical reasoning, especially on high-school and competition-level benchmarks [2, 6, 10, 14, 15, 44] . Yet their capabilities on advanced mathematics remain much less understood. Because advanced mathematics requires the model construct trajectory with rigorous intermediate claims, rather than captured by the final short-answer prediction. Mathematical proof therefore provides a natural stress test for evaluating whether models can reason beyond producing the correct final answer. Existing benchmarks provide limited support for evaluating this capability. Most benchmarks still emphasize competition-style problems, and remain limited in coverage for undergraduate, graduate, and research levels of mathematics [5, 8, 9, 12, 39] . More importantly, many benchmarks are evaluated primarily through final-answer checking or coarse solution matching, which cannot determine whether a proof is mathematically valid. Recent work has explored process verification and LLM-as-Judge [4, 20, 23, 25, 31, 40, 41], but them still suffer from bias and inconsistency. Therefore, current evaluations may lead to an inaccurate estimation of the mathematical capabilities of a model, as they overlook the validity of the underlying reasoning process. This paper aims to provide a more rigorous foundation for measuring model behavior in advanced mathemat- † Corresponding authors Performance Score (%) 100 90 80 70 60 50 40 30 20 10 Meta-Verification TNR 100 90 80 70 60 50 40 30 65 70 75 80 85 90 95 100 Meta-Verification TPR Figure 1: AdvancedMathBench exposes complementary weaknesses in advanced ma

FAQs

Why do existing math benchmarks fail to assess advanced mathematical proof capability?

They often emphasize final-answer correctness or coarse solution matching, which cannot reliably determine whether the reasoning steps form a mathematically valid proof.

What are the two main components of AdvancedMathBench?

ProverBench evaluates proof generation, while VerifierBench evaluates proof verification, including validity judgment and justification of verification.

How is proof quality evaluated for model-generated proofs?

A dedicated automatic verification pipeline is trained on large-scale expert annotations to output correctness verdicts, fine-grained error assessments, and error localization, aligning well with human expert evaluation.

Jul 16, 2026

Upload by Ophelia