Towards Spoken Mathematical Reasoning - Benchmarking Speech-based Models over Multi-faceted Math Problems

Oct 5, 2026

Upload by jay

Large language models and multimodal LLMs show strong reasoning across many tasks, yet mathematical reasoning from spoken input is insufficiently studied. This work introduces Spoken Math Question Answering (Spoken-MQA), a benchmark evaluating speech-based models, including ASR+LLM cascades and end-to-end speech LLMs, on multi-faceted spoken math problems. Experiments cover pure arithmetic, single- and multi-step contextual reasoning, and knowledge-oriented math tasks. Results show difficulty on direct arithmetic, bias toward LaTeX symbols, and degraded math-knowledge reasoning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Abstract

Introduction

Spoken reasoning evaluation gap

Mathematical reasoning as a rigorous benchmark

Spoken-MQA benchmark design

Data construction pipeline

Experimental findings

Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems Chengwei Wei♢ , Bin Wang♢ , Jung-jae Kim ♢ , Nancy F. Chen♢ ,†♢Institute for Infocomm Research (I2R), A*STAR, Singapore †Centre for Frontier AI Research (CFAR), A*STAR, Singapore [wei_chengwei@i2r.a-star.edu.sg](wei_chengwei@i2r.a-star.edu.sg) arXiv :2505 . 15000v1 [ cs .CL] 21 May 2025 Abstract Recent advances in large language models (LLMs) and multimodal LLMs (MLLMs) have led to strong reasoning ability across a wide range of tasks. However, their ability to perform mathematical reasoning from spoken input remains underexplored. Prior studies on speech modality have mostly focused on factual speech understanding or simple audio reasoning tasks, providing limited insight into logical step-by-step reasoning, such as that required for mathematical problem solving. To address this gap, we introduce Spoken Math Question Answering (Spoken-MQA) 1 , a new benchmark designed to evaluate the mathematical reasoning capabilities of speech-based models, including both cascade models (ASR + LLMs) and end-to-end speech LLMs. Spoken-MQA covers a diverse set of math problems, including pure arithmetic, single-step and multi-step contextual reasoning, and knowledge-oriented reasoning problems, all presented in unambiguous natural spoken language. Through extensive experiments, we find that: (1) while some speech LLMs perform competitively on contextual reasoning tasks involving basic arithmetic, they still struggle with direct arithmetic problems; (2) current LLMs exhibit a strong bias toward symbolic mathematical expressions written in LaTex and have difficulty interpreting verbalized mathematical expressions; and (3) mathematical knowledge reasoning abilities are significantly degraded in current speech LLMs. 1 Introduction Recent advancements in large language models (LLMs) and multimodal large language models (MLLMs) have demonstrated impressive capabilities across various types of tasks, including question answering (Lewkowycz et al., 2022 ; Guo et al., 2023 ; Shao et al., 2024), code generation (Zheng et al., 2024 ; Wei et al., 2024 ; Liu et al., 2024), and 1 Our evaluation code and datasets will be open-sourced at [https://github.com/amao0o0/Spoken-MQA](https://github.com/amao0o0/Spoken-MQA) speech understanding (Gong et al., 2023 ; An et al., 2024 ; Huang et al., 2025) . Among this wide range of tasks, reasoning stands out as a core capability that underpins performance in complex downstream tasks. Consequently, considerable research has focused on evaluating and improving reasoning abilities across different modalities (Shao et al., 2024 ; Yang et al., 2024b ; Xu et al., 2024 ; Hurst et al., 2024 ; Thawakar et al., 2025) . While substantial progress has been made in evaluating reasoning in textual and visual domains, reasoning over spoken inputs remains comparatively underexplored. Initial studies (Ghosh et al., 2024 ; Sakshi et al., 2024) have begun probing the reasoning capabilities of speech LLMs, but mainly in the context of basic comprehension or factual question answering, offering limited insight into multi-step reasoning processes such as Chain-ofThought (CoT) reasoning. More recently, Xie et al.(2025) introduced Audio-Reasoner, a large-scale audio-language model trained on an audio QA pair dataset, with a specific focus on CoT-style reasoning over spoken inputs. These developments mark important progress, but significant gaps remain, particularly in the domain of mathematical reasoning from speech. Mathematical reasoning serves as a rigorous benchmark for evaluating model reasoning due to its reliance on logical step-by-step deduction, and sensitivity to fine-grained errors (Mishra et al., 2022 ; Lu et al., 2023 ; Zhang et al., 2024) . Yet, most existing multimodal evaluations of mathematical reasoning focus heavily on visual-text settings (Lu et al., 2023 ; Zhang et al., 2024 ; Wang et al., 2024

FAQs

What gap does Spoken-MQA target?

It addresses the lack of a comprehensive benchmark for evaluating mathematical reasoning from spoken input, especially multi-step logical problem solving rather than basic comprehension.

What types of problems are included in Spoken-MQA?

Spoken-MQA includes pure arithmetic, single-step and multi-step contextual reasoning, and questions requiring specialized mathematical domain knowledge, all presented as unambiguous spoken language.

What do the experiments reveal about current speech LLMs?

Some models are competitive on contextual reasoning with basic arithmetic, but they still struggle with direct arithmetic, show bias toward LaTeX-style symbolic expressions, and have significantly degraded math-knowledge reasoning.

Oct 5, 2026

Upload by jay