STITCH - Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models

Sep 5, 2026

Upload by Grenda

Spoken Language Models (SLMs) generate spoken responses from speech inputs, yet current systems cannot perform internal, unspoken thinking before speaking. Humans instead reason silently to improve clarity and conciseness. Direct chain-of-thought (CoT) generation enables thinking but adds uncontrolled latency because the spoken output must wait for the full text-CoT. STITCH addresses this by alternately generating unspoken reasoning chunks and spoken response chunks, using free audio playback time to continue reasoning, matching baseline latency and improving math reasoning by 15% while maintaining performance on non-reasoning datasets.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages27

Reading Time9 min

AccessFree to download

Introduction

Spoken language models and latency challenge

Chunked simultaneous thinking framework

Contributions and approach overview

arXiv :2507 . 15375v2 [ cs .CL] 9 Feb 2026 STITCH: SIMULTANEOUS THINKING AND TALKING WITH CHUNKED REASONING FOR SPOKEN LANGUAGE MODELS Cheng-Han Chiang1,2 ∗ Xiaofei Wang2† Linjie Li2 Chung-Ching Lin2 Kevin Lin2 Shujie Liu2 Zhendong Wang2 Zhengyuan Yang2 Hung-yi Lee 1 Lijuan Wang2 1National Taiwan University 2Microsoft ABSTRACT Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses. However, current SLMs lack the ability to perform an internal, unspoken thinking process before responding. In contrast, humans typically engage in complex mental reasoning internally, enabling them to communicate ideas clearly and concisely. Thus, integrating an unspoken thought process into SLMs is highly desirable. While naively generating a complete chain-of-thought (CoT) reasoning before starting to talk can enable thinking for SLMs, this induces additional latency for the speech response, as the CoT reasoning can be arbitrarily long. To solve this issue, we propose STITCH, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks. Since the audio duration of a chunk of spoken response is much longer than the time to generate the tokens in a chunk of spoken response, we use the remaining free time to generate the unspoken reasoning tokens. When a chunk of audio is played to the user, the model continues to generate the next unspoken reasoning chunk, achieving simultaneous thinking and talking. Remarkably, STITCH matches the latency of baselines that cannot generate unspoken CoT by design while outperforming those baselines by 15% on math reasoning datasets; STITCH also performs equally well on non-reasoning datasets as those baseline models. Some animations and demonstrations are on the project page: [https://d223302.github.io/](https://d223302.github.io/)[ ](https://d223302.github.io/)STITCH. 1 INTRODUCTION Spoken language models (SLMs) can take speech inputs and generate speech outputs (OpenAI, 2024b; Dfossez et al., 2024; Arora et al., 2025) . SLMs generate speech by emitting some speech tokens, discrete tokens that encode the information in speech (Dfossez et al., 2023; Zhang et al., 2024), which will be converted into an audio waveform by a speech decoder. Most SLMs are trained from a backbone text-based LLM (Dfossez et al., 2024; Ding et al., 2025); we can obtain an SLM by teaching the backbone LLM to understand speech input and generate the speech tokens as the output (Xiezhifei, 2024; Fang et al., 2025a) . Teaching the backbone LLM to directly generate speech tokens is challenging, as the LLM is originally trained to predict text tokens, so the speech tokens are completely new for the LLM backbone. To make SLMs better predict the speech tokens, current mainstream methods first generate the text tokens before generating the speech tokens (Chen et al., 2024; Zeng et al., 2024; Nguyen et al., 2025) . By design, those text tokens and the speech tokens are aligned, in the sense that the text tokens correspond to the semantic content of the speech tokens. While the aligned text-speech generation pipeline makes it easier to generate the speech response, these SLMs directly generate the output text-speech tokens without any inner thinking process, though explicitly modeling such reasoning steps has been shown helpful in text generation tasks (OpenAI, ∗[Work done during an internship at Microsoft GenAI.](Work done during an internship at Microsoft GenAI. dcml0714@gmail.com)[ dcml0714@gmail.com](Work done during an internship at Microsoft GenAI. dcml0714@gmail.com) †Correspondence: [xiaofei.wang@microsoft.com](xiaofei.wang@microsoft.com) User received speech timeline SLM token output timeline Receive user input tokens tokens tokens Figure 1: The timing diagram during generation for STITCH-R. The model first generates the first Nreason CoT reasoning tokens, Ntext text tokens, and Nspeech speech tokens. Once the first

FAQs

Why do current spoken language models have a latency problem when using chain-of-thought reasoning?

Generating a complete unspoken CoT before speaking forces the speech output to wait, creating uncontrollable latency because the CoT can be arbitrarily long.

How does STITCH achieve simultaneous thinking and talking?

STITCH alternates between unspoken reasoning chunks and spoken response chunks, and then uses the remaining free time during audio playback to generate the next reasoning chunk.

What performance and latency results does STITCH report?

STITCH matches the latency of baselines that do not generate unspoken CoT by design and outperforms them by 15% on math reasoning datasets while performing similarly on non-reasoning datasets.

Sep 5, 2026

Upload by Grenda