[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83941-en":3,"doc-seo-83941-105":28,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":11,"language":21,"language_code":22,"site_id":23,"html_lang":22,"table_of_contents":24,"faqs":25,"seo_title":13,"seo_description":14,"update_tm":26,"read_time":27},83941,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","SPEARBench：用于流式语音到语音语言模型自然性评估的基准","Streaming speech-to-speech language models aim to respond to spoken queries with synthetic speech, yet conventional speech and text benchmarks miss how conversational naturalness is perceived. SPEARBench evaluates naturalness from question–answer interactions by constructing controlled dialogue prompts, running multiple models, and scoring outputs with a multidimensional protocol. Metrics cover response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines. Human answers serve as a reference condition, and results compare several contemporary systems.","SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech  \nLanguage Models  \nThomas Thebaud∗ , Yuzhe Wang∗ , Hao Zhang∗ , Sathvik Manikantan Napa Ugandhar∗ , Ashish Hallur∗ , Georgi Tinchev†, Venkatesh Ravichandran†, Laureano Moro-Velazquez∗  \n∗ Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA  \nEmail: [tthebau1@jhu.edu](tthebau1@jhu.edu)  \n†Amazon Inc., Seattle, WA, USA  \narXiv :2607 .05365v 1 [ cs .CL] 6 Jul 2026  \nAbstract—Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. We introduce SPEARBench, a benchmark for evaluating naturalness in speech-to-speech language models from question-answer interactions. SPEARBench constructs controlled dialogue prompts from the Seamless Interaction corpus, runs inference across multiple models, and evaluates generated answers using a multidimensional protocol that covers response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines. The benchmark includes original human answers as a reference condition and reports results for several contemporary models. Results show that current models can achieve high signal-level quality and low ASR error while still differing from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.  \nIndex Terms—speech-to-speech language models, spoken dialogue systems, naturalness evaluation, turn-taking, conversational speech, benchmark  \nI. INTRODUCTION  \nLarge language models have recently become central to natural language processing, human-computer interaction, and artificial intelligence more broadly [1] . Their success has motivated spoken language models that extend text-based interaction to speech. In this paper, we focus on speechto-speech (S2S) language models, defined as systems that take speech as input and produce speech as output, enabling interaction without an explicit text interface [2] . Some recent systems operate in streaming mode, producing answers with low enough latency for real-time interaction, and some are full-duplex systems, meaning that they can process incoming speech while simultaneously producing outgoing speech [3] . Together, these capabilities make natural spoken conversation with a model increasingly realistic. However, many current systems still rely on encoding, processing, and decoding pipelines that can remove paralinguistic cues or limit synthesized-speech flexibility, reducing conversational  \nnaturalness. Human conversational naturalness is shaped by coupled phenomena including response timing, pauses, overlap, interruptions, prosody, affect, stance, language choice, dialect consistency, context, and speaker relationship [4], [5] . A S2S model that produces intelligible, high-quality audio may still feel unnatural if it responds too quickly, interrupts the user, shifts dialect unexpectedly, flattens emotional dynamics, or answers with an inappropriate voice.  \nExisting spoken dialogue and speech generation evaluations provide valuable tools for some aspects of naturalness evaluation [6], [7] . However, most existing benchmarks either require human annotators, which makes evaluation costly, or focus on one aspect of the conversation, such as latency [6], or interruptions [8] . We present SPEARBench, a benchmark for naturalness evaluation in conversational S2S systems, and an online platform for sharing model performance across a unified set of metrics. SPEARBench is built on two-speaker question-answer evaluation clips from the Se","cbCaid4CGP55rNp9","https://ap.wps.com/l/cbCaid4CGP55rNp9","pdf",4495480,1,"English","en",105,"# Introduction\n# Related Work\n## Streaming Speech-to-Speech Language Models","[{\"question\":\"SPEARBench主要评估流式语音到语音模型的哪些自然性维度？\",\"answer\":\"SPEARBench通过多维评测协议覆盖响应延迟、打断、语音质量、ASR鲁棒性、语言与方言一致性、情绪自然性、交际立场以及可解释的分布式基线等指标。\"},{\"question\":\"SPEARBench的数据与评测流程如何构建问答对话？\",\"answer\":\"基于Seamless Interaction数据集的双人问答评价片段提取情境：一位说话者提问，另一位说话者作答。随后将原始人类答案与流式S2S模型生成答案进行对比，并计算多类声学、文本、时间、语言学、情绪与立场等指标。\"},{\"question\":\"基准结果表明当前模型在什么方面仍与人类对话存在差异？\",\"answer\":\"结果显示模型可达到较高的信号层质量并保持较低ASR错误，但在延迟、重叠/插入行为、方言保留、情绪适配以及交际立场的动态变化方面仍与人类会话行为不同。\"}]",1784191576,20,{"code":4,"msg":29,"data":30},"ok",{"site_id":23,"language":22,"slug":31,"title":13,"keywords":32,"description":14,"schema_data":33,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":26},"spearbench-a-benchmark-for-naturalness-evaluation-in-streaming-speech-to-speech-language-models","",{"@graph":34,"@context":84},[35,52,67],{"@type":36,"itemListElement":37},"BreadcrumbList",[38,42,46,49],{"item":39,"name":40,"@type":41,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":43,"name":44,"@type":41,"position":45},"https://docshare.wps.com/document/","Document",2,{"item":47,"name":12,"@type":41,"position":48},"https://docshare.wps.com/document/research-report/",3,{"item":50,"name":13,"@type":41,"position":51},"https://docshare.wps.com/document/spearbench-a-benchmark-for-naturalness-evaluation-in-streaming-speech-to-speech-language-models/83941/",4,{"url":50,"name":13,"@type":53,"author":54,"headline":13,"publisher":56,"fileFormat":59,"inLanguage":22,"description":14,"dateModified":60,"datePublished":61,"encodingFormat":59,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":55},"Person",{"url":39,"name":57,"@type":58},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"SPEARBench主要评估流式语音到语音模型的哪些自然性维度？","Question",{"text":74,"@type":75},"SPEARBench通过多维评测协议覆盖响应延迟、打断、语音质量、ASR鲁棒性、语言与方言一致性、情绪自然性、交际立场以及可解释的分布式基线等指标。","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"SPEARBench的数据与评测流程如何构建问答对话？",{"text":79,"@type":75},"基于Seamless Interaction数据集的双人问答评价片段提取情境：一位说话者提问，另一位说话者作答。随后将原始人类答案与流式S2S模型生成答案进行对比，并计算多类声学、文本、时间、语言学、情绪与立场等指标。",{"name":81,"@type":72,"acceptedAnswer":82},"基准结果表明当前模型在什么方面仍与人类对话存在差异？",{"text":83,"@type":75},"结果显示模型可达到较高的信号层质量并保持较低ASR错误，但在延迟、重叠/插入行为、方言保留、情绪适配以及交际立场的动态变化方面仍与人类会话行为不同。","https://schema.org",{"og:url":50,"og:type":86,"og:title":13,"og:site_name":57,"og:description":14},"article",{"robots":88,"canonical":50},"index,follow",{"doc_id":7,"site_id":23},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":20,"doc_module":4,"doc_module_name":44,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":45,"doc_module":4,"doc_module_name":44,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":51,"doc_module":4,"doc_module_name":44,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":44,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":44,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":44,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":44,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":44,"category_name":124,"show_sort_weight":27,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":27,"doc_module":4,"doc_module_name":44,"category_name":127,"show_sort_weight":27,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":44,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":44,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]