[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273654-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273654-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","hear2act-benchmarking-when-prosody-should-change-what-an-assistant-does","HEAR2ACT - Benchmarking When Prosody Should Change What an Assistant Does","","Prosodic cues can carry task-relevant information that redirects the trajectory and outcomes of task-oriented dialogue, even when wording stays unchanged. Existing benchmarks often test prosody perception or response quality separately, which makes it hard to measure whether prosodic evidence changes downstream decisions. HEAR2ACT provides a unified evaluation for text and spoken assistants using 480 persona-grounded scenarios with verifiable outcomes. For fixed tasks and user needs, concern delivery varies between explicit words and prosody-first signals, and evaluations compare transcript, audio, and concern-state access.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/hear2act-benchmarking-when-prosody-should-change-what-an-assistant-does/273654/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/hear2act-benchmarking-when-prosody-should-change-what-an-assistant-does/273654.png","ImageObject",300,407,{"name":42,"@type":43},"Miles","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-10","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",16,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does HEAR2ACT address in evaluating task-oriented assistants?","Question",{"text":63,"@type":64},"It evaluates whether prosodic evidence actually changes downstream task decisions, not just whether prosody is perceived or whether responses sound appropriate.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does HEAR2ACT vary how concerns are communicated to the assistant?",{"text":68,"@type":64},"It keeps the task and user needs fixed while changing whether the same hidden concern is conveyed explicitly in words or primarily through prosody, and it tests different assistant access conditions (transcript, audio, and concern state).",{"name":70,"@type":61,"acceptedAnswer":71},"What do the results show about the impact of audio and prosody on decision quality?",{"text":72,"@type":64},"Under prosody-mediated feedback, adding audio changes the optimal-solution rate only slightly, but when models infer concern status from audio and use it for next-action selection, the rate rises substantially, approaching ground-truth performance.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273654,1789472382,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":145,"preview_page_count":4},13056703019404,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does  \nXinyi Liu1,2 * , Hooshang Nayyeri1 , Dilek Hakkani-Tur1,2 ,  \nEmine Yilmaz1,3 , Joo-Kyung Kim1 , Yifei Zhang1 ,  \nCharith Peris1 , Hari Thadakamalla1  \n1Amazon 2University of Illinois Urbana-Champaign 3University College London  \n{liu323, [dilek}@illinois.edu](dilek}@illinois.edu)  \n{hooshang, jookyk, jimmyzyf, perisc, [thadakah}@amazon.com](thadakah}@amazon.com)  \n[eminey@amazon.co.uk](eminey@amazon.co.uk)  \narXiv : 2608 . 195 15v2 [cs .CL] 2 1 Aug 2026  \nAbstract  \nProsodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions. We introduce HEAR2ACT, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes. For each scenario, we keep the task and user needs fixed while varying whether the same concern is conveyed explicitly in words or primarily through prosody, and evaluate decisions under transcript, audio, and concern-state access.  \nUsing HEAR2ACT, we evaluate two audiocapable LLMs. Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15 .3% . In contrast, when models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state. This contrast, however, largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance. Together, these results show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do not reliably carry it into action without an explicit intermediate representation.  \n1 Introduction  \nTask-oriented dialogue (TOD) systems guide users toward concrete decisions through multiturn interaction (Budzianowski et al., 2018 ; Rastogi et al., 2020 ; Si et al., 2023) . In spoken TOD,  \n*Work done during an internship at Amazon.  \nthese decisions may depend on information beyond the transcript. A hesitant “Okay, sounds good” may indicate that an important concern remains unresolved, while the same words delivered brightly may signal genuine acceptance. This distinction can determine whether the assistant continues gathering information or commits prematurely to an unsuitable solution. The resulting evaluation problem is to determine when prosodic evidence should guide information gathering and final selection, and how effectively current assistants use it. The central question is therefore not only whether a model perceives a prosodic cue, but when that cue provides information beyond the words and whether the model carries it into subsequent task decisions.  \nExisting benchmarks address complementary parts of this problem, but not the full link from prosodic evidence to task action. Task-oriented dialogue benchmarks support multi-turn interaction and verifiable success, but do not isolate how prosodic feedback changes sequential decisions (Budzianowski et al., 2018 ; Rastogi et al., 2020 ; Si et al., 2023) . Paralinguistic benchmarks preserve vocal cues, but primarily evaluate perception, response appropriateness, or emotional interaction without task-grounded hidden needs and verifiable outcomes (Ao et al., 2024 ; Gao et al., 2025 ; Deng et al., 2025 ; Wang et al., 2026b) . StyleTalk and ParaS2S control lexical content, but evaluate the appropriate next response rather than the resulting multi-turn task trajectory (Lin et al., 2024 ; Yang et al., 2026) . To our knowledge, no existing protocol traces prosodic evidence through multi-turn task de","cbCaihRccjhW6M3C","https://ap.wps.com/l/cbCaihRccjhW6M3C","pdf",1416082,21,"English","# Abstract\n# Introduction\n## Motivation and central question\n## Related work and gaps\n## HEAR2ACT overview","[{\"question\":\"What problem does HEAR2ACT address in evaluating task-oriented assistants?\",\"answer\":\"It evaluates whether prosodic evidence actually changes downstream task decisions, not just whether prosody is perceived or whether responses sound appropriate.\"},{\"question\":\"How does HEAR2ACT vary how concerns are communicated to the assistant?\",\"answer\":\"It keeps the task and user needs fixed while changing whether the same hidden concern is conveyed explicitly in words or primarily through prosody, and it tests different assistant access conditions (transcript, audio, and concern state).\"},{\"question\":\"What do the results show about the impact of audio and prosody on decision quality?\",\"answer\":\"Under prosody-mediated feedback, adding audio changes the optimal-solution rate only slightly, but when models infer concern status from audio and use it for next-action selection, the rate rises substantially, approaching ground-truth performance.\"}]","HEAR2ACT - Benchmarking When Prosody Should Change What an Assistant Does | PDF",53]