[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83621-en":3,"doc-seo-83621-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83621,16904993612988,"Olivia Brown","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Towards a Phonology-Informed Evaluation of Multilingual TTS","Neural TTS can sound natural across languages, but that naturalness may hide whether a system preserves phonological sound contrasts tied to grammar. A classifier-based audit framework checks TTS outputs against language-specific phonological patterns using human speech as the benchmark. Tested on Assamese Advanced Tongue Root (ATR) vowel harmony with Meta’s MMS TTS, the approach shows transfer with minimal loss and exposes token-level faithfulness bias. Harmony is classified more accurately at the word level using predicted ATR labels than transcription labels, revealing a gap between intended and produced phonology.","Towards a Phonology-Informed Evaluation of Multilingual TTS  \nSneha Ray Barman  1 ,∗∗ , Neeraj Kumar Sharma  1 ,2 , Shakuntala Mahanta  1 ,3  \n1 Centre for Linguistic Science & Technology, IIT Guwahati  \n2 Mehta Family School for Data Science & Artificial Intelligence, IIT Guwahati  \n3 Department of Humanities & Social Sciences, IIT Guwahati  \n[sneha.barman, neerajs, smahanta]@ [iitg.ac.in](iitg.ac.in)  \narXiv :2607 .0 1965v 1 [ cs .CL] 2 Jul 2026  \nAbstract  \nNeural TTS systems can sound natural across languages, but naturalness does not guarantee the preservation of sound contrasts that distinguish words from their grammatical forms. Standard metrics like MOS do not test for this. We propose a classifier-based framework that audits TTS output against language-specific phonological patterns using human speech asa benchmark. Testing Assamese advanced tongue root (ATR) vowel harmony with Meta’s MMS TTS, we show that a classifier trained on human speech transfers to synthesized speech with minimal loss. The faithfulness audit reveals that [+ATR] mid vowels are realized as [-ATR] in 1/3 tokens despite an underlying [+ATR] specification, a bias absent in human speech. At the word level, predicted ATR labels classify harmony more accurately than transcription labels, indicating a gap between intended and produced phonology. The framework offers taskspecific diagnostics and generalizes to other phonological contrasts with measurable acoustic cues.  \nIndex Terms: TTS evaluation, vowel harmony, phonological faithfulness, cross-domain classification, Assamese  \n1. Background  \nNeural architectures and the massive scaling of training data have rapidly improved multilingual text-to-speech (TTS) systems. Standard evaluation metrics based on Mean Opinion Scores (MOS) focus on perceived naturalness and whether words are recoverable by automatic recognition. However, sounding natural does not guarantee that a system reproducesa language’s sound patterns, especially when grammatical context determines which sounds appear together. [1] showed that modern neural TTS has largely closed the perceived naturalness gap with human speech. This makes the problem more pressing: if a system scores well on MOS, systematic phonological errors may go entirely undetected.  \nThe dominant evaluation paradigm relies on human listening tests, most commonly MOS and MUSHRA-style protocols [2, 3, 4], as established through benchmarks such as the Blizzard Challenge [5] . These protocols provide global ratings rather than targeted evidence about whether a system preserves specific sound contrasts or grammatical alternations. Decontextualized listening tests assume a stable gold standard for speech quality that does not exist, and evaluation should instead be task-specific and diagnostic [6] . [7]’s review of MOS studies reveals further limitations, including listener variability, scale anchoring effects, and the collapse of multidimensional speech quality into a single uninformative scalar. [8] surveyed 133 recent TTS papers and identified that most do not report basic  \n**indicates the corresponding author.  \nmethodological details such as scale labels or increments, and showed experimentally that even small changes in test instructions can significantly alter scores.  \nAutomatic alternatives include mel cepstral distortion, spectral/F0 error, perceptual metrics such as PESQ [9] and STOI [10], neural MOS predictors [11, 12, 13, 14], and ASRbased intelligibility (WER/CER) . Speaker similarity metrics based on embedding comparison [15] extend evaluation to voice identity but do not address phonological structure. Work on Indian language TTS, including the IndicTTS corpus [16], and multilingual benchmarks such as Common Voice [17] have relied on MOS and WER throughout. While useful for tracking overall quality and consistency, none of these measures tests whether a system respects the phonological patterns that distinguish languages, varieties, or accents. Segment-lev","cbCaidM6gHXH9tAt","https://ap.wps.com/l/cbCaidM6gHXH9tAt","pdf",689201,1,5,"English","en",105,"# Background\n## Limits of standard TTS evaluation\n## Task-specific and diagnostic evaluation\n# Phonology-informed framework for evaluation\n## Assamese ATR vowel harmony as a test case\n## Acoustic cues and classifier-based auditing","[{\"question\":\"Why can standard TTS metrics like MOS miss phonological errors?\",\"answer\":\"Perceived naturalness and recognition recoverability do not guarantee preservation of phonological contrasts required by grammatical context. As a result, systematic phonological mistakes may remain undetected under global rating protocols.\"},{\"question\":\"What phonological phenomenon is used to test the proposed evaluation framework?\",\"answer\":\"Assamese Advanced Tongue Root (ATR) vowel harmony. [+ATR] suffix vowels trigger harmony on [-ATR] stem vowels, constraining which vowel qualities can co-occur within a word.\"},{\"question\":\"What does the faithfulness audit reveal about ATR realization in synthesized speech?\",\"answer\":\"The audit finds that [+ATR] mid vowels are realized as [-ATR] in about one third of tokens despite an underlying [+ATR] specification. This mismatch bias is not present in human speech.\"}]",1784189333,13,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"towards-a-phonology-informed-evaluation-of-multilingual-tts","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/towards-a-phonology-informed-evaluation-of-multilingual-tts/83621/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why can standard TTS metrics like MOS miss phonological errors?","Question",{"text":75,"@type":76},"Perceived naturalness and recognition recoverability do not guarantee preservation of phonological contrasts required by grammatical context. As a result, systematic phonological mistakes may remain undetected under global rating protocols.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What phonological phenomenon is used to test the proposed evaluation framework?",{"text":80,"@type":76},"Assamese Advanced Tongue Root (ATR) vowel harmony. [+ATR] suffix vowels trigger harmony on [-ATR] stem vowels, constraining which vowel qualities can co-occur within a word.",{"name":82,"@type":73,"acceptedAnswer":83},"What does the faithfulness audit reveal about ATR realization in synthesized speech?",{"text":84,"@type":76},"The audit finds that [+ATR] mid vowels are realized as [-ATR] in about one third of tokens despite an underlying [+ATR] specification. This mismatch bias is not present in human speech.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":21,"slug":137},19,"General","general"]