[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83839-en":3,"doc-seo-83839-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83839,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","FormalRx Rectify and eXamine Semantic Failures in Autoformalization","Autoformalization requires reliable semantic alignment to support formal mathematical reasoning, yet existing evaluations often provide only opaque binary verdicts or single scalar scores, giving little insight into where translations fail or why. FormalRx introduces a comprehensive diagnostic evaluation framework that converts black-box assessment into actionable feedback. It is built on Sci Error Taxonomy, a hierarchical scheme covering 28 prioritized error categories. Using FormalRx-8B and the released FormalRx-Test benchmark, the approach delivers interpretable verdicts, categorization, localization, and correction, outperforming general LLMs and specialized baselines.","arXiv :2607 .04655v 1 [ cs .CL] 6 Jul 2026  \nFormalRx: Rectify and eXamine Semantic Failures in Autoformalization  \nHaocheng Wang∗1,4, Baiyu Huang∗ 1 , Yingjia Wan∗ 2 , Xiao Zhu1 , Xiaoyang Liu5 , Yinya Huang†3,4 and Zhijiang Guo†1,6  \n1 LARK Lab, HKUST(GZ), 2UCLA, 3 ETH AI Center, 4 ETH Zurich, 5 SJTU, 6 HKUST  \n∗ Equal Contribution †Corresponding Author  \nGithub Page: [https://lark-ai-lab.github.io/formalrx/](https://lark-ai-lab.github.io/formalrx/)  \nBenchmark: [https://huggingface.co/datasets/LARK-Lab/FormalRx-Test](https://huggingface.co/datasets/LARK-Lab/FormalRx-Test)  \nModel Weight: [https://huggingface.co](https://huggingface.co)  \n~~ ~~ Abstract ~~ ~~  \nThe veracious semantic alignment in autoformalization is significant for formal mathematical reasoning. However, existing evaluations provide only opaque binary verdicts or scalar scores, offering no interpretable insight into where or why translations fail. This opacity severely limits both human understanding and automated system improvement. To bridge this gap, we introduce FormalRx, a comprehensive diagnostic evaluation framework that transforms autoformalization assessment from black-box judgments into actionable feedback. At its core is Sci Error Taxonomy, a hierarchical classification scheme decomposing autoformalization errors into 28 distinct categories with strict priority ordering. Building on this taxonomy, FormalRx provides four critical diagnostic capabilities: alignment verdicts, error categorization, error localization, and correction. We instantiate the framework with a diagnostic model FormalRx-8B, trained on 56,287 NL–FL pairs with fine-grained diagnostic annotations, and release FormalRx-Test as the first fine-grained diagnostic benchmark. FormalRx-8B achieves F1-scores of 0.88 (verdict) and 0.71 (categorization), along with accuracies of 0.75 (localization) and 0.73 (correction), substantially outperforming both general-purpose LLMs and specialized baselines. By connecting evaluation with actionable insights, FormalRx enables systematic diagnosis and improvement of autoformalization systems.  \n1. Introduction  \nFormal mathematical reasoning has emerged as a new frontier in Artificial Intelligence, offering a path to rigorous and verifiable logical inference (Yang et al., 2024) . Within this paradigm, proof assistants such as Lean (de Moura and Ullrich, 2021; de Moura et al., 2015) have become pivotal. By grounding informal reasoning in strict formal systems, autoformalization translates mathematical problems from natural language into formal statements (Wu et al., 2022), serving as a critical bridge to mitigate the reasoning unfaithfulness and hallucinations (Wang et al., 2025b) frequently observed in Large Language Models (LLMs) . However, the development of effective metrics and systematic evaluation frameworks to characterize translation failures remains notably limited (Weng et al., 2025; Yang et al., 2024) .  \nExisting approaches to autoformalization evaluation are fundamentally limited by a lack of interpretability, hindering systematic error diagnosis and model improvement (Table 1) . Early work relied on syntactic validity checks (Leanprover Community, 2024) and surface-level metrics such as BLEU (Papineni et al., 2002) or compiler type-check feedback (Azerbayev et al., 2023) . Subsequent evaluations introduced rule-based equivalence checks (Jana et al., 2025; Liu et al., 2025b) and  \nCorresponding author(s): Zhijiang Guo ([zhijiangguo@hkust-gz.edu.cn](zhijiangguo@hkust-gz.edu.cn)), Yinya Huang ([yinya.huang@ai.ethz.ch](yinya.huang@ai.ethz.ch)) 1  \nFigure 1 | An illustrative example of FormalRx compared with other semantic evaluation methods for autoformalization. On this misaligned candidate, all FL-FL and NL-FL baselines fail, scoring it as aligned or confirming only syntactic correctness, whereas FormalRx returns the verdict, error category, error location, and a corrected statement.  \nstructural metrics (Liu et al., 2025c) to compare a","cbCaidNWw0pgincQ","https://ap.wps.com/l/cbCaidNWw0pgincQ","pdf",4023149,2,1,44,"English","en",105,"# Introduction\n## Sci Error Taxonomy\n## FormalRx Dataset & FormalRx-Test\n## FormalRx-8B","[{\"question\":\"为什么现有的自动形式化评估方法缺乏有效的诊断价值？\",\"answer\":\"现有评估通常只给出不透明的二元对齐判定或单一分数，缺少可解释的信号，难以定位失败原因与错误位置，因此难以支持系统调试与持续改进。\"},{\"question\":\"Sci Error Taxonomy 在 FormalRx 中扮演什么角色？\",\"answer\":\"Sci Error Taxonomy 将自动形式化的语义失配系统化拆解为 28 类错误，并采用严格的优先级层级结构来避免错误归因的歧义，从而提供可操作的诊断基础。\"},{\"question\":\"FormalRx-8B 如何进行诊断评估并输出结果？\",\"answer\":\"FormalRx-8B 在一次前向计算中联合完成四项任务：对齐判定、错误分类、错误定位与错误修正。实验结果显示其在这些维度上优于通用 LLM 与专用基线。\"}]",1784190903,111,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"formalrx-rectify-and-examine-semantic-failures-in-autoformalization","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/formalrx-rectify-and-examine-semantic-failures-in-autoformalization/83839/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么现有的自动形式化评估方法缺乏有效的诊断价值？","Question",{"text":75,"@type":76},"现有评估通常只给出不透明的二元对齐判定或单一分数，缺少可解释的信号，难以定位失败原因与错误位置，因此难以支持系统调试与持续改进。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Sci Error Taxonomy 在 FormalRx 中扮演什么角色？",{"text":80,"@type":76},"Sci Error Taxonomy 将自动形式化的语义失配系统化拆解为 28 类错误，并采用严格的优先级层级结构来避免错误归因的歧义，从而提供可操作的诊断基础。",{"name":82,"@type":73,"acceptedAnswer":83},"FormalRx-8B 如何进行诊断评估并输出结果？",{"text":84,"@type":76},"FormalRx-8B 在一次前向计算中联合完成四项任务：对齐判定、错误分类、错误定位与错误修正。实验结果显示其在这些维度上优于通用 LLM 与专用基线。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]