[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-201209-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-201209-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","divlogiceval-a-framework-for-benchmarking-logical-reasoning-evaluation-in-large-language-models","DivLogicEval - A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models","","Logic reasoning in natural language is used as an important intelligence measure for large language models, yet popular benchmarks can mix multiple reasoning skills and produce unreliable results for the targeted logical reasoning ability. Existing logic benchmarks also suffer from limited language diversity and distribution mismatch, biasing evaluations. This paper proposes DivLogicEval, a diverse classical-logic benchmark with natural sentences, plus an evaluation metric that reduces bias and randomness, and reports experiments comparing different LLMs.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/divlogiceval-a-framework-for-benchmarking-logical-reasoning-evaluation-in-large-language-models/201209/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/divlogiceval-a-framework-for-benchmarking-logical-reasoning-evaluation-in-large-language-models/201209.png","ImageObject",300,407,{"name":42,"@type":43},"Ethan Miller","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-23","2026-09-04",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",14,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why do popular benchmarks fail to reliably measure logical reasoning ability in LLMs?","Question",{"text":63,"@type":64},"They often entangle multiple reasoning skills, so models may succeed for reasons other than logical reasoning. This can overestimate logical reasoning performance.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What problems does the paper identify with existing logic reasoning benchmarks?",{"text":68,"@type":64},"They have limited language diversity and their distributions deviate from an ideal benchmark distribution, leading to biased evaluation results.",{"name":70,"@type":61,"acceptedAnswer":71},"What is DivLogicEval and how does it improve evaluation?",{"text":72,"@type":64},"DivLogicEval is a classical-logic benchmark built from natural sentences with diverse statements arranged counterintuitively. The paper also introduces a metric designed to mitigate bias and randomness inherent in LLMs.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},201209,1788527239,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":145,"preview_page_count":4},687207017582,"https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d","DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models  \nTsz Ting Chung1 Lemao Liu2 Mo Yu3 Dit-Yan Yeung 1  \n1The Hong Kong University of Science and Technology  \n2Fudan University 3WeChat AI, Tencent  \n[ttchungac@connect.ust.hk](ttchungac@connect.ust.hk) [lemaoliu@gmail.com](lemaoliu@gmail.com)[ ](lemaoliu@gmail.com)[moyumyu@global.tencent.com](moyumyu@global.tencent.com) [dyyeung@cse.ust.hk](dyyeung@cse.ust.hk)  \nAbstract  \nLogic reasoning in natural language has been recognized as an important measure of human intelligence for Large Language Models (LLMs) . Popular benchmarks may entangle multiple reasoning skills and thus provide unfaithful evaluations on the logic reasoning skill.  \nMeanwhile, existing logic reasoning benchmarks are limited in language diversity and their distributions are deviated from the distribution of an ideal logic reasoning benchmark, which may lead to biased evaluation results.  \nThis paper thereby proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way. To ensure amore reliable evaluation, we also introduce anew evaluation metric that mitigates the influence of bias and randomness inherent in LLMs.  \nThrough experiments, we demonstrate the extent to which logical reasoning is required to answer the questions in DivLogicEval and compare the performance of different LLMs in conducting logical reasoning.  \n1 Introduction  \nLarge Language Models (LLMs) have been evaluated across diverse reasoning capabilities, including narrative reasoning (Yu et al., 2025a ; Karpinska et al., 2024), mathematical reasoning (Hendryckset al., 2021 ; Cobbe et al., 2021), inductive reasoning (Yu et al., 2025b ; Chollet, 2019), and logical reasoning (Saparov et al., 2023 ; Yu et al., 2020) . Among these, logical reasoning has long been regarded as a key indicator of human intelligence, frequently employed in contexts such as academic admissions, employment screening, and civil service recruitment. Many globally recognized standardized tests, such as the LSAT, GMAT, civil service examinations, and general aptitude tests, include a substantial number of logic-based questions.  \nAlthough some popular benchmarks such as ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2020)  \n\n|  | Diversity | Logic-centered |\n| --- | --- | --- |\n| ReClor | ✓ | ✗ |\n| LogiQA2 | ✓ | ✗ |\n| RuleTaker | ✗ | ✓ |\n| LogicNLI | ✗ | ✓ |\n| FOLIO | ✗ | ✓ |\n| RobustLR | ✗ | ✓ |\n| PrOntoQA-OOD | ✗ | ✓ |\n| \u003Cbr>DivLogicEval | \u003Cbr>✓ | \u003Cbr>✓ |\n\nTable 1: Comparison between the proposed DivLogicEval and existing logic reasoning benchmarks. DivLogicEval demonstrates great language diversity and provides a logic-centered evaluation that isolates the impact of logical reasoning from factors like commonsense reasoning and pretraining shortcuts.  \nexist for evaluating logical reasoning (Yu et al., 2020 ; Liu et al., 2020, 2023a ; Huang et al., 2023), they do not solely concentrate on logic reasoning. In fact, various reasoning (e.g., commonsense reasoning and logic reasoning) skills may entangle in these benchmarks and some other reasoning skills can make non-negligible contributions to solving the task. For example, as shown in experiments (see §3.1), by simply reducing the model’s ability to utilize logical reasoning through prompting, LLM surprisingly yields better performance on ReClor and LogiQA. Consequently, such benchmarks may overestimate the logic reasoning abilities of LLMs.  \nMeanwhile, some efforts have been made to create datasets based on classical logic (Hahn et al., 2021 ; Pi et al., 2022 ; Sanyal et al., 2022) . For example, FOLIO is equipped with human annotations of classical logic (Han et al., 2022) but it only contains about two hundred instances due to annotation cost. Other datasets are synthetically generated by predefined templates over classical logic expressions (Clark et al., 2020 ; Tian et al., 2021 ; Saparova","cbCaibGNE4ia5AXT","https://ap.wps.com/l/cbCaibGNE4ia5AXT","pdf",577943,15,"English","# Introduction\n## Logical reasoning as evaluation signal\n## Limitations of existing benchmarks\n## Proposed DivLogicEval and evaluation metric\n# Experiments\n## Required logical reasoning analysis\n## Comparison across LLMs\n# Benchmark construction\n## From symbolic logic to natural language","[{\"question\":\"Why do popular benchmarks fail to reliably measure logical reasoning ability in LLMs?\",\"answer\":\"They often entangle multiple reasoning skills, so models may succeed for reasons other than logical reasoning. This can overestimate logical reasoning performance.\"},{\"question\":\"What problems does the paper identify with existing logic reasoning benchmarks?\",\"answer\":\"They have limited language diversity and their distributions deviate from an ideal benchmark distribution, leading to biased evaluation results.\"},{\"question\":\"What is DivLogicEval and how does it improve evaluation?\",\"answer\":\"DivLogicEval is a classical-logic benchmark built from natural sentences with diverse statements arranged counterintuitively. The paper also introduces a metric designed to mitigate bias and randomness inherent in LLMs.\"}]","DivLogicEval - A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models | PDF",38]