CRiT-QA：使用反事实推理链与干扰陷阱评估多跳推理

Jul 16, 2026

Upload by Miles

Evaluating large language models’ multi-hop reasoning remains challenging: strong results on existing QA benchmarks can hide vulnerabilities to two key issues—(1) dependence on internal parametric knowledge rather than the given context, and (2) exploitation of dataset shortcuts such as single-document cues or type matching. CRiT-QA (Counterfactual Reasoning with Traps) addresses both by converting factual reasoning chains with counterfactual entities and inserting multi-anchor distractor chains. Experiments show significant performance drops versus standard datasets, making CRiT-QA a rigorous diagnostic for evidence-grounded reasoning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Abstract

Introduction

Retrieval-Augmented Generation and Multi-hop QA

Vulnerabilities: internal memory and dataset shortcuts

CRiT-QA design: counterfactual chains and distractor traps

arXiv :2607 . 10562v 1 [ cs .AI] 12 Jul 2026 CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps JungMin Yun 1* , JuneHyoung Kwon 1* , YoungBin Kim 1, 2 1 Department of Artificial Intelligence, Chung-Ang University 2 Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University {cocoro357, dirchdmltnv, [ybkim85}@cau.ac.kr](ybkim85}@cau.ac.kr) Abstract Evaluating the multi-hop reasoning capabilities of large language models remains a significant challenge. Although current models achieve strong results on existing multi-hop question answering datasets, such performance often masks two critical vulnerabilities: (1) reliance on internal parametric knowledge rather than adherence to the provided context, and (2) exploitation of dataset shortcuts, such as single-document cues or type-matching, that diminish the need for genuine evidence aggregation across multiple documents. We introduce CRiT-QA (Counterfactual Reasoning with Traps), a dataset explicitly designed to address both limitations. To neutralize reliance on memorized knowledge and enforce strict context dependency, CRiT-QA transforms factual reasoning chains with counterfactual entities. Furthermore, it injects multi-anchor distractor chains, plausible but incorrect reasoning paths that diverge at different hops. These traps require models to follow the entire reasoning process rather than exploiting shallow heuristics. Our experiments show that LLMs exhibit substantial performance degradation on CRiT-QA compared to standard datasets, exposing their vulnerability to counterfactual conditions and distractor traps. CRiT-QA thus serves as a rigorous diagnostic tool for evaluating genuine multi-hop reasoning and provides a foundation for developing more reliable, evidence-grounded LLMs. Keywords: Question Answering, Multi-Hop Reasoning, Large Language Models 1. Introduction Retrieval-Augmented Generation (RAG) has become a dominant approach for enhancing large language models (LLMs) by grounding outputsin external knowledge sources rather than relying solely on internal parametric memory (Lewis et al. , 2020 ; Fan et al. , 2024) . This paradigm has demonstrated promising improvements in factuality and adaptability across diverse domains (Tang and Yang, 2024 ; Gao et al. , 2024) . The effectiveness of RAG systems, however, critically depends on the model’s ability to identify, link, and aggregate multiple pieces of evidence distributed across different documents (Fang et al. , 2024 ; Suryawanshi et al. , 2025) . Multi-hop Question Answering (QA) has thus become a standard task for evaluating the higherorder reasoning capabilities of LLMs. In principle, such datasets are designed to assess whether models can identify and synthesize intermediate evidence to derive logically coherent final answers (Kim et al. , 2024 ; Li et al. , 2024 ; Liu et al. , 2025) . However, recent studies have revealed that the strong performance of LLMs on existing multi-hop QA datasets does not necessarily reflect genuine reasoning ability (Wu et al. , 2024a; Jiang et al. , 2024 ; Parmar et al. , 2024) . Instead, models often exploit artifacts in the dataset or rely on shallow surface-level patterns, thereby inflat- *Equal contribution. | (A) | Question: Who founded the company that distributed the film UHF? ——without any context—— LLM-generated Answer: Mike Medavoy ✓ | | --- | --- | | (B) | Question: In which county is Mark Dismore’s birthplace located? Paragraph 1: Greenfield is a city in and the county seat of Hancock County, Indiana, United States, and a part of the Indianapolis metropolitan area.(...) LLM-generated Answer: Hancock County ✓ Sub-question: What is the place of birth of Mark Dismore? LLM-generated Answer: Unanswerable | Table 1: Examples illustrating evaluation vulnerabilities in multi-hop QA: (A) an LLM answering correctly without context, and (B) a model exploiting single-para

FAQs

CRiT-QA主要用来评估什么能力？

CRiT-QA用于严格评估大语言模型的多跳推理能力，重点检验模型是否真正完成证据聚合与推理过程，而不是依赖捷径。

为什么现有多跳QA数据集的评估结果可能不可信？

文中指出，模型可能依赖内部记忆而不遵循给定上下文；同时也可能利用数据集中的捷径，如单文档线索或类型匹配，从而跳过跨文档的真实推理。

CRiT-QA通过哪些机制提升评测的严谨性？

CRiT-QA用反事实实体转换推理链以抑制记忆依赖，并注入多锚点干扰推理链，要求模型在不同hop上都遵循完整推理路径。

Jul 16, 2026

Upload by Miles