ResearchQA 基于引用的科学论文问答基准评测

Jul 16, 2026

Upload by Quinn

Large language models support scientific reading, yet standard evaluations often miss whether answers are backed by verifiable citations. ResearchQA introduces a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers across eight domains and four question types. The benchmark targets citation-grounded evaluation, allowing multiple valid supporting passages and rewarding grounded refusal when a paper does not support an answer. Experiments on leading closed- and open-weight models use a deterministic citation matcher and an LLM rubric evaluator, and release the dataset and evaluation assets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Abstract

Introduction

Methods

Dataset construction

Evaluation setup and scoring

Metrics and robustness considerations

arXiv :2607 . 1 1074v 1 [ cs .CL] 13 Jul 2026 ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers Saba Imran Debanjum Singh Solanky Khoj Inc. May 4, 2026 Abstract Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed-and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency. We release the benchmark, evaluation harness, and evaluator prompt. Introduction Researchers increasingly rely on large language models to read and reason over scientific literature, where two properties matter more than fluency: (1) every claim should be traceable to verifiable evidence in the source, and (2) the answer should surface the complete, relevant context rather than a confident subset of it. Fabricated or misattributed citations propagate false assumptions and erode trust in the findings that build on them. Standard evaluation methods fail to measure these attributes: LLM evaluator scores reward plausible prose, and embedding similarity passes fabricated-but-on-topic text. We introduce ResearchQA, a benchmark that measures citation-grounded answering directly, pairing LLM-generated questions with a deterministic check that a cited passage actually appears in the paper it claims to. Information retrieval benchmarks have a long history, but the existing experiments did not satisfy our research constraints. They either target a narrower corpus, the different granularity of evidence, or omit the failure modes that matter most for a citation-grounded product. HotPotQA is grounded in Wikipedia and constructed for multi-hop reasoning; it is well-curated and widely used, but Wikipedia articles lack the dense numerical and methodological content—statistics, tables, methods sections, findings—that a “chat with this paper” feature is constantly asked to ground. We eliminated it for that reason. The closest prior art to our experiment is QASPER: a curated corpus of NLP papers with questions, answers, and supporting evidence extracted by human annotators. QASPER’s design choices—paper-level scope, evidence-anchored answers, multiple question types—directly inform ours. The two reasons it was not enough on its own are that it covers only NLP papers (we need a domain-diverse corpus to measure a general-purpose research dataset), and that its annotation pipeline used paid graduate-student labelers, which conflicted with the scale of the rows-per-paper density we wanted. The open question that motivatesa substantial part of this work is whether a strong frontier LLM, paired with a deterministic verification step against the source text, can stand in for human annotators well enough to extend QASPER’s design to a multi-domain dataset at low cost. Two methodology pitfalls shape the design that follows. LLM evaluator scoring suffers from rubric collapse (the evaluators concentrate at the endpoints of a 1–5 scale unless every level is anchored); citation evaluation suffers from a verbatim-vs-paraphrase tradeoff between substring match

FAQs

ResearchQA评测的核心目标是什么？

评测重点是答案是否有可验证的引文支撑，而不仅仅是语言是否流畅或内容是否看似合理。对无法由论文支持的答案，基准会奖励“带依据的拒答”。

ResearchQA包含哪些规模与覆盖范围？

基准包含6,211条单论文问答对，来源于494篇开放获取论文，覆盖八个领域与四类问题：lookup、comprehension、多跳以及对抗式问题。

评测如何判断模型回答是否被论文支持？

采用确定性的引用匹配器，核查声称引用的支持段落是否确实出现在原论文中；同时还使用LLM驱动的rubric评分器对输出进行评分。

Jul 16, 2026

Upload by Quinn