ISOSCI - A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs - Abstract

Sep 21, 2026

ISOSCI introduces isomorphic cross-domain science problem pairs to separate reasoning ability from domain knowledge retrieval in LLM evaluation. Each paired problem preserves identical logical and computational structure while requiring different domain-specific knowledge, enabling controlled attribution of reasoning-mode performance changes. Across five model pairs from four model families (8,408 evaluations), 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant, challenging structure-only reasoning improvement claims.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

Introduction

Related Work

ISOSCI: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs arXiv :2607 .0 143 1v 1 [ cs .CL] 1 Jul 2026 Samir Abdaljalil Electrical and Computer Engineering Texas A&M University College Station, TX USA Erchin Serpedin Electrical and Computer Engineering Texas A&M University College Station, TX USA Hasan Kurban∗ College of Science and Engineering Hamad Bin Khalifa University Doha, Qatar Abstract We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of reasoningmode gains. Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving. Reasoning toggles on highly capable models provide less than 5pp accuracy gain across all domains, and a reasoningspecialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2pp) underperforms on ISOSCI (−24.7pp), showing that benchmark choice determines conclusions about reasoning utility. We release ISOSCI at [https://huggingface.co/datasets/isosci/isosci](https://huggingface.co/datasets/isosci/isosci) 1 Introduction Recent advances in large language models have increasingly emphasized reasoning as a key driver of performance on complex tasks [28, 4, 12] . Techniques such as chain-of-thought prompting [26], reasoning-specific training [21, 24], and test-time compute scaling [19] have shown substantial gainson benchmarks such as GPQA [18], SciBench [23], and MMLU-STEM [8] . The problem is that these benchmarks conflate two distinct capabilities: retrieving the correct domainspecific knowledge, and applying the appropriate reasoning procedure over that knowledge. When a model fails a chemistry problem, it is unclear whether the failure reflects an inability to recall the relevant formula or an inability to execute the required reasoning steps. Without disentangling these factors, a basic question goes unanswered: do reasoning mechanisms improve reasoning itself, or do they primarily improve knowledge utilization? We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs designed to answer this question directly. Each problem is paired with a structurally identical counterpart from a ∗ Corresponding Author: [hkurban@hbku.edu.qa](hkurban@hbku.edu.qa) [Preprint.](Preprint.) different scientific domain: both require the same sequence of logical and computational steps, but depend on entirely different domain knowledge. If a model succeeds on one problem but fails on its isomorphic counterpart, the gap must be attributed to missing knowledge, not to reasoning ability. Using ISOSCI, we evaluate five model pairs across four model families, covering both traditional reasoning-vs-standard comparisons and toggle-based comparisons (same model, [reasoning on vs. off](reasoning on vs. off)). Across 8,408 evaluations spanning four scientific domains, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains across all five pairs; Wilson 95% CI [82.3%, 96.0%]), for short-horizon procedural science problems. Enabling reasoning has minimal effect on overall accuracy for high-capability models (below 5pp across domains), and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA (+19.2pp) underperforms on ISOSCI (−24.7pp), showing that benchmark choice determines conclusions about reasoning model utility. These findings suggest that reasoning mechanisms function primarily as ex

FAQs

What is ISOSCI and what problem does it address?

ISOSCI is a benchmark of isomorphic cross-domain science problem pairs designed to disentangle reasoning ability from domain knowledge retrieval in LLM evaluation.

How does ISOSCI ensure a fair comparison between reasoning and knowledge?

Each problem pair keeps the logical and computational steps identical while swapping the required domain-specific knowledge, so success or failure can be attributed to missing knowledge rather than reasoning structure.

What do the results show about reasoning-mode gains?

Across 8,408 evaluations, 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant, and reasoning toggles yield minimal overall accuracy gains for highly capable models.

Sep 21, 2026

Uploaded by Olivia Brown

ISOSCI - A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs - Abstract_Page 1

Open in WPS to view all 19 pages