Information-seeking failures of large language models in agentic clinical reasoning

Q: What evaluation task does the agentic framework require from models?

Models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan in hematologic oncology.

Q: How does information utilization relate to diagnostic accuracy?

The fraction of available data the models actually request is the strongest predictor of diagnostic accuracy (R = 0.69, P < 0.001).

Q: What failure modes were identified in the models’ reasoning process?

Error analysis attributes dominant failures to search satisficing, anchoring, and premature closure.

Jul 16, 2026

Upload by Ezra

Large language models can score well on medical knowledge tests, yet clinical reasoning depends on choosing which information to obtain under uncertainty. This paper introduces an agentic evaluation framework in hematologic oncology where models proactively request data over three rounds before forming a diagnosis and treatment plan. Across 32 frontier models, top performance reaches 68% accuracy. Information utilization predicts accuracy, but utilization drops sharply in later rounds, leaving crucial molecular data unexamined.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages66

Reading Time2 h 46 min

Abstract

Agentic evaluation framework

Accuracy and information utilization

Reasoning traces and error modes

Introduction

Clinical decision-making under uncertainty

Limits of existing LLM evaluations

arXiv :2607 . 10275v 1 [ cs .AI] 11 Jul 2026 Information-seeking failures of large language models in agentic clinical reasoning Krischan Braitsch 1 , Laura K. Schmalbrock2,3 , Theresa Weltermann4 , Andrew F. Berdel5 , Isabella Miller6 , Kai Tran7 , Michael Heider8 , Sabrina Kraus9 , Florian Bassermann 1,10-12 , Jacqueline Lammert 13-15 , Sebastian Ziegelmayer 16 , Marcus Makowski 16 , Lisa C Adams 16* , Keno K Bressem 16,17* 1 Technical University of Munich, School of Medicine and Health, TUM University Hospital, Klinikum rechts der Isar, Department of Medicine III, Munich, Germany 2 Department of Hematology, Oncology and Cancer Immunology, Charité -Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany. 3 German Cancer Research Center (DKFZ), Heidelberg, Germany 4 Department of Medicine III, LMU University Hospital, Munich, Germany 5 Department of Medicine A, Hematology/Oncology, University Hospital Münster, Münster, Germany 6 Onkologie/Hämatologie im Elisenhof, Munich, Germany 7 Department of Hematology, Oncology and Palliative Care, Klinikum Traunstein, Traunstein, Germany 8 Department of Molecular and Cell Biology, University of California at Berkeley, Berkeley, CA, USA 9 Department of Internal Medicine II, University Hospital Würzburg, Würzburg, Germany 10 TranslaTUM, Center for Translational Cancer Research, Technical University of Munich, Munich, Germany 11 Deutsches Konsortium für Translationale Krebsforschung, Heidelberg, Germany 12 Bavarian Cancer Research Center, Munich, Germany 13 Department of Gynecology and Center for Hereditary Breast and Ovarian Cancer, Technical University of Munich (TUM), School of Medicine and Health, Klinikum rechts der Isar, TUM University Hospital, Munich, Germany 14 Institute of Artificial Intelligence and Informatics in Medicine (AIIM), TUM University Hospital, Technical University of Munich (TUM), Munich 81675, Germany 15 German Cancer Consortium (DKTK), partner site Munich, a partnership between DKFZ and TUM University Hospital, Munich, Germany 16 Technical University of Munich, School of Medicine and Health, TUM University Hospital, Klinikum rechts der Isar, Department of Diagnostic and Interventional Radiology, Germany 17 Technical University of Munich, School of Medicine and Health, TUM University Hospital, German Heart Center, Department of Cardiovascular Radiology and Nuclear Medicine, Germany Corresponding author: Keno K. Bressem, [MD.](MD. Keno.bressem@tum.de)[ Keno.bressem@tum.de](MD. Keno.bressem@tum.de) Abstract Large language models achieve high scores on medical knowledge assessments, yet clinical reasoning requires actively deciding what to investigate under uncertainty. We developed an agentic evaluation framework in hematologic oncology in which models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan. Across 32 frontier models, the best achieved only 68% overall accuracy. Information utilization-the fraction of available data actually requested-was the strongest predictor of diagnostic accuracy (R = 0.69, P < 0.001), yet utilization collapsed from 57% to 26% in the final round, leaving molecular and cytogenetic data critical for treatment selection unexamined. Reasoning traces scored high on a clinical reasoning rubric (91% above threshold) but decorrelated from accuracy, revealing a gap between locally coherent rationales and globally correct conclusions. Error analysis identified search satisficing, anchoring and premature closure as the dominant failure modes-the same cognitive biases that characterize novice clinicians under dual-process models of diagnostic reasoning. These findings demonstrate that the primary limitation of current models in clinical oncology is not insufficient medical knowledge but a systematic failure of information-seeking under uncertainty. Introduction

FAQs

What evaluation task does the agentic framework require from models?

Models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan in hematologic oncology.

How does information utilization relate to diagnostic accuracy?

The fraction of available data the models actually request is the strongest predictor of diagnostic accuracy (R = 0.69, P < 0.001).

What failure modes were identified in the models’ reasoning process?

Error analysis attributes dominant failures to search satisficing, anchoring, and premature closure.

Jul 16, 2026

Upload by Ezra