Asking the Right Questions - Benchmarking Large Language Models in the Development of Clinical Consultation Templates

Sep 11, 2026

This study evaluates the capacity of large language models (LLMs) to generate structured clinical consultation templates for electronic consultation. Using 145 expert-crafted templates developed and routinely used by Stanford’s eConsult team, frontier models are assessed on producing clinically coherent, concise, and prioritized question schemas. A multi-agent pipeline with prompt optimization, semantic autograding, and prioritization analysis shows strong comprehensiveness for some models but persistent overlong outputs and misprioritization under length constraints, with larger degradation in narrative-driven specialties like psychiatry and pain medicine. Findings support improved structured exchange while motivating stronger evaluation for real-time physician communication, noting limitations from template specificity and concordance-based grading.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages17

Reading Time6 min

AccessFree to download

1. Introduction

Consultation structure and communication risks

Synchronous vs. asynchronous e-consult workflows

Reliance on initial clinician-provided information

Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates Liam G. McCoy† Division of Neurology, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Email: [lmccoy@ualberta.ca](lmccoy@ualberta.ca) Fateme Nateghi Haredasht† and Kanav Chopra Stanford Center for Biomedical Informatics Research Stanford, CA, USA [Email: fnateghi@stanford.edu](Email: fnateghi@stanford.edu); [kanav@stanford.edu](kanav@stanford.edu) David Wu Department of Dermatology, Mass General Brigham, Harvard Medical School, Boston, MA, USA [Email: dwu@mgh.harvard.edu](Email: dwu@mgh.harvard.edu) David JH Wu and Abass Conteh Department of Radiation Oncology, Stanford Cancer Center, Palo Alto, CA, USA [Email: davidjwu@stanford.edu](Email: davidjwu@stanford.edu); [aconteh@stanford.edu](aconteh@stanford.edu) Sarita Khemani and Saloni Kumar Maharaj Division of Hospital Medicine, Stanford University School of Medicine, Stanford, CA, USA Email: [skhemani@stanford.edu](skhemani@stanford.edu); [salonikumar@stanford.edu](salonikumar@stanford.edu) Arth Pahwa Division of Neurology, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada [Email: apahwa@ualberta.ca](Email: apahwa@ualberta.ca) Vishnu Ravi Stanford Mussallem Center for Biodesign, Stanford University; Stanford University School of Medicine, Stanford, CA, USA Email: [vishnur@stanford.edu](vishnur@stanford.edu) Yingjie Weng Quantitative Sciences Unit, Stanford University School of Medicine, Stanford, CA, USA [Email: yweng7@stanford.edu](Email: yweng7@stanford.edu) Leah Rosengaus and Lena Giang Stanford Health Care, Stanford Medicine, Palo Alto, CA, USA [Email: LRosengaus@stanfordhealthcare.org](Email: LRosengaus@stanfordhealthcare.org); [LGiang@stanfordhealthcare.org](LGiang@stanfordhealthcare.org) Kelvin Zhenghao Li Department of Ophthalmology, Stanford Byers Eye Institute; Department of Ophthalmology, Tan Tock Seng Hospital; Centre of AI in Medicine, Lee Kong Chian School of Medicine, Nanyang Technological University Palo Alto, CA, USA; Singapore, Singapore [Email: kelvinli@stanford.edu](Email: kelvinli@stanford.edu) Olivia Jee Division of Primary Care and Population Health, Stanford University School of Medicine, Palo Alto, CA, USA [Email: ojee@stanford.edu](Email: ojee@stanford.edu) Daniel Shirvani Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada [Email: dshirv@student.ubc.ca](Email: dshirv@student.ubc.ca) Ethan Goh* Stanford Center for Biomedical Informatics Research Clinical Excellence Research Center, Stanford School of Medicine, Stanford, CA, USA Email: [ethangoh@stanford.edu](ethangoh@stanford.edu) Jonathan H. Chen* Stanford Center for Biomedical Informatics Research Division of Hospital Medicine, Stanford School of Medicine; Clinical Excellence Research Center, Stanford School of Medicine; Department of Medicine, Stanford University, Stanford, CA, USA [Email: jonc101@stanford.edu](Email: jonc101@stanford.edu) Abstract: This study evaluates the capacity of large language models (LLMs) to generate structured clinical consultation templates for electronic consultation. Using 145 expert-crafted templates developed and routinely used by Stanford’s eConsult team, we assess frontier models—including o3, GPT-4o, Kimi K2, Claude 4 Sonnet, Llama 3 70B, and Gemini 2.5 Pro—for their ability to produce clinically coherent, concise, and prioritized clinical question schemas. Through a multi-agent pipeline combining prompt optimization, semantic autograding, and prioritization analysis, we show that while models like o3 achieve high comprehensiveness (up to 92.2%), they consistently generate excessively long templates and fail to correctly prioritize the most clinically important questions under length constraints. Performance varies acr

FAQs

What is the main goal of this study?

The study evaluates whether large language models can generate structured clinical consultation templates for electronic consultations that are coherent, concise, and properly prioritized.

How were the language models evaluated?

Models were tested using 145 expert-crafted Stanford eConsult templates, with a multi-agent pipeline that includes prompt optimization, semantic autograding, and prioritization analysis.

What key limitations or failure modes were observed?

Even when models achieve high comprehensiveness, they often produce excessively long templates and fail to prioritize the most clinically important questions within length constraints.

Sep 11, 2026

Uploaded by Taylor Morgan