Scalable tracking of symptoms in the electronic health record using large language models in patients with central nervous system cancers undergoing therapy

Sep 23, 2026

Advances in large language models (LLMs) enable scalable symptom tracking from electronic health records (EHRs) for clinical trials and post-marketing surveillance. This study validated LLM-extracted symptoms against expert manual review using 499 neuro-oncology clinical notes and achieved average sensitivity and specificity of 0.97. The model was then applied to 51,541 external notes (1,642 patients) to quantify real-world symptom prevalence across multiple therapies.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Abstract

Background

Methods

Results

Conclusions

Importance of the Study

206 Neuro-Oncology 28(1), 206–217, 2026 | [https://doi.org/10.1093/neuonc/noaf223 | Advance Access](https://doi.org/10.1093/neuonc/noaf223 | Advance Access) publication September 18, 2025 Scalable tracking of symptoms in the electronic health record using large language models in patients with central nervous system cancers undergoing therapy John Y. Rhee, Zachary Tentor, Thomas Sounack, Brigitte Durieux, Paul J. Miller, Rameen Beroukhim†, and Charlotta Lindvall† All author affiliations are listed at the end of the article Corresponding Authors: JohnY. Rhee, MD MPH, Dana-Farber Cancer Institute, Jimmy Fund Building, Mailstop JF-836, 450 Brookline Ave, Boston, MA 02215, United States ([john_rhee@dfci.harvard.edu](john_rhee@dfci.harvard.edu)); Rameen Beroukhim, MD PhD, Dana-Farber Cancer Institute, 450 Brookline Ave, Boston, MA 02215, United States ([rameen_beroukhim@dfci.harvard.edu](rameen_beroukhim@dfci.harvard.edu)) . †These authors are co-senior author. Abstract Background. Advances in large language models (LLMs) provide a means for scalable tracking of patient symptoms in clinical trials and post-marking surveillance using the electronic health record (EHR). Therefore, we sought to validate symptoms extracted from the EHR using a LLM to scale symptom extraction from the EHR. Methods. Across a dataset of 499 randomly chosen clinical notes from patients seen in a neuro-oncology clinic, GPT-4o annotated symptoms (headache, fatigue, nausea, anxiety, difficulties sleeping, numbness and tingling, rash, constipation, and diarrhea) with an average sensitivity and specificity of 0.97 relative to expert manual review. We then applied the LLM to an external dataset of 51,541 notes representing 1,642 patients to obtain real-world symptom prevalence for temozolomide, bevacizumab, lomustine, immune checkpoint inhibitors (ICI), and methotrexate. Results. In the external dataset, the average number of symptoms per note was 3.92, and the most common symptom was fatigue (83% of patients) . Surprisingly, patients receiving ICIs suffered from the most symptoms (mean = 4.68) and those receiving methotrexate had the least (mean = 2.92). We also found that the prevalence of reported symptoms in this real-world cohort was often much greater than the prevalence of reported symptoms in clinical trials of similar treatment regimens. Conclusions. Large language models offer the ability to scale symptom extraction from health records, which is crucial to understand symptom burden and power symptom-related interventions and studies in real-world patient cohorts. Key Points • Patients in our real-world cohort undergoing treatment experienced 5-250 times greater symptom rates compared to trials. • Large language models offer the ability to scale symptom extraction from health records, allowing for post-marketing surveillance. Received March 26, 2025; accepted September 15, 2025. © The Author(s) 2025. Published by Oxford University Press on behalf of the Society for Neuro-Oncology. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License ([https://creativecommons.org/](https://creativecommons.org/)[ ](https://creativecommons.org/)[licenses/by-nc/4.0/](licenses/by-nc/4.0/)), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact [reprints@oup.com](reprints@oup.com) for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please [contact](contact journals.permissions@oup.com)[ journals.permissions@oup.com](contact journals.permissions@oup.com). 207 Rhee et al.: Scalable tracking of symptoms using large language models Importance of the Study To date, estimates of the symptom prevalence for brain tumor patien

FAQs

How were symptoms extracted from the electronic health record in this study?

GPT-4o annotated symptoms (e.g., headache, fatigue, nausea, anxiety, sleep difficulties and other listed symptoms) from neuro-oncology clinical notes, and then the same LLM was applied to a large external dataset.

How accurate was the LLM compared with expert manual review?

Across 499 randomly chosen clinical notes, symptom annotations achieved an average sensitivity and specificity of 0.97 relative to expert manual review.

What did the external dataset show about symptom prevalence across treatments?

In the real-world cohort, fatigue was the most common symptom (83% of patients). Patients receiving immune checkpoint inhibitors experienced the most symptoms on average, while patients receiving methotrexate had the fewest.

Sep 23, 2026

Uploaded by Maya Linwood