EVALUATING LARGE LANGUAGE MODELS TO SUPPORT DEMENTIA CAREGIVERS - Abstract citation ID igaf122.4109

Sep 29, 2026

Upload by Rizky

Awareness and access to dementia caregiving resources are critical for informal caregivers of people with early-stage dementia. A mixed-methods study evaluated large language models, comparing a baseline ChatGPT-4o with an enhanced, prompt-engineered variant grounded in health science and gerontology. Surveys and interviews with 12 stakeholders assessed model responses to common caregiver questions using validated multidimensional evaluation measures. The enhanced model scored higher for actionability, relevance, and satisfaction, while accuracy, understanding, safety, trust, and potential harm showed no significant differences. Interview themes highlighted wordiness, depth, empathy, actionability, accuracy, relevance, and bias, indicating caregiver-centered improvements.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages1

Reading Time3 min

AccessFree to download

Overview

Study design and models

Evaluation approach

Key findings and implications

Innovation in Aging, 2025, Vol. 9, No. S2 1229 Abstract citation ID: igaf122.4109 EVALUATING LARGE LANGUAGE MODELS TO SUPPORT DEMENTIA CAREGIVERS: IDENTIFYING OPPORTUNITIES FOR IMPROVEMENT Kyeung Mi Oh1, Sungsoo Hong1, Ziwei Zhu2, Huayu Zhou2, and Jung Ah Lee3, 1. George Mason University, Fairfax, Virginia, United States, 2. George Mason University, George Mason University, Virginia, United States, 3. University of California, Irvine, University of California, Irvine, California, United States Awareness and access to the dementia caregiving resources is crucial for informal caregivers of people with early-stage dementia. Large language models (LLMs) offer easy access to caregiving information, but the risks, challenges, and ways to improve LLM-generated responses remain understudied. This mixed methods study evaluated LLMs, including the baseline ChatGPT-4o model and an enhanced version refined through prompt engineering grounded in health science and gerontology literature, to support informal dementia caregivers. This study aimed to assess key factors influencing preferred responses from LLMs and to identify related risks and challenges, thereby informing opportunities for improvement. Surveys and interviews with 12 stakeholders, including 10 healthcare professionals and 2 caregivers, were conducted to assess model responses to questions commonly asked by caregivers. The responses were assessed using validated multidimensional measures based on a human evaluation framework for LLMs in healthcare. Survey results showed the enhanced ChatGPT-4o model scored significantly higher in actionability, relevance, and satisfaction than the baseline ChatGPT-4o version. However, no significant differences were observed between the models in accuracy, understanding, intelligibility, trust, safety, or potential harm. Key themes from interview data influencing preferred responses included wordiness, in-depth content, empathy, actionability, accuracy, relevance, and bias. Overall, while both models’ responses were perceived as overly verbose, the enhanced model provided more comprehensive and caregiver-centered information than the baseline ChatGPT-4o. These findings suggest that LLMs can deliver more practical and satisfying guidance for dementia caregiving. Incorporating domain-specific frameworks into model design may enable scalable, evidence-based support for caregivers in real-world settings.

FAQs

What problem does the study address for dementia caregivers?

Caregivers need reliable access to dementia caregiving resources, but the risks and improvement pathways for LLM-generated answers are not well understood.

How did the researchers compare the two language models?

They evaluated a baseline ChatGPT-4o against an enhanced prompt-engineered version using surveys and interviews with stakeholders and validated multidimensional measures.

What improvements did the enhanced model show compared with the baseline?

The enhanced model scored significantly higher in actionability, relevance, and satisfaction, and provided more comprehensive, caregiver-centered information, despite similar results for accuracy, safety-related aspects, and trust.

Which interview themes influenced preferred LLM responses?

Themes included wordiness, depth of content, empathy, actionability, accuracy, relevance, and bias.

Sep 29, 2026

Upload by Rizky