From Voices to Validity - Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews

Oct 15, 2026

Uploaded by Pentious

Stakeholder interviews are critical for shaping effective K–12 education policy, yet manual qualitative coding is often slow and resource-intensive. This study evaluates large language models, using GPT-4 (GPT-4 Turbo) with human expertise, to analyze stakeholder interviews from a U.S. state. Experts created a codebook and iteratively guided prompting for thematic and sentiment analysis. GPT-4 achieved 78% agreement with human coding at detailed levels and 96% alignment on broader themes, producing sentiment results closer to expert judgment, supporting validity, efficiency, and interpretability through human–AI collaboration.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

AccessFree to download

From Voices to Validity

Large language models for stakeholder interview analysis

Mixed-methods design with human-developed codebook

Thematic and sentiment results versus human coding

Implications for validity, efficiency, and interpretability

AERA Open January-December 2025, Vol. 11, No. 1, pp. 1–23 DOI: 10. 1177/23328584251374595 Article reuse guidelines: [sagepub.com/journals-permissions](sagepub.com/journals-permissions)[ ](sagepub.com/journals-permissions)© The Author(s) 2025. [https://journals.sagepub.com/home/ero](https://journals.sagepub.com/home/ero) From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews Alex Liu Min Sun College of Education, University of Washington Stakeholder feedback is essential for policymakers to evaluate and develop effective policies, but traditional qualitative analysis methods are often labor-intensive and time-consuming. This study investigates the use of Large Language Models (LLMs) like GPT-4 Turbo (GPT-4) with human expertise to analyze stakeholder interviews regarding K–12 education policy in a U.S. state. The research employed a mixed-methods approach where human experts developed a codebook and iterative promptsfor GPT-4 to conduct thematic and sentiment analysis. Results demonstrated that GPT-4’s thematic coding achieved 78% agreement with human coding at detailed levels and 96% alignment for broader themes, exceeding traditional Natural Language Processing methods by over 25%. GPT-4 also produced sentiment analysis results more closely aligned with a human expert’s judgment. Our qualitative comparisons between human and GPT-4 analysis results highlight the complementary roles of human expertise and LLMs in enhancing efficiency, validity, and interpretability of educational policy research. Keywords: artificial intelligence, educational policy, equity, validity/reliability, textual analysis, mixed methods, content analysis, measurements, qualitative research, policy stakeholders, human–AI interaction, large language model (LLM), education policy, natural language processing (NLP), thematic analysis, sentiment analysis Policymakers seek reliable, valid, and meaningful evidence to support decision-making in a timely manner. An important source of policy evidence comes from stakeholders’lived experiences regarding the implementation of current policies and their suggestions for improvement (Davidson et al., 2022; Fedorowicz & Aron, 2021) . These stakeholders include individuals and organizations concerned with or affected by a policy’s creation, enactment, and evaluation. Insights can be gleaned from stakeholder interviews, openended surveys, or social media posts (Berry & Herrington, 2013; Rosenberg et al., 2021; Wallner, 2008) . Qualitative data on stakeholders’ insights, combined with quantitative causal analysis of policy impact, are often utilized in policy analysis. However, when decisions must be made swiftly, the cost of manually analyzing even a moderately sized corpus of text can impede the actual incorporation of stakeholders’voices (Grimmer & Stewart, 2013) . The rapid advancement of natural language processing (NLP) techniques—from traditional rule-based and statistical models to deep learning-based models, such as large language models (LLMs)—is enabling time-efficient comprehension of contents and sentiments in large text corpora. Traditional NLP approaches, including methods like Structural Topic Model (STM), have found favor in social sciences and policy studies for uncovering latent themes in extensive collections of texts, such as political speeches, news articles, and social media content (Blei et al., 2003; DiMaggio et al., 2013; Gao et al., 2023) . Policy researchers are also interested in gauging stakeholders’ satisfaction about a given policy, perceived intended and unintended consequences, and areas for further improvement. Lexicalbased sentiment analysis has been a popular tool for exploring public reactions expressed on social media or other platforms (Abdulaziz et al., 2021; Das et al., 2021) . Recent advances in LLMs have demonstrated impressive abilities to capture textual nuances. Researchers across various

FAQs

How were large language models used in the study?

GPT-4 (GPT-4 Turbo) was used to perform thematic and sentiment analysis of stakeholder interviews, guided by human expertise.

What approach did the researchers use for analysis?

A mixed-methods approach combined a human-developed codebook with iterative prompts for GPT-4 to carry out qualitative coding and sentiment analysis.

How did GPT-4 performance compare with human coding?

GPT-4 thematic coding reached 78% agreement with human coding at detailed levels and 96% alignment for broader themes, with sentiment results more closely aligned to expert judgment.

Oct 15, 2026

Uploaded by Pentious