Persona Non Grata: LLM Persona-Driven Generations in MCQA Are Unstable in Distinct Dimensions

Jul 16, 2026

Upload by Finn

Persona-driven generations (PDGs) use an LLM “persona” to complete tasks, yet stability concerns remain underexplored when persona is expressed in non-text-heavy outputs such as multiple-choice question answering (MCQA). This work studies PDG instability in MCQA by introducing three metrics that measure performance, outcome, and question correctness stability across dimensions. Results show instability varies by model family and size, and by question domain, with math/commonsense yielding higher instability. Prompt format changes instability more than temperature.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Abstract

Introduction

Related Work

Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions César Guerra-Solano, Xiang Lorraine Li Department of Computer Science, University of Pittsburgh Correspondence: {cguerrasol, [xianglli}@pitt.edu](xianglli}@pitt.edu) arXiv :2607 .00937v 1 [ cs .CL] 1 Jul 2026 Abstract Persona-driven generations (PDGs) have seen prolific use in research and industry applications, where a large language model (LLM) takes on a “persona” while completing some task. While persona expressed through freeform text (like dialogue) has substantial work investigating stability or consistency, relatively, persona expressed in non-text-heavy outputs (like in multiple-choice question answering, or MCQA) is often overlooked. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks. We develop three metrics investigating the per formance, outcome, and question correctness stability, evaluating three distinct dimensions. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity. This reveals the importance of checking hyperparameter instability in PDGs. 1 Introduction Large language models (LLMs) have seen prolific use in a variety of domains, with their adaptability and broad capabilities lending themselves to high performance across a range of tasks and opportunities for personalization for user-and task-specific needs (Kojima et al., 2022 ; Brown et al., 2020 ; Grattafiori et al., 2024 ; Yang et al., 2025 ; Zhang et al., 2024) . With this, persona-driven generations (PDGs) have become prevalent – here, by leveraging the power of prompting, LLMs can roleplay as a "persona" while carrying out some task, such as general user assistance (e.g. "You are a TPF = justanswer a lawyer an old person Figure 1: Depicting instability in non-text-heavy persona-driven generations (PDGs) . Given different, although still reasonable, experiment configurations, represented by different settings for the task prompt format (TPF) hyperparameter, an LLM’s persona-driven performance on a multiple-choice evaluation differs greatly. teacher. Explain [...]") or more specific task completion (e.g. "You are a hiring manager. Rate these resumes [...]") . As PDGs see use in high-stakes domains, from medicine to education, evaluating their stability is critical (Sun et al., 2025 ; Yuan et al., 2025 ; Kyung et al., 2025 ; Li et al., 2024) . With PDGs, two main uses can be observed: in free-form text generation settings, such as in dialogue systems like Character.AI (Character.AI), and non-text-heavy outputs, such as in persona-based multiple-choice question answering (MCQA) . In the latter case, LLM persona is expressed through task completion: rather than producing text that reflects the style of a persona, an LLM answers questions in accordance with the capabilities associated with that persona, when relevant (e.g., a biologist answering biology questions), or to remain unaffected by persona choice when the persona is irrelevant (e.g., race should not influence performance on mathematical questions) . However, relative to text generation, little work has characterized or improved PDGs in MCQA. Prior studies indicate that PDGs in this setting are implicitly biased and unpredictable (Gupta et al., 2024 ; Zheng et al., 2024), motivating further work to characterize them and their potential flaws. Additionally, these studies bear little standardization, with large differences in model choice and hyperparameters a

FAQs

What problem does the paper study about persona-driven generations in MCQA?

The paper investigates how unstable LLM persona-driven generations can be when the persona affects performance in multiple-choice question answering tasks, rather than free-form dialogue outputs.

How does the paper measure instability in MCQA?

It develops three metrics that evaluate stability in performance, overall outcomes, and question correctness across distinct experimental dimensions.

What factors most affect instability according to the results?

Instability differs consistently by model family and size, varies across question domains (with math/commonsense more unstable), and is more influenced by task prompt format than by hyperparameters like temperature.

Jul 16, 2026

Upload by Finn