Rating the Pitch Not the Product User Evaluations of LLMs Reflect Expectations More Than Performance

Q: How did the study frame users’ expectations of the LLM before they interacted with it?

Participants first viewed a landing page that either overstated, matched, or understated the model’s true capability relative to what was actually served in the backend.

Q: Did framing change task performance or only users’ impressions?

Framing shifted user impressions and interaction behavior, but task performance did not change across the framing conditions.

Q: What predicted changes in users’ impressions after using the LLM?

Impression changes were driven by whether the model met users’ expectations and how confident users felt while working with it, rather than by the task performance itself.

Jul 16, 2026

Upload by Oliver

A controlled study compares how two users rate the same LLM when pre-interaction messaging frames the model’s capability differently. Across three collaborative tasks and six LLMs, 162 participants viewed landing pages that overstated, matched, or understated each model’s true performance. Task outcomes stayed unchanged, but user impressions shifted toward reality only partially. Impression change and interaction behavior tracked expectation fit and user confidence, with oversold users prompting more directly and undersold users writing more collaborative prompts.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

Abstract

Introduction

Study Design and Method

Framing Conditions and Participants

Tasks and Measurements

Results

Impression Changes

Interaction Behavior and Prompting

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance Robert Morabito1 Tyler McDonald1 Charitra Viswanath2 Angel Hsing-Chi Hwang3 Susanne Gaube4 Jad Kabbara5 Ali Emami2 1Brock University 2Emory University 3University of Southern California 4University College London 5Massachusetts Institute of Technology arXiv :2607 .05 1 13v 1 [ cs .CL] 6 Jul 2026 Abstract Imagine two users interact with the same LLM. One has been told it is the cutting-edge flagship model; the other, an older, weaker model. They walk away with markedly different ratings of its usefulness and intelligence, yet they used the same model. In a controlled study, 162 participants each used one of six LLMs from two families across three collaborative tasks, after first viewing a landing page that matched, overstated, or understated their model’s true capability. This pre-interaction framing shifted user opinions and interaction behavior while task performance did not. Oversold users rated the model more favorably and used more directive prompting, while Undersold users wrote longer, more collaborative prompts. The quality of what users and the model produced together depended only on the model’s true capability, not on what users were told. Participants’ change in model impressions after use, measured across two impression measures, was not predicted by task performance (β = −0 .01 and 0. 11, both n.s.), but by whether the model met users’ expectations (β = 0 .47 and 0 .50, both p < .001) and how confident they felt working with it (β = 0 .47 and 0 .36, both p < .001) . After interaction, users are still rating the pitch, not the product: user-elicited LLM evaluations, including the preference data driving public leaderboards, measure expectation management at least as much as the model itself. 1 Introduction Most users meet an LLM before they actually meet it. They might have seen a benchmark score, a launch post, a leaderboard ranking (Chiang et al., 2024), or heard a colleague’s recommendation. These pre-interaction signals may have lasting consequences for how users evaluate, adopt, and trust the systems they go on to use. Do initial expectations fade once users accumulate experience with the model, or do they persist? Do they only affect users’ impressions, or also how users interact with it and the quality of what they produce together? And when impressions do change after use, what drives that change: the quality of what the model produced, or something else about the experience? Pre-interaction expectations are well-known drivers of technology evaluation, adoption, and abandonment (Bhattacherjee, 2001 ; Budhathoki et al., 2024), and post-interaction impressions shape satisfaction often independently of objective performance (Blut et al., 2021 ; Avcılar and Yenilmez, 2026) . As LLMs become embedded in workflows where mismatched expectations can drive misuse or misplaced trust (Noy and Zhang, 2023), how expectations evolve into impressions through use is consequential. Work in human-AI interaction has begun to ask these questions. The framing of an AI system (the description users encounter before use) has been shown to shift judgments of trustworthiness (Pataranutaporn et al., 2023), performance during collaboration (Bangerl et al., 2025), and sense of ownership over the resulting artifacts (Lee et al., 2026) . None, however, have followed users across iterative LLM use to ask whether these effects persist, generalize to behavior and output, or what sustains them. For LLMs, where users engage in multi-turn collaboration and capability is signaled through public benchmarks (Kocielnik et al., 2019), marketing, and word of mouth, these questions matter for individual users and the preference-based evaluation pipelines that aggregate their judgments. We ran a controlled between-subjects study with 162 participants, each randomly assigned to one of three framing conditions (Oversold, Matched,

FAQs

How did the study frame users’ expectations of the LLM before they interacted with it?

Participants first viewed a landing page that either overstated, matched, or understated the model’s true capability relative to what was actually served in the backend.

Did framing change task performance or only users’ impressions?

Framing shifted user impressions and interaction behavior, but task performance did not change across the framing conditions.

What predicted changes in users’ impressions after using the LLM?

Impression changes were driven by whether the model met users’ expectations and how confident users felt while working with it, rather than by the task performance itself.

Jul 16, 2026

Upload by Oliver