Robust for the Wrong Reasons - The Representational Geometry of LLM Robustness to Science Skepticism

Q: Do LLMs retreat from scientific consensus when users express skepticism?

No sycophantic retreat is observed. Under skepticism, models follow distinct policies rather than drifting toward a false balance.

Q: What policy differences appear across models under skeptical pressure?

The study reports reactive assertion (consensus commitment increases), surface hedging (tone softens while holding the position), and non-response, varying qualitatively across models.

Q: Does robustness to skepticism transfer across domains and multi-turn conversations?

No. The robustness attenuates across domains and, in the vaccine safety setting, can reverse; robustness is further weakened under skeptical pressure with myth-rebuttal effects.

Sep 21, 2026

Uploaded by Aurora

Large language models (LLMs) are increasingly used for contested scientific questions, raising concern that they may yield to skeptical users by retreating from established consensus into a false balance. This study tests that concern across three open instruction-tuned models (Llama-3.1-8B, Qwen2.5-7B, Mistral-7B), three domains (climate change, vaccine safety, evolution), and single- and multi-turn settings. Results show no sycophantic retreat; instead, models follow distinct policies under skepticism and internal representations differentiate stance tracking from non-response, with limited robustness transfer and possible reversal in safety-critical settings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Introduction

Research questions

Methods overview

arXiv :2607 .01951v1 [physics .soc-ph] 2 Jul 2026 Robust for the Wrong Reasons: The Representational Geometry of LLM Robustness to Science Skepticism Minjong Cheon Department of Computer Science and Engineering, Sejong University [jmj2316@sejong.ac.kr](jmj2316@sejong.ac.kr) Abstract Large language models (LLMs) are increasingly consulted on contested scientific questions, and a natural concern is that they will sycophantically retreat from established consensus when a user signals doubt—abandoning a firm position fora false balance that treats settled science as merely one view among several. We test this concern directly across three widely used open instruction-tuned models (Llama-3.1-8B, Qwen2.5-7B, Mistral-7B), three consensus-science domains (climate change, vaccine safety, evolution), and single-and multi-turn settings, combining behavioral measurement with linear probing and activation patching. We do not observe sycophantic retreat. Instead, models exhibit three qualitatively distinct policies under the same skeptical pressure: reactive assertion, in which consensus assertion increases rather than decreases (Llama); surface hedging, in which tone softens while the position holds (Qwen); and non-response (Mistral) . Pairwise judgments confirm that the reactive shift reflects stance rather than style (63 .6%, p= .007), anda component decomposition identifies increased consensus assertion, not false balance, as its driver (β=+0 .042 per dose, p <10−77) . Linear probes localize the divergence to middle layers—perfect separation in Llama and Qwen versus 72% in Mistral, with non-overlapping confidence intervals—indicating that the non-responsive model does not linearly represent the skepticism signal at all. Crucially, this robustness does not transfer: it attenuates across domains and, in the safety-critical vaccine domain, can reverse, with myth-rebuttal weakening under skeptical pressure. We synthesize these results into a four-way taxonomy separating active from accidental robustness, and argue that behavioral evaluation alone cannot distinguish a model that resists skepticism because it understands the signal from one that appears to resist only because it fails to perceive it. Keywords: AI safety · mechanistic interpretability · robustness evaluation · science communication · sycophancy 1 Introduction Large language models are now a routine first stop for questions on contested scientific topics. People ask them whether climate change is real, whether vaccines are safe, whether evolution is settled—often not as neutral inquiries but with a stance already attached (“I think this is overblown, but. . .”) . As these systems move from novelty to infrastructure, the way they handle a skeptical user is no longer a curiosity; it shapes what a large population comes to believe about questions where a scientific consensus already exists [6] . A widely voiced concern is that LLMs are sycophantic: that they seek user approval in unwanted ways and shade their answers toward what a user appears to want to hear [21] . Applied to consensus science, the worry has a specific form. When a user signals doubt, a sycophantic model would soften its commitment to the consensus, hedge, and drift toward a false balance that presents an established finding and its fringe rejection as comparably credible positions. This failure mode is well documented in adjacent settings: in mental-health deployments, LLM counselors over-validate and reinforce users’ harmful beliefs rather than challenge them [9, 16] . The expectation, then, is that skeptical framing should pull models away from scientific consensus. However, little is known about whether this expectation actually holds when models are probed systematically on consensus-science questions, and still less about why a model behaves as it does. Most existing work measures behavior alone—whether an output shifts—without examining the internal computation that produces the

FAQs

Do LLMs retreat from scientific consensus when users express skepticism?

No sycophantic retreat is observed. Under skepticism, models follow distinct policies rather than drifting toward a false balance.

What policy differences appear across models under skeptical pressure?

The study reports reactive assertion (consensus commitment increases), surface hedging (tone softens while holding the position), and non-response, varying qualitatively across models.

Does robustness to skepticism transfer across domains and multi-turn conversations?

No. The robustness attenuates across domains and, in the vaccine safety setting, can reverse; robustness is further weakened under skeptical pressure with myth-rebuttal effects.

Sep 21, 2026

Uploaded by Aurora