BRAND-AS-MEMORY - VISION-LANGUAGE MODELS ENCODE CAUSAL, MECHANISTICALLY LOCALIZABLE CREDIBILITY PRIORS FOR NEWS SOURCES

Sep 21, 2026

Uploaded by Oliver

Vision-language models (VLMs) increasingly assess news and web images by the publisher identity that is visually present. The work shows a strong outlet-identity credibility prior that can override article content evidence and varies by model and scale. A cross-model benchmark, CueTrust with a Source-Override Index, pinpoints surface cues driving the override (masthead, logo, domain) and excludes author, authority, and layout. A mechanistic analysis localizes causal formation in layers 19–21 and demonstrates debiasing and steering effects, with released stimuli and benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Abstract

Introduction

Brand-as-memory prior and source cue override

Contributions: CueTrust benchmark and mechanistic account

arXiv :2607 .03365v 1 [ cs .CV] 3 Jul 2026 BRAND-AS-MEMORY: VISION-LANGUAGE MODELS ENCODE CAUSAL, MECHANISTICALLY LOCALIZABLE CREDIBILITY PRIORS FOR NEWS SOURCES Chih-Ting Liao∗ Xin Cao University of New South Wales ABSTRACT Vision-language models (VLMs) increasingly read news and web content as images, where the publisher’s identity is visually present. We show that VLMs carry a strong source-credibility prior keyed on outlet identity, and study it along three axes. (i) A cross-model benchmark. We introduce CueTrust, a cross-model diagnostic that measures which surface source cue overrides an article’s content evidence, via a Source-Override Index (SOI) . Across seven VLMs and five cues the vulnerability profile is model-and scale-dependent, and the override is outletidentity-specific and encoding-invariant, firing from the masthead name, the logo image, or the bare domain, but not from a named author, in-text authority, or page layout (clean negative controls) . (ii) A mechanistic account. For the brand cue we give a full mechanistic account: swapping only the masthead moves credibility across an ∼ 11-log-odds range that tracks professional ratings (ρ=0 .88 vs Media Bias/Fact Check); the prior is dual-coded (name and logo), strengthens with scale, is causally formed at layers 19–21, carried by interpretable seed-stable sparseautoencoder features, and recurs at the same relative locus in a second family; it overrides content (∼ 1.8×) as a signal-magnitude effect within a shared pathway, not a privileged route. Steering the localized direction selectively reduces the override (−41%) and generalizes to held-out outlets, confirming the prior is causally used, not merely decodable. Deployed VLMs may thus defer to source identity over the evidence in front of them, a reliability failure we can measure across models, localize, and causally probe. We release the stimulus suite and CueTrust. 1 INTRODUCTION VLMs now mediate how people read information presented as images: screenshots of news articles, social posts, and web pages. When such a model judges whether a story is credible, does its judgment turn on the evidence in the article, or on who the article appears to come from? We show that for current VLMs it turns substantially on the latter, that the deciding cue is the publisher’s brand identity, and (unlike prior behavioral studies) we localize where in the network that judgment is formed and what it is made of. Concretely, we render neutral, topic-matched news articles as screenshots and vary only the masthead. A VLM’s internal credibility log-odds then span an ∼ 11-log-odds range ordered almost exactly as humans rank these outlets. We call this a brand-as-memory prior: a learned association between a source’s visual identity and its perceived trustworthiness, retrieved from the model’s parametric memory at the moment of judgment. Crucially the cue is visual: a rendered brand logo, with no readable brand text, already recovers the ordering. We make two contributions. (1) A cross-model benchmark: we introduce CueTrust, a diagnostic measuring which surface source cue overrides an article’s content via the Source-Override Index over seven models and five cues, and establish that the override is outlet-identity-specific and ∗ Corresponding author, first author, and project lead. mill .liao@unsw .edu .au Outlet-identity credibility override in VLMs: a cross-model benchmark (b) and a mechanistic account (a,c,d) credibi l ity ( log-odds) 5 4 3 2 1 0 ~~ ~~ 1 (a) Source overrides content good brand + bad content still trusted (SOI=1.79) | | | bad content,
good brand still +1.58

| | --- | --- | --- | | | | | | | | | good content bad content qwen3b qwen7b intern2b intern8b intern14b llava_ov smolvlm R² / transfer 1.0 0.8 0.6 0.4 0.2 0.0 (c) Mechanism: decodable broadly, causally used at L19~~ ~~21 (+ L20 SAE feature) |
| |
| | --- | -

FAQs

What does the paper claim vision-language models use to judge news credibility?

The paper claims VLM judgments substantially depend on the publisher’s visual brand identity rather than the article’s content evidence.

How is the source-content override measured in the study?

It introduces CueTrust and a Source-Override Index (SOI) to diagnose which surface source cue overrides content evidence across multiple VLMs.

Where in the model is the brand credibility prior formed and how is it validated?

The mechanistic account localizes causal formation to layers 19–21 and shows steering the localized direction selectively reduces the override, confirming causal use.

Sep 21, 2026

Uploaded by Oliver