Quantifying the Sources of Instability in LLM-Based Stance Analysis of Public Discourse

Q: What two main sources of instability does the paper identify in LLM-based stance analysis pipelines?

It distinguishes instability from (1) the preprocessing pipeline itself, such as diarization and segmentation choices, and (2) the downstream measurement instrument, comparing LLM-based annotation with keyword-lexicon scoring.

Q: How is pipeline sensitivity operationally defined in the study?

Pipeline sensitivity measures how a derived metric changes under alternative, equally plausible preprocessing configurations, evaluated per unit (e.g., per speaker/video) and then aggregated to find when preprocessing choices matter.

Q: What is the key finding about the stability of aggregate valence proportions?

Aggregate valence proportions remain highly stable, with |Δp(neg)| less than 6 percentage points, even though pipeline and measurement choices can still affect correlation and coupling directions.

Jul 16, 2026

Upload by Sage

Computational social science increasingly depends on automated preprocessing pipelines, such as speaker diarization, ASR transcript cleaning, and sentence segmentation, to convert raw media into analyzable text. When identical inputs yield different downstream data, instability can originate from the preprocessing pipeline or from the measurement instrument. Using 256 YouTube interviews across 41 public figures, two diarization pipelines and two annotation approaches are compared to quantify effects on affective valence and epistemic modality coupling.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Related Work

Speaker Diarization in CSS

arXiv :2607 . 10846v 1 [ cs .CL] 12 Jul 2026 Quantifying the Sources of Instability in LLM-Based Stance Analysis of Public Discourse Bo Chen 1 1Institute of Computing Technology, Chinese Academy of Sciences Abstract Computational social science increasingly relies on automated preprocessing pipelines—speaker diarization, ASR transcript cleaning, sentence segmentation—to convert raw media into analyzable text. When these pipelines produce different outputs from the same input, two distinct sources of instability can arise: the preprocessing pipeline itself (diarization method, segmentation rules) and the downstream measurement instrument (LLM annotation vs. keyword lexicon) . Using 256 YouTube interviews across 41 public figures from five domains, we compare two speaker-diarization pipelines and two measurement methods, all targeting the coupling between affective valence and epistemic modality. We find that (1) preprocessing pipeline sensitivity is concentrated in speakers with limited video samples (N ≤ 5); for the four best-sampled speakers (N ≥ 16), the mean absolute pipeline-induced change in r(neg, emph) is only 0. 13; (2) cross-method disagreement is larger and more systematic—the LLM and keyword-lexicon methods assign opposite coupling directions to several well-sampled speakers, even within the same preprocessing pipeline; and (3) aggregate valence proportions are highly stable (|∆p(neg)| < 6pp) regardless of pipeline or method, masking both sources of instability. The contribution is a diagnostic framework that separates pipeline effects from measurement effects: researchers studying cross-dimensional relationships in interview data should verify that their conclusions are robust to both sources of variation, with particular attention to measurement method choice. Keywords: pipeline sensitivity, preprocessing robustness, speaker diarization, LLM annotation, key word lexicon, affective valence, epistemic modality, computational social science 1 Introduction The computational social science (CSS) pipeline is increasingly automated. Raw interview footage passes through automatic speech recognition (ASR), speaker diarization, transcript cleaning, and sentence segmentation before any analysis begins. Each of these preprocessing steps embeds choices—which ASR engine, which diarization method, which sentence boundary detector—and each choice may leave a fingerprint on the resulting data. The CSS literature has devoted substantial attention to annotation validity: whether LLM-generated labels match human judgments [4, 5] . It has devoted far less attention to two upstream sources of instability: preprocessing validity—whether pipeline choices change downstream conclusions—and measurement validity—whether the choice of annotation method (LLM vs. keyword lexicon) leads to different conclusions even when applied to the same preprocessed text. These two sources are often conflated, making it difficult to diagnose whether an observed instability originates in the pipeline or in the measurement instrument. We propose pipeline sensitivity analysis as a systematic approach to this problem: Pipeline sensitivity is the degree to which a derived metric changes under alternative, equally plausible preprocessing configurations, measured per-unit (per-speaker, per-video) and aggregated across units to identify conditions under which preprocessing choice is consequential. Three features distinguish pipeline sensitivity from classical measurement error. First, it is metricdependent: aggregate means may be stable while cross-dimensional correlations reverse sign. Second, it is non-uniform: some speakers or domains may be far more sensitive than others. Third, it is methodinteractive: the same preprocessing divergence may affect LLM-based and lexicon-based measurements differently. We demonstrate pipeline sensitivity analysis through a case study of speaker diarization. Using a corpus of 256 YouTube interviews acros

FAQs

What two main sources of instability does the paper identify in LLM-based stance analysis pipelines?

It distinguishes instability from (1) the preprocessing pipeline itself, such as diarization and segmentation choices, and (2) the downstream measurement instrument, comparing LLM-based annotation with keyword-lexicon scoring.

How is pipeline sensitivity operationally defined in the study?

Pipeline sensitivity measures how a derived metric changes under alternative, equally plausible preprocessing configurations, evaluated per unit (e.g., per speaker/video) and then aggregated to find when preprocessing choices matter.

What is the key finding about the stability of aggregate valence proportions?

Aggregate valence proportions remain highly stable, with |Δp(neg)| less than 6 percentage points, even though pipeline and measurement choices can still affect correlation and coupling directions.

Jul 16, 2026

Upload by Sage