RPAM: 用相对概率关联度量评估生成式语言模型的关联性（高预测效度）

Jul 16, 2026

Introduces the Relative Probability Association Metric (RPAM) to evaluate social associations in generative language models with high predictive validity for downstream outputs. Addresses a core limitation of existing evaluation methods: downstream-only metrics depend on specialized datasets and vary with generation quality, while prior upstream metrics have not reliably matched real-world associations observed in humans or in generated text. Across multiple models (Mistral-7B-Instruct, Mistral-7B, GPT-2) and datasets, RPAM shows strong alignment with implicit/explicit human associations and downstream bias measures, outperforming prior results when applicable.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Abstract

Introduction

Problem of social biases in generative LMs

Limitations of downstream vs. upstream metrics

RPAM approach and design

Experimental setup and goals

RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs Damian Hodel 1 , Jevin West 1 , Aylin Caliskan 1 , 1University of Washington, Seattle, USA {hodeld, jevinw, [aylin](aylin}@uw.edu)[}](aylin}@uw.edu)[@uw.edu](aylin}@uw.edu) arXiv :2607 .05679v 1 [ cs .CL] 6 Jul 2026 Abstract Language models (LMs) exhibit problematic biases, such as stereotypes. Effectively analyzing and mitigating such biases requires accurate and generalizable evaluation methods ofthe underlying associations. Some existing approaches focus on downstream metrics that analyze associations in generated text. Since generated text content can vary drastically across LMs, such metrics often require specialized evaluation datasets, which limits the generalization of such downstream metrics. In contrast, upstream metrics examine LMs at the fundamental level of embeddings or continuation probabilities, enabling principled association analyses across LMs. Yet, to date, no upstream metric for generative LMs has uncovered a strong relationship with real-world associations, including those measured in generated text. To address this gap, we introduce the Relative Probability Association Metric (RPAM), an association evaluation metric for generative LMs. For three LMs of different quality of language generation and purpose (Mistral-7B-Instruct, Mistral- 7B, and GPT-2) and well-studied evaluation datasets (WEATWS, Bellezza, WS-353, and SST2), we find a strong relationship between upstream RPAM measurements and corresponding implicit and explicit associations observed in humans, as well as biases measured downstream with LMspecific tasks, outperforming prior record values where applicable. 1 Introduction Generative language models (LMs) such as chatbots exhibit associations between concepts, for example, between women and arts. While necessary in language, associations can be harmful for example, when LMs generate texts involving stereotypes and negative attitudes towards specific social groups (Ghosh and Caliskan 2023), or when systems built on these LMs are used for automated decisions in highstakes settings such as healthcare (Apell and Eriksson 2023), and content moderation (Boicel 2024) . In social contexts, such problematic associations are often referred to as social biases (Bender et al. 2021; An et al. 2023; Rudinger et al. 2018; Hofmann et al. 2024) . Effective strategies to mitigate these risks of harm, such as artificial intelligence regulations, require accurate and generalizable association measurement methods, typically consisting of an evaluation metric and a dataset fed to the LM (Gallegos et al. 2024) . For generative LMs, existing ap- proaches tend to focus on downstream metrics that aim to measure associations directly in LMs’ generated text (e.g. Kotek, Dockum, and Sun 2023; Wan et al. 2023; Dhamala et al. 2021) . Since the quality of language generation depends on a LM’s type (e.g. architecture, size, fine-tuning purpose, etc., illustrated in Figure 4 in the appendix), downstream metrics often rely on specialized evaluation datasets for specific concepts and LMs, which limits the generalization of downstream metrics (Gallegos et al. 2024) . In contrast, the majority of upstream metrics examine LMs at the fundamental level of embeddings (Wolfe and Caliskan 2022; May et al. 2019; Tan and Celis 2019) or continuation probabilities 1 (Nadeem, Bethke, and Reddy 2021; Kurita et al. 2019; Hofmann et al. 2024) . Independent of text generation and decoding, upstream metrics could enable principled association evaluation, involving systematic analysis at scale grounded in social science and applicable across various LM types, thereby addressing key limitations of specialized methods by design. However, some evaluations using prior metrics suggest that upstream measures may not fully capture the harmful behavior of LMs in real-world applications (Cao et al. 2022; Steed e

FAQs

为什么仅使用下游（downstream）指标来评估关联性会受到限制？

下游指标依赖生成文本来测量关联，而不同语言模型的生成质量与内容差异很大，导致通常需要专用评估数据集，从而难以跨模型泛化。

RPAM在评估关联性时相较于现有上游（upstream）指标做了什么改进？

RPAM提出一种用于生成式语言模型的上游关联评价度量，通过在同一数据集中对两段输入的相对关联进行归一化，以期形成与现实关联（人类隐式/显式以及下游生成文本）强相关的测量。

RPAM如何在实验中被验证其与真实关联的对应关系？

论文选择不同生成质量的三个语言模型，并使用WEAT-WS、Bellezza、WS-353和SST2等评估数据集，在三个实验中比较RPAM的上游测量与人类隐式/显式关联以及下游任务中观测到的偏差结果。

Jul 16, 2026

Upload by Theodore