Who Do Language Models Think Is Competent - A Mechanistic Analysis of Occupational Bias

Sep 15, 2026

Uploaded by Skyler

Language models (LMs) can satisfy behavioral bias evaluations while still retaining internal associations that create bias. This study shows that representational biases remain detectable even when behavioral disparities are not visible. A causal framework decomposes occupational bias into two points: internal representations of a user’s competence and observable outputs. Steering vectors for user expertise are derived and causally tested in question-answering and hiring tasks, revealing how gender, race, and socioeconomic status shape representations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

AccessFree to download

Abstract

Introduction

Methods

Data

Professional Questions

Hiring

Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Bias Keren Fuentes Independent Researcher [kerenfuentes313@gmail.com](kerenfuentes313@gmail.com) Aaron Mueller Boston University [amueller@bu.edu](amueller@bu.edu) arXiv : 2608 .20347v 1 [cs .CL] 15 Jun 2026 Abstract Language models (LMs) often pass behavioral bias evaluations, but it remains unclear whether they no longer represent the underlying associations that give rise to biases, or have merely learned not to express them. In this study, we show that representational biases are often detectable, even when behavioral biases are not visible. We introduce a causal framework that decomposes occupational bias into two measurement points: a model’s internal representation of a user’s competence, and its observable outputs. We derive steering vectors for representations of user expertise, and verify that they causally mediate model behavior in both a question-answering task and a hiring task. Applying this framework to several open-weight models, we find that demographic attributes, such as gender, race, and socioeconomic status, influence a model’s representation of user expertise, even in cases where behavioral metrics detect no disparity between demographics. We show that these model representations can influence downstream behavior under intervention, suggesting failure modes that behavioral metrics alone may not detect.1 1 Introduction Humans can hold subconscious biases about particular demographic groups (Greenwald and Banaji, 1995 ; Greenwald et al., 2009); even when not aware of it; such biases often influence downstream decision-making (Greenwald et al., 2022) . In language models (LMs), this has parallels to the known phenomenon of shortcut learning (Du et al., 2023): LMs often preferentially rely on simpler spurious heuristics over more robust causally relevant features. One extensively studied form of LM shortcut is demographic biases (Bolukbasi et al., 2016 ; Caliskan et al., 2017 ; Li et al., 2024 ; Gupta et al., 2025, inter alia) . 1We make the codebase available at [https://github](https://github). com/kere-nel/representational_bias. Post-training methods have been shown to reduce the appearance of bias, but more recent work shows that demographic biases can still be elicited indirectly (Bai et al., 2025), and that models can encode associations between demographic features and social roles in their representations even when their outputs appear benign (Karvonen and Marks, 2025) . Existing representational methods primarily localize demographic information in LMs’ latent representations (Neplenbroek et al., 2025) . Building on these insights, we introduce a framework for measuring latent biases by decomposing biases into internal representations and observable model behaviors. We define bias as a model implementing mechanisms in which causally irrelevant attributes, including gender, race, and/or socioeconomic status, inform its internal reasoning about a person’s capabilities. To study the extent of mechanistic biases in LMs, we derive vector representations that capture LMs’ internal representations of a user’s expertise in a given domain (§2.2) . To verify their causal role in the model, we steer with these vectors in a hiring task (Karvonen and Marks, 2025), as well asa career-related question answering dataset that we propose. Steering directly influences hiring ratesand the technical complexity of models’ answers to questions. Using pairs of minimally differing prompts, we find that expertise representations are sensitive to whether the user states they have relevant experience or not. However, holding all else fixed, changing only demographic information in the prompt also significantly changes how strongly the model’s activations align with expertise representations. Meanwhile, behavioral measures such as reading levels do not always track with demographic information; this suggests that represe

FAQs

What problem does the paper address about bias evaluations in language models?

It investigates whether passing behavioral bias evaluations means models stop representing the underlying associations that cause bias, or whether they learn to hide them.

How does the paper define and measure occupational bias in language models?

It proposes a causal framework that decomposes occupational bias into internal representations of user competence and observable model outputs, using expertise steering vectors.

What tasks are used to verify the causal role of expertise representations?

The causal steering is verified in both a question-answering task and a hiring task.

Sep 15, 2026

Uploaded by Skyler