TypeProbe Recovering Type Representations from Hidden States of Pre-trained Code Models

Jul 16, 2026

State-of-the-art code models perform well yet their internal encoding of type information is not well understood. TypeProbe investigates residual streams of pretrained code models using a parallel dataset of Java and Python programs. It finds cross-lingual type representations can arise even from untyped code. Linear probing is used to test whether hidden states encode result types implied by typed function application across languages, and the discovered structure shows partial robustness to lexical perturbations and cross-language syntactic variation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Introduction

Interpreting Code Models

Experimental Setup

Dataset and Design

Type Systems

arXiv :2607 .08339v 1 [ cs .CL] 9 Jul 2026 TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models Giuliano Gorgone 1[0009−0004−0827−592X], Fausto Carcassi 1[0000−0001−6843−1737] ILLC, University of Amsterdam, Amsterdam, The Netherlands [giuliano.gorgone@student.uva.nl](giuliano.gorgone@student.uva.nl) , [f.carcassi@uva.nl](f.carcassi@uva.nl) Abstract. State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets 1 . Keywords: Code LLMs · Linear Probing · Type Semantics · Cross-lingual Representations · Adversarial Renaming · Lexical Interference 1 Introduction State-of-the-art code models excel at next-token prediction but can still violate formal type-system constraints. For instance, in generated TypeScript, nearly 94% of compilation errors stem from type-check failures [10] . Type-constrained decoding [10] mitigates this by enforcing language-specific rules externally, but such approaches assume that models cannot internalize formal type information and require ad hoc machinery for each language. Diagnostic probing has become a standard tool for studying abstract information encoded in neural representations [1,7 ,4], and prior work on pretrained code models shows that they capture syntax, identifiers, and namespaces more readily than complex semantic properties [13] . At the same time, recent activationsteering results suggest that code models contain a controllable type-prediction mechanism shared across languages [9], motivating a more direct investigation of how such information is represented internally. In natural language, multilingual representations often exhibit approximately aligned manifolds across languages 1 [https://github.com/anticleiades/TypeProbe](https://github.com/anticleiades/TypeProbe) 2 G. Gorgone, F. Carcassi [3,8]; here we ask whether an analogous structure emerges for type information in code models. In this work, we use layer-wise linear probes to test whether pretrained code models encode type information in their residual streams, where this information is localized across depth, and whether it transfers across Java and Python. 2 Interpreting Code Models We investigate the internal logic of code models by addressing three primary research questions. First, we examine representation: do pretrained code models develop linearly decodable representations of type-level semantics, and in which transformer layers does this information emerge? Second, we explore invariance and robustness: is this representation invariant to language syntax and robust against lexical-level adversarial perturbations? More specifically, do models prioritize formal type constraints over superficial heuristics when identifier names explicitly contradict their true underlying types? Finally, we assess inference: beyond type prediction, can models leverage these internal representations to resolve data flow and perform typed function application? 2.1 Experimental Setup Dataset and Design. We construct a programmatically generated dataset with three 90K-example partitions: Java (Java), pyUnt (Python without type annotations), and pyTag (Pyt

FAQs

What is the main goal of TypeProbe?

TypeProbe aims to recover and analyze internal type representations in pretrained code models by probing their residual streams, with a focus on cross-lingual behavior between Java and Python.

How is the dataset constructed to avoid lexical shortcuts?

The dataset includes Java and Python variants with and without type annotations, while identifiers and literals are independently randomized per partition so that operationally equivalent examples do not share exact names or values across languages.

What transfer and robustness effects does the work report?

The work reports that cross-lingual type representations emerge even from untyped code. Probes trained in one language can infer argument and result types in the other, and the learned structure is partly robust to lexical perturbations and syntactic variations across languages.

Jul 16, 2026

Upload by River Wang