Directional Contextual Representations for Dependency Relations - Why Cross-Direction Pairing Fails

Sep 15, 2026

Splitting a bidirectional LSTM’s contextual representation into forward-only Fi (tokens 1..i) and backward-only Bi (tokens i..n) improves dependency relation-type classification over either direction alone and over a fused self-attention representation. Yet cross-direction pairing—comparing Fi to a candidate’s backward state Bj—consistently underperforms same-direction pairing, with a penalty that grows with token distance. Diagnostics rule out architectural leakage and most training co-adaptation, revealing partial redundancy and anticipatory encoding while positional and decay probes show distance-limited directional information propagation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Introduction

Research questions

Model setup and directional split

Findings (preview)

F/B splitting vs fused baselines

Cross-direction pairing failure and distance growth

Diagnostic analyses

Validation across backbones and datasets

Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails Sai Krishna Arthanari, JaeHyeong Chang, Chengzhe Sun, Siwei Lyu Institute for Artificial Intelligence and Data Science (IAD), University at Buffalo Buffalo, NY, USA [arthanarisaikrishna@gmail.com](arthanarisaikrishna@gmail.com), [jchang46@buffalo.edu](jchang46@buffalo.edu), [csun22@buffalo.edu](csun22@buffalo.edu), [siweilyu@buffalo.edu](siweilyu@buffalo.edu) arXiv : 2608 .20647v 1 [cs .CL] 2 1 Aug 2026 Abstract—Splitting a bidirectional LSTM’s contextual representation into a forward-only Fi (strictly a function of tokens 1..i) and a backward-only Bi (strictly a function of tokens i..n) beats either alone and beats a fused self-attention representation for dependency relation-type classification. But a specific, natural extension of this idea – pairing a token’s forward state against a candidate’s backward state (“cross-direction” pairing, Fi vs. Bj )– consistently underperforms same-direction pairing, and the penalty grows, not shrinks, with token distance, both paired-bootstrap significant. We diagnose why using a frozentrunk methodology: architectural information leakage between directions is impossible by construction (a single-layer BiLSTM, verified by code inspection); 93% of the same-vs-cross gap survives freezing the trunk and training only fresh heads, ruling out training-co-adaptation as the primary cause; linear regression shows partial representational redundancy between Fi and Bi (R2 =0 .324 vs. 0.028 for a shuffled control) and a linear probe shows partial anticipatory encoding of upcoming tokens in Fi (36.5% vs. 17.2% majority baseline)– real effects, but neither alone, nor combined, cleanly explains the full gap. Extended frozen-trunk diagnostics (a positional probe and a distancedecay probe) show directional information is genuinely stored but not exactly positioned, and propagates only a few tokens before decaying to baseline – consistent with, and mechanistically underneath, the distance-growth finding. We validate the core claim three ways: a parameter-matched Transformer backbone (a fresh end-to-end BiLSTM still significantly outperforms it, Cohen’s d shrinking from 0.216 to 0.085 but staying significant); a second UD English treebank of a substantially different genre (GUM), on which both the directional-splitting and crossdirection-failure findings replicate closely; and a literature search that found no prior work performing this specific cross-direction ablation, reported plainly as a novelty claim rather than assumed. Index Terms—dependency parsing, self-attention, bidirectional LSTM, directional representations, Universal Dependencies, empirical evaluation I. INTRODUCTION Contextual encoders that process a sequence in both directions – bidirectional LSTMs, and, less explicitly, self-attention – are ubiquitous in NLP, but the specific contribution of directional structure to relational tasks like dependency parsing is rarely isolated. Most architectures fuse forward and backward (or all-to-all) information into a single representation before any relational comparison happens, so it is not obvious whether keeping directions separate through the comparison step itself matters, and if so, in which specific combinations. This paper asks that question narrowly and empirically: does directional structure in a contextual representation carry recoverable relational information, and specifically, is comparing a token’s state from one direction against a candidate’s state from the opposite direction informative, or not? The O (n2 ) cost of unrestricted self-attention [1] is useful background for why representation structure is worth studying carefully, but reducing that cost is not this paper’s question – we make no adaptive-computation or efficiency claim here. We use a single-layer bidirectional LSTM trunk, whose two directions architecturally guarantee a forward-only representation Fi (

FAQs

How does directional splitting (Fi and Bi) affect dependency relation-type classification?

Splitting a bidirectional LSTM into forward-only Fi and backward-only Bi yields better relation-type classification than either direction alone and also outperforms a fused self-attention baseline.

Why does cross-direction pairing (Fi vs Bj) perform worse than same-direction pairing?

Cross-direction pairing underperforms consistently, and the penalty increases with token distance. The paper diagnoses this using multiple controls, including checks against architectural information leakage and training co-adaptation.

What evidence supports the proposed explanations for the cross-direction failure?

The work finds partial representational redundancy between Fi and Bi and shows partial anticipatory encoding in Fi. Extended frozen-trunk probes indicate directional information is stored but imprecisely positioned and decays to baseline over only a few tokens, supporting the distance-growth effect.

Sep 15, 2026

Uploaded by Clementine