Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

Jul 16, 2026

Upload by Finn

Complex problems in math, logic, and planning are often solved via step-by-step reasoning where each intermediate result informs the next. This work frames Transformer reasoning as being limited by a fixed maximum number of computation steps along latent paths. It introduces Turbo Connection (TurboConn), routing residual information from higher-layer hidden states for token t into lower layers for token t+1. Fine-tuning yields accuracy gains from 0.9% to over 10% on GSM8K, Parity, and multi-step arithmetic. Dense backward connections are crucial, and the method improves reasoning without full retraining or added latency.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages20

Reading Time50 min

Introduction

Motivation: limits of fixed computation depth

TurboConn: higher-to-lower layer routing

Trade-offs and implementation notes

Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers Mohan Tang 1 Sidi Lu 1 arXiv :2602 . 17993v2 [ cs .LG] 12 Jul 2026 Abstract Complex problems, whether in math, logic, or planning, are solved by humans through a sequence of steps where the result of one step informs the next. In this work, we adopt the perspective that the reasoning power of Transformers is fundamentally limited by a fixed maximum number of steps along any latent path of computation. To address this, we introduce Turbo Connection (TurboConn), a novel architecture that overcomes the fixed-depth constraint by routing multiple residual connections from the higher-layer hidden states of each token t to the lower layers of token t + 1 . Fine-tuning pre-trained LLMs with our method not only yields accuracy gains of 0.9% to over 10% on benchmarks like GSM8K, Parity, and multi-step arithmetic, but also demonstrates that the density of these backward connections is critical; our dense interaction significantly outperforms "sparse" alternatives that only pass a single hidden state or vector. Notably, TurboConn can be integrated into pre-trained LLMs to overcome task-specific plateaus: while a fine-tuned Qwen- 3-1.7B achieves only 53.78% on Parity, adding our architectural modification enables the model to reach 100% accuracy, all without the necessity to retrain the full model from scratch or sophisticated curriculum learning. Our results provide strong empirical evidence that the depth of the computational path is a key factor in reasoning ability, also offering a new mechanism to enhance LLMs without significantly affecting generation latency. 1. Introduction While Transformer-based Large Language Models (LLMs) have advanced significantly in recent years, they continue to exhibit shortcomings on tasks that demand complex reason- 1UCLA. Correspondence to: Mohan Tang <tangmo[hanp@outlook.com](hanp@outlook.com)>. Preprint. July 14, 2026. | Layer 2 | | --- | | | | Layer 1 | | | | Embed | | Layer 2 | | --- | | | | Layer 1 | | | | Embed | | Layer 2 | | --- | | | | Layer 1 | | | | Embed | Figure 1. Modified Transformer architecture with downward connections (orange arrows) from higher to lower decoder layers. ing. The popular Chain-of-Thought (CoT) framework (Wei et al., 2022) addresses this by allocating dynamic computation through intermediate steps. However, this approach places a considerable strain on computational resources and often requires specialized training data (DeepSeek-AI et al., 2025) . A growing area of research (Dehghani et al., 2019 ; Geiping et al., 2025 ; Fan et al., 2025a) focuses on performing additional computation in the depth dimension of a Transformer rather than along the token sequence dimension. This is achieved by recursively applying Transformer layers, which enables increased latent reasoning. In this approach,"thinking" occurs within the model’s hidden states instead of being explicitly projected onto tokens, potentially allowing for more complex information processing. A key advantage of this approach is that it allows training on unlabeled pre-training data (Geiping et al., 2025 ; Zeng et al., 2025) . However, it still requires increased time and GPU costs during training, as well as longer inference times that scale proportionally with the number of recursion steps. These factors pose considerable scalability concerns for future development. Is enhancing reasoning solely a matter of increasing the total amount of computation? We argue that the answer is no. In this work, we aim to increase the effective depth—defined as the maximum length of the computational path available to process information—with negligible impact on total floating-point operations. In standard Transformers, information flows strictly from lower to higher layers. As a result, the maximum length of the computational path is always bounded by a fixed number (proportional to the depth of the model) . Because

FAQs

What limitation does the paper claim about Transformer reasoning?

Transformer reasoning is limited by a fixed maximum number of computation steps along any latent path, which constrains effective reasoning depth.

How does Turbo Connection (TurboConn) change information flow?

It routes multiple residual connections from higher-layer hidden states of token t into lower layers processing token t+1, allowing effective reasoning depth to grow with sequence length.

What evidence shows TurboConn improves reasoning beyond fixed-depth models?

Fine-tuning pretrained LLMs with TurboConn improves accuracy on benchmarks such as GSM8K and Parity, and a dense interaction pattern substantially outperforms sparse alternatives while avoiding full retraining and sophisticated curricula.

Jul 16, 2026

Upload by Finn