StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

Jul 16, 2026

Upload by Himbo

Large Vision-Language Models rely on Visual Instruction Tuning (VIT) to develop multimodal reasoning, yet a mismatch emerges between multi-turn training and single-turn benchmarking. The same-image, multi-turn conversational format can trigger visual attention decay and contextual overfitting, reducing effectiveness when evaluated in isolated settings. StochasT addresses this gap by stochastically grouping language tasks for one image into clusters with varying turn depth while keeping natural order. A Balanced Latin Square–based, benchmark-agnostic evaluation further measures robustness under differing contextual dependencies. Extensive experiments show improved harmonized performance across single-turn and multi-turn use.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages29

Reading Time1 h 13 min

Introduction

Visual Instruction Tuning for LVLMs

Single-T vs Multi-T Discrepancy

Research Questions

StochasT Method

Robustness Evaluation

arXiv :2607 .00465v 1 [ cs .CV] 1 Jul 2026 StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning Yuan Qing 1 , Chengzhi Mao2 , and Boqing Gong 1 1 Boston University, Boston, USA 2 Rutgers University, New Brunswick, USA {ymqing, [bgong}@bu.edu](bgong}@bu.edu), [chengzhi.mao@rutgers.edu](chengzhi.mao@rutgers.edu) Abstract. Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs’ robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases. Code is available at: [https://yuanqing-ai.github.io/StochasT](https://yuanqing-ai.github.io/StochasT). Keywords: Large Vision-Language Models · Single-Turn, Multi-Turn, and Stochastic Turn-Depth Evaluation · Visual Instruction Tuning 1 Introduction Large Vision-Language Models (LVLMs) [2, 3, 11, 48, 51, 59] have demonstrated remarkable capabilities across various vision applications, spanning Perception (e.g., object detection, segmentation, OCR), Reasoning (e.g., visual question answering, multi-hop reasoning, chart understanding), and Action (e.g., visual instruction following, embodied navigation, GUI control) . Standard approaches for building such systems typically integrate a pretrained Large Language Model (LLM) [19,44] with a pretrained visual encoder, bridged by an alignment module such as an MLP projection layer or a Q-Former [30] . To unlock the multimodal reasoning and generalization potential of LVLMs, Visual Instruction Tuning (VIT) [33,34,71] is critical. Prior studies suggest that much of the world knowledge embedded within foundational LLMs is acquired 2 Y. Qing et al. | | Multi-turn (MultiT) | Stochastic Turn Depth (StochasT) (Ours) Masked from history | | --- | --- | --- | Fig. 1: A visual instruction tuning example (left) and three grouping mechanisms: multi-turn (multiT), singleT, and our proposed stochastic turn depth (StochasT) . during large-scale pretraining [50, 69] . Consequently, VIT primarily serves to activate and align this latent knowledge toward downstream multimodal task objectives, rather than learning it from scratch. Unlike pure language tasks, the high information density inherent to visual data naturally affords multi-turn (multiT) language queries. A single image often grounds multiple distinct instructions (as illustrated in Fig. 1), and this one-imagemultiT format has been frequently used in VIT [34] . However, a significant discrepancy exists between this multiT training paradigm and currently prevalent singleturn (singleT) evaluation protocols. As depicted in Fig. 1, multiT training groups all instruction-answer pairs about the same image into one training example, while singleT evaluation anchors every instruction individually to the image. As a result, a model may fail to answer a simple question in isolation but succeed when the ex- Qwen2.5-V

FAQs

What discrepancy does StochasT address in visual instruction tuning?

VIT often trains LVLMs using multiple language tasks tied to the same image in a multi-turn setting, while common benchmarks evaluate them in isolated single-turn scenarios, causing performance mismatch.

How does StochasT work to improve single-turn and multi-turn performance?

StochasT stochastically groups tasks for the same image into clusters of varying turn depth, preserving organic order, which enriches context diversity without adding data or extra computation overhead.

How is LVLM robustness evaluated under different contextual dependencies?

The document introduces a benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to test robustness when contextual dependencies vary.

Jul 16, 2026

Upload by Himbo