ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

Jul 16, 2026

Upload by Ezra

Streaming 3D reconstruction uses a compact recurrent scene state to integrate unbounded image streams in linear time with bounded memory, but repeated updates can corrupt the state, overwriting reliable history with noisy or ambiguous observations. ReCal3R introduces reliability-calibrated learning rates: it estimates state token reliability from the maintained scene state and uses it to calibrate a candidate update rate derived from token alignment, reconstruction residual, and update pressure. Reliable tokens adapt while unreliable tokens avoid aggressive updates for cleaner reconstructions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Introduction

Streaming 3D reconstruction with recurrent state

Limitations of bounded recurrence over long streams

Reliability-calibrated update learning rate (ReCal3R)

arXiv :2607 .05356v 1 [ cs .CV] 6 Jul 2026 RECAL3R: RELIABILITY-CALIBRATED LEARNING RATES FOR STREAMING 3D RECONSTRUCTION Xinze Li1 , Yiyuan Wang1,2 , Pengxu Chen3 , Wentao Fan1,4 , Weifeng Su1,4 , Weisi Lin5 , Wentao Cheng1 ∗ 1Beijing Normal-Hong Kong Baptist University 2Hong Kong Baptist University 3Jilin University 4 Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science 5Nanyang Technological University | | | | | | | | <br> | | --- | --- | --- | --- | --- | --- | --- | --- | | | ··· | | ··· | | ··· | | | | | | | | | | | | | | | | --- | --- | Figure 1: Given the current image and the recurrent scene state, ReCal3R derives a candidate learning rate and estimates state token reliability from intermediate signals produced by the recurrent forward pass. This prevents unreliable state tokens from receiving aggressive updates and leads to cleaner reconstructions than CUT3R over long image streams. ABSTRACT Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information tobe overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive ∗ [Corresponding author: Wentao Cheng. Email: wentaocheng@bnbu.edu.cn](Corresponding author: Wentao Cheng. Email: wentaocheng@bnbu.edu.cn) updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7 × reduction in ATE, with comparable runtime and memory. Code is available at: [https://github.com/Powertony102/ReCal3R](https://github.com/Powertony102/ReCal3R) 1 INTRODUCTION Embodied agents, AR/VR systems, and mobile robots increasingly operate as continuous observers of the world: they consume image streams that are unbounded in length, arrive in real time, and must be integrated into a coherent geometric understanding on the fly. Recent feed-forward 3D geometry models such as DUSt3R (Wang et al., 2024) and VGGT (Wang et al., 2025a) have shown impressive geometry, pose, and correspondence prediction from RGB image sets. Yet they are primarily designed for finite view collections rather than online state maintenance. Streaming 3D reconstruction reframes the problem: a bounded scene representation must be continuously updated as new frames arrive, enabling linear time processing with bounded memory. Recurrent reconstruction implements this formulation with a compact latent scene state that is updated and read out online. CUT3R (Wang et al., 2025b) is the canonical recurrent state model in this space, compressing the growing history into a bounded state. However, compact recurrence alone does not guarantee accuracy over long streams. As the input sequence grows, errors can accumulate inside the state, and reliable historical content may be overwritten by noisy or ambiguous observations. The central difficulty is therefore not only how to bound the representation, but how to update the bounded state reliably over time. This limitation has motivated inference time modifications of recurrent state updates. TTT3R (Chenet al., 2025) casts the recurrent transition as Test-Time Training (Sun et al., 2025), treating the scene state as fast weights and deriving a closed-form per

FAQs

What problem does ReCal3R address in streaming 3D reconstruction?

ReCal3R targets the gradual corruption of a compact recurrent scene state during long image streams, where reliable historical information may be overwritten by noisy or ambiguous observations from repeated updates.

How does ReCal3R compute the learning rate for updating the recurrent state?

ReCal3R first derives a candidate learning rate from token alignment, state reconstruction residual, and recent update pressure, then estimates state token reliability from the maintained scene state and uses reliability to calibrate/interpolate the final token-wise learning rate.

What is the main benefit of reliability-calibrated learning rates compared with using the candidate rate directly?

The calibrated rule suppresses aggressive updates on unreliable tokens while preserving adaptation to informative frames, leading to cleaner reconstructions over long sequences with strong improvements in pose and depth quality.

Jul 16, 2026

Upload by Ezra