Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations

Jul 16, 2026

Upload by Anda

Cross-modal super-resolution (SR) on real-world misaligned data is difficult because only unlabeled low-resolution (LR) sources and high-resolution (HR) guide images are available, often with complex spatial misalignment across modalities and resolutions. Prior approaches depend on simulated training or use weaker alignment that ignores cross-modal dependencies, limiting practical results. RobSelf is a self-supervised framework that jointly optimizes a misalignment-aware feature translator and a content-aware reference filter online to produce high-resolution, high-fidelity predictions, achieving state-of-the-art accuracy and up to 15.3× faster efficiency.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Introduction

Problem background: cross-modal SR and misalignment

Proposed approach: RobSelf

Contributions and evaluation setup

Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations 18822v3 [ cs .CV] 10 Jul 2026 Xiaoyu Dong 1 , Jiahuan Li 1 ,2 , Ziteng Cui 1 , and Naoto Yokoya 1 ,2 􀀀 1 The University of Tokyo, Japan 2 RIKEN AIP, Japan Abstract. Cross-modal super-resolution (SR) on real-world misaligned data is challenging, as only unlabeled low-resolution (LR) source and high-resolution (HR) guide images with complex spatial misalignment are available. Previous methods either rely on simulated training data or adopt suboptimal alignment strategies that overlook cross-modal dependencies, limiting their practical performance. To address these issues, we propose RobSelf, a self-supervised model that jointly optimizesa misalignment-aware feature translator and a content-aware reference filter online. The translator resolves unsupervised cross-modal and crossresolution alignment via weakly-supervised, misalignment-aware translation, yielding an aligned guide feature. Guided by this feature, the filter performs reference-based discriminative self-enhancement on the source, enabling SR prediction with high resolution and high fidelity. Experiments on synthesized data and collected real-world data demonstrate that RobSelf achieves state-of-the-art performance, outperforming existing self-supervised and supervised methods. Moreover, it achieves superior efficiency, being up to 15.3 × faster than prior self-supervised methods. [https://github.com/palmdong/RobSelf](https://github.com/palmdong/RobSelf) Keywords: Self-supervised super-resolution · Multi-modal vision Fig. 1: Real-world misaligned RGB-guided depth SR (×4) . Our model achieves stateof-the-art performance, requiring no training data, ground-truth supervision, or prealignment. (a) LR source; (b) HR guide; (c) pre-aligned guide by MINIMA [36]; (d) SSGNet [40] + pre-alignment; (e) SGNet [49] + pre-alignment; (f) RobSelf-Re (Ours) . arXiv :2602 . 2 X. Dong et al. 1 Introduction Multi-modal images, e.g ., RGB, depth, and near-infrared (NIR), capture complementary properties of objects and environments [11,57,66,67] . However, nonvisible modalities generally suffer from lower spatial resolution than RGB due to sensing and hardware limits, which hinders downstream tasks and necessitates cross-modal super-resolution (SR) [56, 71, 72] . Cross-modal SR enhances a low-resolution (LR) source image using structural cues from a high-resolution (HR) guide image of another modality. Modern methods fall into supervised and self-supervised categories. Supervised methods rely on large-scale domain-specific training data and ground truth [19, 47, 49, 50, 71, 72] . This reliance hinders their practical generalization, as building such datasets is costly and labor-intensive. Self-supervised methods require neither training data nor ground truth, but instead optimize online on each test pair, making them more data-efficient and generalizable [12, 30, 38, 40] . While promising progress has been achieved, most existing methods assume that the source and guide images are well-aligned [12,19,30,40,47,49,50,71,72] . In real-world scenarios, however, spatial misalignment is inevitable in multimodal images due to inherent cross-sensor discrepancies [28,51](e.g ., differences in lens distortion, field of view, and physical position) and environmental factors, such as platform-induced viewpoint variation [5,21] and object motion overtime [27, 46] . Although several methods consider misalignment, they either rely on simulated training data [48] or adopt suboptimal alignment strategies [15,38], limiting their practical performance. We identify two difficulties in cross-modal SR on real-world misaligned data:(I) The scarcity of training data and the absence of SR ground truth for the source [12, 30]; (II) Complex misalignments across modalities and resolutions, along with the lack of alignment ground truth for the guide [7,61] . These hinder the development of reliab

FAQs

What makes cross-modal super-resolution on real-world misaligned data challenging?

Real-world settings provide only unlabeled LR source images and HR guide images, while spatial misalignment across modalities and resolutions is complex and often lacks alignment ground truth. This reduces the effectiveness of both supervised and self-supervised methods.

How does RobSelf address cross-modal and cross-resolution misalignment without ground-truth supervision?

RobSelf uses a misalignment-aware feature translator that performs weakly-supervised, misalignment-aware translation to generate an aligned guide feature, and a content-aware reference filter that performs reference-based discriminative self-enhancement on the source guided by that feature.

What evidence shows that RobSelf works well on complex real-world misaligned observations?

Experiments on both synthesized data and collected real-world RGB-depth and RGB-NIR data demonstrate state-of-the-art performance. The method also shows superior efficiency, up to 15.3× faster than prior self-supervised methods.

Jul 16, 2026

Upload by Anda