Simon-SR Spatially Adaptive Modulation and Visual Prompt Adaptation for Text-Reinforced Super-Resolution

Jul 16, 2026

Single Image Super-Resolution (SISR) reconstructs high-quality images from low-resolution inputs, but existing approaches can produce overly smooth results and remain sensitive to incorrect priors and costly annotations. Simon-SR introduces a multi-modal framework that treats text as latent, learnable semantic variables. It combines Contrastive Prompt Learning with Prompt-Guided Spatially Adaptive Refinement for robust text-image fusion and improved multi-modal alignment. Experiments show state-of-the-art gains, reaching up to 0.50 dB in PSNR, 0.0133 in SSIM, and 0.0695 in LPIPS.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

Related Works

From Single-Modal to Prior-Guided Multi-Modal

Text-Driven Super-Resolution

Proposed Methods

Overview

arXiv :2607 .09351v1 [ cs .CV] 10 Jul 2026 Simon-SR: Spatially Adaptive Modulation and Visual Prompt Adaptation for Text-Reinforced Super-Resolution H. Cheng† X. Li† Z. Cui† L. Tan and C. Wang College of Electronic Science and Engineering, Jilin University, Changchun China {chenght9923, yxli1923, cuizj1923, tanrl1923, [cywang1923}@mails.jlu.edu.cn](cywang1923}@mails.jlu.edu.cn) Abstract Single Image Super-Resolution (SISR) reconstructs high-quality images from low-resolution inputs. While recent multi-modal methods improve perceptual quality, they remain sensitive to erroneous priors and require expensive annotations. To address these issues, we propose Simon-SR, a multi-modal SISR framework leveraging learnable prompts for efficient semantic mining and robust text-image fusion. Our approach combines Contrastive Prompt Learning with Prompt-Guided Spatially Adaptive Refinement to enhance multi-modal alignment. Experiments demonstrate that Simon-SR surpasses state-of-the-art methods, achieving maximum improvements of 0.50 dB in PSNR, 0.0133 in SSIM, and 0.0695 in LPIPS. Code-will-be-released CCS Concepts • Computing methodologies → Reconstruction; Image processing; • Theory of computation → Design and analysis of algorithms; 1. Introduction Super-Resolution (SR) aims to reconstruct high-quality images from low-resolution inputs. While deep learning has significantly improved SR performance, the ill-posed nature of SR often yields overly smooth outputs, especially at extreme downsampling rates (e.g., ×16) . To enhance perceptual quality, existing single-modal methods are typified by adversarial architectures [WYW∗18] while the multi-modal ones generally leverage textual semantics as priors [QYZ∗24] . Recent breakthrough of pre-trained multi-modal large language models reveals the potential of textual semantics for image restoration [QYZ∗24] . However, they suffer from sensitivity to erroneous priors and substantial annotation overhead. Moreover, existing multi-modal methods struggle with insufficient attention to critical details due to semantic biases during text-image fusion, as demonstrated in Figure 1(b) . To this end, we propose a novel multi-modal super-resolution framework termed Spatially Adaptive Modulation and Visual Prompt Adaptation (Simon-SR) . Existing text-driven SR methods assume texts as ground-truth semantic priors, whereas Simon-SR treats texts as latent, learnable semantic variables jointly optimized with image restoration. As illustrated in Figure 1 (c), our method efficiently extracts textual features with minimal computational overhead while adaptively modulating image features. † Equal Contribution Figure 1: (a) Existing single-modal methods fail at extreme downsampling rates (e.g., ×16). (b) Existing multi-modal models suffer from sub-optimal fusion strategies and text bias. (c) The proposed learnable prompts for textual semantic mining reduce annotation cost, mitigate prior bias, and enhance detail recovery. In summary, our contributions are threefold: (1) We propose a learnable prompt-based approach for SR that extracts textual semantics from unannotated images, effectively © 2026 The Author(s) . Proceedings published by Eurographics-The European Association for Computer Graphics. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. 2 of 5 H. Cheng & S. Behnke / EG LATEX Author Guidelines Figure 2: Overview of Simon-SR framework. Given the input low-resolution image ILRx and pre-trained prompts, two stages operate sequentially. (a) Contrastive Prompt Learning extracts learnable textual semantics from unannotated images based on frozen CLIP encoders. Then, the optimized prompts are passed to (b) Prompt-Guided Spatially Adaptive Refinement, where text–image fusion is conducted via PTRBlocks. Spatially adaptive affine

FAQs

What problem does Simon-SR address in single image super-resolution?

Simon-SR targets the ill-posed nature of super-resolution that often yields overly smooth outputs, especially under extreme downsampling, and the sensitivity of multimodal methods to erroneous textual priors and annotation cost.

How does Simon-SR use textual information differently from prior text-driven SR methods?

Instead of treating text as fixed ground-truth semantic priors, Simon-SR models text as latent, learnable semantic variables jointly optimized with image restoration to reduce bias.

What are the two main stages of the Simon-SR framework?

The first stage, Contrastive Prompt Learning, extracts learnable textual semantics from unannotated images using frozen CLIP encoders. The second stage, Prompt-Guided Spatially Adaptive Refinement, performs text-image fusion through PTRBlocks with spatially adaptive affine refinements during iterative improvement.

Jul 16, 2026

Upload by Clementine