HyFL-CLIP Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

Jul 16, 2026

Upload by Himbo

HyFL-CLIP introduces a hyperbolic fine-tuning framework for CLIP-style vision-language alignment under long-context descriptions. CLIP struggles with text over 77 tokens because absolute positional encoding and short-caption pretraining limit robustness to sentence reordering, summarization, and omission. The method addresses this by distilling Euclidean CLIP alignment into hyperbolic space using cross-manifold similarity distillation, enabling hierarchical and entailment modeling via hyperbolic entailment and Einstein midpoint aggregation. Experiments across long-context retrieval and perturbation benchmarks show up to 19.5% improvement, and the approach integrates into SDXL.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages48

Reading Time2 h 1 min

Abstract

Introduction

Robust Long-Context Retrieval under Caption Perturbation

arXiv :2607 .00428v 1 [ cs .CV] 1 Jul 2026 HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding Ji Ha Jang 1 ,∗, Hayeon Kim 1 ,∗, Chulwon Lee2, Junghun James Kim2, Se Young Chun 1 ,2 ,3 ,† 1 Dept. of Electrical and Computer Engineering, 2 IPAI, 3 INMC & AIIS, Seoul National University, Republic of Korea {jeeit17, khy5630, chul0e, jonghean12, [sychun}@snu.ac.kr](sychun}@snu.ac.kr) [Abstract.](Abstract. CLIP)[ CLIP](Abstract. CLIP) ([Contrastive Language-Image Pre-training](Contrastive Language-Image Pre-training)) [has be](has be)come a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (> 77 tokens) due to absolute positional encoding and pretraining on short captions. In long contexts, sentences are often reordered, summarized, or partially omitted. Although prior works extend CLIP with longer positional encodings, they often suffer from degraded image-text alignment under such text perturbations. We attribute this limitation to the Euclidean contrastive objective, which enforces strict one-to-one matching and lacks explicit mechanisms for modeling hierarchical relationships between global context and its constituent elements. To address this issue, we propose HyFL-CLIP, a hyperbolic fine-tuning framework that distills the well-established textimage alignment learned in Euclidean CLIP into hyperbolic space viacross-manifold similarity distillation, leveraging its geometry to capture hierarchical and entailment relations. Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part–whole relationships via hyperbolic entailment with Einstein midpoint aggregation. Experiments on diverse benchmarks, including longcontext cross-modal retrieval, cross-modal retrieval with caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval, show that HyFL-CLIP achieves more robust long-context understanding. In particular, it yields up to 19 .5% improvement in long-text crossmodal retrieval under textual perturbations over the best prior method. We also show HyFL-CLIP can be seamlessly integrated into other model frameworks by applying it to Stable Diffusion XL (SDXL) . The project page is available at [https://janeyeon.github.io/hyflclip](https://janeyeon.github.io/hyflclip). Keywords: hyperbolic representation learning · long-context vision-language alignment · cross-manifold distillation 1 Introduction Vision-language contrastive pre-training has laid the foundation for a wide range of vision-language learning tasks. Models such as CLIP [59], ALIGN [29] have * Authors contributed equally. † Corresponding author. 2 JH Jang, H Kim et al. Robust Long-Context Retrieval under Caption Perturbation Long text query In the foreground, there's a pedestrian crossing the street marked black car in the crosswalk. The traffic light for pedestrians is visible, displaying a red hand signal. To the left, another pedestrian appears to be wearing a white and red outfit crossing the street. Buildings with varying facades line the street, and clear blue skies with scattered clouds are above. Random subsampling To the left, another pedestrian appears to be wearing a white and red outfit crossing the street. The traffic light for pedestrians is visible, displaying a red hand signal. Order shuffling The traffic light for pedestrians is visible, displaying a red hand signal. To the left, another pedestrian appears to be wearing a white and red outfit crossing the street. In the foreground, there's a pedestrian crossing the street marked black car in the crosswalk. Buildings with varying facades line the street, and clear blue skies with scattered clouds are above. Word dropout In the foreground, there's a the street marked black car in the crosswalk. The traffic light for is visible, displaying a re

FAQs

Why does CLIP perform poorly on long-context image-text alignment?

CLIP is primarily trained on short captions and relies on absolute positional encoding, which becomes unstable for text longer than 77 tokens. Long-context descriptions often get reordered, summarized, or partially omitted, breaking strict one-to-one alignment.

What is HyFL-CLIP and how does it improve robustness?

HyFL-CLIP distills Euclidean CLIP alignment into hyperbolic space via cross-manifold similarity distillation. In hyperbolic geometry, the model captures hierarchical and entailment relations and part–whole structure using hyperbolic entailment with Einstein midpoint aggregation.

How effective is HyFL-CLIP and can it be used with other frameworks?

Experiments on multiple long-context and perturbation benchmarks show improved robustness, including up to 19.5% gains in long-text cross-modal retrieval under textual perturbations. It can also be applied to Stable Diffusion XL (SDXL) for seamless integration.

Jul 16, 2026

Upload by Himbo