SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction

Jul 16, 2026

Panoramic dense-prediction models for semantic segmentation and depth estimation are commonly trained with a strict gravity-aligned assumption, yet real captures often violate it due to handheld jitter and aerial or mobile attitude changes. Standard spherical Transformers can overfit global latitude cues and fail under 3D reorientation. SO3UFormer learns intrinsic spherical features largely independent of the coordinate frame by removing absolute latitude encoding, using quadrature-consistent spherical attention, and applying gauge-aware relative positional bias, reinforced by a logit-space SO(3) SO(3)-consistency regularizer during training.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

Abstract

Introduction

Gravity-alignment assumption and failure under rotation

SO3UFormer: intrinsic spherical perception

Pose35 robustness protocol

Approach and architecture details

arXiv :2602 .22867v2 [ cs .CV] 12 Jul 2026 SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction Qinfeng Zhu 1,2 , Yunxi Jiang3 , Lei Fan2* 1 University of Liverpool, United Kingdom. 2 Xi’an Jiaotong-Liverpool University, China. 3 CNRS, France. *Corresponding author(s). E-mail(s): [lei.fan@xjtlu.edu.cn](lei.fan@xjtlu.edu.cn) ; Abstract Panoramic dense-prediction models, spanning semantic segmentation and depth estimation, are typically trained under a strict gravity-aligned assumption. Real-world captures, however, routinely violate it: handheld devices jitter and aerial platforms change attitude, so the camera is rarely upright. Under such 3D reorientation, standard spherical Transformers overfit global latitude cuesand collapse. We introduce SO3UFormer, an architecture that learns intrinsic spherical features largely decoupled from the underlying coordinate frame, through three geometric components: (1) removing absolute latitude encoding, which breaks the dependence on the gravity axis; (2) quadrature-consistent spherical attention, which corrects for non-uniform sampling density; and (3) a gauge-aware relative positional bias built from local tangent-plane angles rather than global axes. A logit-space SO(3)-consistency regularizer, used only during training, further suppresses residual discretization effects. To benchmark robustness, we introduce Pose35, a variant of Stanford2D3D perturbed by random rotations within ±35◦ , and evaluate under a full, arbitrary SO(3) stress test. There, the baseline SphereUFormer collapses from 67.53 mIo U on Pose35 to 25.26 under the full SO(3) test, whereas SO3UFormer reaches 72.03 on Pose35 and retains 70.67 under the same test. Similarly, on a second real-world dataset (Matterport3D) for segmentation and on panoramic depth estimation, SO3UFormer remains essentially rotation-invariant while the gravity-anchored baseline again loses most of its accuracy. Code and models are available at [https://github.com/zhuqinfeng1999/SO3UFormer](https://github.com/zhuqinfeng1999/SO3UFormer). Keywords: Panoramic Images, Semantic Segmentation, Depth Estimation, Spherical Transformer, SO(3) Robustness, Geometric Deep Learning 1 1 Introduction Panoramic imaging offers a comprehensive 360◦ field of view [1–3], serving as a critical modality for embodied agents [4], aerial drones [5], and immersive reality applications [6] . Unlike standard perspective cameras [7], panoramic sensors capture the entire surrounding geometry in a single shot [8] . However, this completeness comes with a hidden assumption in current computer vision pipelines: canonical gravity alignment [9] . Most state-of-the-art panoramic dense-prediction models, from semantic segmentation to depth estimation [10], implicitly assume the camera is perfectly upright. While this assumption often holds for street-view datasets collected by specialized vehicles, it breaks down in dynamic capture settings such as aerial and mobile robotic platforms [11] . Drones tilt during flight, handheld devices jitter, and robots move over uneven terrain, leading to significant roll and pitch rotations [12] . The consequence of this assumption is a fragility to 3D rotations that is often masked by standard benchmarks [13] . Existing methods, whether based on Equirectangular Projection (ERP) [14] or [15], typically bake in absolute positional encodings (e.g., latitude and longitude) [16] . These encodings act as a strong prior, anchoring the semantic understanding to a global “North Pole.” When the camera rotates, the physical “floor” is no longer at the bottom of the image, yet the model, misguided by absolute coordinates, continues to look for it there. As we demonstrate in this work, this reliance causes a catastrophic performance collapse (Figure 1): a leading spherical Transformer, SphereUFormer [10], drops from 67.53% mean Intersection-over-Union (mIo U) to 25.26% under arbitrary SO(3) rot

FAQs

Why do panoramic dense-prediction models fail under arbitrary camera rotations?

They often rely on an upright, gravity-aligned assumption and absolute positional encodings like latitude, which anchor understanding to a global reference. When the camera rotates, the model continues to search for the “floor” where the global prior expects it, causing large performance collapse.

What are the core components of SO3UFormer that improve rotation robustness?

SO3UFormer removes absolute latitude encoding to break dependence on the gravity axis, introduces quadrature-consistent spherical attention to address non-uniform sampling density, and uses a gauge-aware relative positional bias based on local tangent-plane angles rather than global axes. A training-time logit-space SO(3)-consistency regularizer further suppresses residual discretization effects.

How is rotation robustness evaluated in the paper?

The paper introduces Pose35, a stress-test protocol derived from Stanford2D3D with random rotations within ±35°. It also evaluates under a full, arbitrary SO(3) stress test to measure robustness beyond standard benchmarks.

Jul 16, 2026

Upload by Lucas Martin