Li_SAS_Segment_Any_3D_Scene_with_Integrated_2D_Priors_ICCV_2025_paper

Sep 29, 2026

Upload by Dipper

Open vocabulary 3D scene understanding is increasingly important because fixed-category training cannot recognize unseen objects in complex dynamic scenes. This ICCV paper proposes SAS, which integrates multiple 2D open-vocabulary models and migrates their capability to 3D. It aligns 2D model features in a shared embedding space via text, then builds annotation-free model capabilities with diffusion-based quantification. Point-cloud features are fused using the constructed capabilities and transferred to 3D through feature distillation, outperforming prior methods on ScanNet v2, Matterport3D, and nuScenes, and validating generalizability on downstream tasks including gaussian segmentation and instance segmentation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Abstract

Introduction

This ICCV paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. SAS: Segment Any 3D Scene with Integrated 2D Priors Zhuoyuan Li 1 *, Jiahao Lu 1 *, Jiacheng Deng 1 , Hanzhi Chang 1 , Lifan Wu 1 , Yanzhe Liang 1 , Tianzhu Zhang 1 ,2† 1University of Science and Technology of China 2Deep Space Exploration Laboratory Project Page: [https://peoplelu.github.io/SAS.github.io](https://peoplelu.github.io/SAS.github.io) Figure 1 . Left: The leading 2D open vocabulary models like LSeg [40] and SEEM [100] often misidentify objects, which makes the distilled 3D model perform the same misidentification. Middle: Our proposed SAS successfully correct the misidentified object. Right: SAS distills open vocabulary knowledge from multiple 2D models with novel designs, e.g., Annotation-free Model Capability Construction. Abstract The open vocabulary capability of 3D models is increasingly valued, as traditional methods with models trained with fixed categories fail to recognize unseen objects in complex dynamic 3D scenes. In this paper, we propose a simple yet effective approach, SAS, to integrate the open vocabulary capability of multiple 2D models and migrate it to 3D domain. Specifically, we first propose Model Alignment via Text to map different 2D models into the same embedding space using text as a bridge. Then, we propose Annotation-Free Model Capability Construction to explicitly quantify the 2D model’s capability of recognizing different categories using diffusion models. Following this, point cloud features from different 2D models arefused with the guide of constructed model capabilities. Finally, the integrated 2D open vocabulary capability is transferred to 3D domain through feature distillation. SAS outperforms previous methods by a large margin across multiple datasets, including ScanNet v2, Matterport3D, and nuScenes, while its generalizability is further validated on downstream tasks, *Equal Contribution †Corresponding Author e.g., gaussian segmentation and instance segmentation. 1. Introduction 3D scene understanding is a fundamental task that aims to predict the semantics for every 3D point. Numerous realworld applications, such as autonomous driving [1, 20, 21], virtual reality [56], and robot manipulation [69], depend heavily on 3D scene understanding. Traditional methods in this area perform supervised training on labeled 3D datasets with a limited number of categories, which hinders the model’s ability to identify unseen objects. This prompts researchers to turn their attention to open vocabulary capabilities of 3D scene understanding models. In 2D open vocabulary understanding, a typical approach is to align image features to language features by contrast learning on a large number of image-text pairs, e.g., CLIP [64] . By analogy, contrast learning on a large number of point-text pairs is supposed to enable 3D open vocabulary understanding. However, point-text pairs are much harder to acquire than image-text pairs due to the high cost and time-consuming nature of point cloud annotation [18, 57] . A compromise approach is to distill the open vocabulary ca- pability of 2D open vocabulary models onto the 3D models [5, 18, 29, 57, 80, 88, 92, 98] . For example, OpenScene [57] adopts 2D open vocabulary segmentaters to extract pixel features for point clouds to enable point-language alignment. These pixel features are distilled to transfer the 2D open vocabulary capability to 3D models. Although distillation-based methods have become dominant in 3D scene understanding, the distilled model inherited its 3D open vocabulary capability from the adopted 2D models, making the performance of the distilled model vary with the performance of the adopted 2D model. As shown in Fig. 1, the 2D open vocabulary model may incorrectly recognize ob

FAQs

What problem does SAS address in open vocabulary 3D scene understanding?

Traditional fixed-category 3D models struggle to recognize unseen objects in complex scenes. Distillation from 2D open-vocabulary models can also inherit 2D misidentifications, producing ambiguous supervision for 3D learning.

How does SAS align multiple 2D open-vocabulary models for use in 3D?

SAS proposes Model Alignment via Text, mapping different 2D models into a shared embedding space using text as a bridge, so their representations become comparable.

What is Annotation-Free Model Capability Construction and how is it used?

SAS uses diffusion models to explicitly quantify a 2D model’s recognition capability across categories without requiring annotation. The constructed capabilities then guide the fusion of point-cloud features from multiple 2D models.

Sep 29, 2026

Upload by Dipper