TOLiD Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

Jul 16, 2026

Cross-modal distillation from vision foundation models (VFMs) to LiDAR backbones has emerged as a self-supervised pretraining strategy that reduces dependence on dense point-wise annotation for 3D scene understanding. Existing pipelines often freeze the VFM and train a heterogeneous 3D backbone to match fixed image embeddings, forcing the student to handle both modality and cross-architecture gaps. TOLiD addresses these gaps by coupling a LiDAR backbone with a student ViT teacher, using token-level distillation with visibility masking and token lifting to per-point representations. Evaluations on multiple datasets show improved transfer using frozen backbones and lightweight heads.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Cross-modal distillation from VFMs to LiDAR

Architectural gap and limitations

Proposed method: TOLiD

Token representation via Frustum Pooling and Frustum Attention

TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation Sutharsan Mahendran 1 ,2 , Darshana Priyasad 1 , Kaushik Roy2 , Tharindu Fernando 1 , Sridha Sridharan 1 , Clinton Fookes 1 , Peyman Moghadam 1 ,2 arXiv :2607 . 10762v1 [ cs .CV] 12 Jul 2026 Abstract—Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged asa self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding. However, existing distillation pipelines typically treat the VFM as a frozen feature source and train a heterogeneous 3D backbone to match fixed image embeddings, forcing the student to bridge both the modality gap and the cross-architecture gap between dense ViT token representations and sparse 3D encoders. We propose TOLiD, a self-supervised pretraining method for LiDAR representation learning that addresses this gap by coupling a LiDAR backbone with a student Vision Transformer (ViT) initialized from a frozen VFM teacher and applying supervision over compatible patch-token representations. TOLiD converts the set of point features within each image patch frustum into a token using Frustum Pooling followed by Frustum Attention, and performs token-level distillation with visibility masking. For LiDAR-only deployment, we lift token features back to per-point representations using masked bilinear sampling to avoid patches that have limited LiDAR points. We extensively evaluate TOLiD on five heterogeneous LiDAR datasets and four cross-sensor adaptation pairs, demonstrating improved transfer with frozen backbones and lightweight heads. I. INTRODUCTION 3D semantic segmentation is a core capability for autonomous robots and self-driving cars, enabling 3D scene understanding for navigation, mapping, and safe interaction [1]–[5] . In many robotics systems, 3D scene understanding relies primarily on LiDAR, as it provides metric geometric measurements that are largely invariant to illumination and support reliable long-range perception. LiDAR-based learning models must operate across heterogeneous sensor suites that differ in beam count, field of view, point density, and scanning pattern, and these variations induce cross-sensor domain shifts when models are transferred between platforms. Addressing such sensor gaps by collecting new annotations is often impractical, since dense point-wise labelling of LiDAR scans is labourintensive and costly [1] . While 2D perception tasks have benefited substantially from Vision Foundation Models (VFMs) pre-trained on large-scale datasets [6], [7], progress for LiDAR-based tasks has been comparatively constrained by the absence of suitable 3D foundation models and by the modality gap that 1 SAIVT Group, School of Electrical Engineering and Robotics, Queensland University of Technology, Australia. E-mails: {s2 .mahendren, s.sridharan, c.fookes, [peyman.moghadam}@qut.edu.au](peyman.moghadam}@qut.edu.au) 2CSIRO Robotics, CSIRO, Australia. E-mails:{sutharsan .mahendran, kaushik .roy, [peyman.moghadam}@csiro.au](peyman.moghadam}@csiro.au) arises when utilizing VFMs for LiDAR representations [8],[9] . To address these limitations, recent work has leveraged VFMs as teachers and distilled their representations directly to LiDAR backbones as a self-supervised pretraining step, followed by a finetuning stage that requires substantially less labeled 3D data. These VFM-to-LiDAR cross-modal distillation methods typically follow a two-stage pipeline. In the pretraining stage, synchronized camera–LiDAR pairs are used to establish 2D–3D correspondences: LiDAR points [10],[11](or superpoints [12], [13]) are projected into the image plane and associated with dense VFM features (patch tokens or intermediate feature embeddings) . Then LiDAR encoder is trained to predict or align to these VFM embeddings, often through feature regression [11], [14], or contrastive obje

FAQs

What problem does TOLiD aim to solve in VFM-to-LiDAR distillation?

TOLiD targets the fundamental limitation where current pipelines must bridge both the modality gap and the cross-architecture gap between dense ViT token representations and sparse 3D encoders. This mismatch can cause information loss and unstable transfer.

How does TOLiD convert LiDAR points within an image patch frustum into tokens?

TOLiD represents each image patch by its corresponding 3D frustum and converts the set of point features in that frustum into a single token. It uses Frustum Pooling for stable aggregation and Frustum Attention to emphasize the most informative points.

How are token features lifted back for LiDAR-only deployment?

For LiDAR-only use, TOLiD lifts token features back to per-point representations using masked bilinear sampling. This masking avoids patches with limited LiDAR points.

Jul 16, 2026

Upload by River Wang