Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching

Sep 11, 2026

Upload by Mabel

Self-supervised multimodal learning addresses non-rigid 3D shape matching when triangle meshes and point clouds are constrained by different strengths and limitations. The approach combines functional-map regularisation over triangle meshes with a contrastive loss that couples meshes and corresponding point clouds. It produces intramodal correspondences for full and partially observed point clouds, as well as cross-modality correspondences, improving robustness beyond either single-modality unsupervised baselines. Experiments show state-of-the-art benchmark performance and stronger cross-dataset generalisation than recent supervised methods.

LanguageEnglish

CategoryPaper Templates

FormatPDF

Pages10

Reading Time4 min

AccessFree to download

Introduction

Related Work

Method Overview

Experiments and Results

This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching Dongliang Cao Florian Bernard University of Bonn Abstract The matching of 3D shapes has been extensively studied for shapes represented as surface meshes, as well as for shapes represented as point clouds. While point clouds are a common representation of raw real-world 3D data (e.g. from laser scanners), meshes encode rich and expressive topological information, but their creation typically requires some form of (often manual) curation. In turn, methods that purely rely on point clouds are unable to meet the matching quality of mesh-based methods that utilise the additional topological structure. In this work we close this gap by introducing a self-supervised multimodal learning strategy that combines mesh-based functional map regularisation with a contrastive loss that couples mesh and point cloud data. Our shape matching approach allows to obtain intramodal correspondences for triangle meshes, complete point clouds, and partially observed point clouds, as well as correspondences across these data modalities. We demonstrate that our method achieves state-of-the-art results on several challenging benchmark datasets even in comparison to recent supervised methods, and that our method reaches previously unseen cross-dataset generalisation ability. Our code is available at [https://github.com/](https://github.com/)[ ](https://github.com/)dongliangcao/Self-Supervised-Multimodal-Shape-Matching. 1. Introduction Matching 3D shapes, i.e. finding correspondences between their parts, is a fundamental problem in computer vision and computer graphics that has a wide range of applications [11, 16, 31] . Even though it has been studied for decades [56, 57], the non-rigid shape matching problem remains highly challenging. One often faces a large variability in terms of shape deformations, or input data with severe noise and topological changes. With the recent success of deep learning, many learningbased approaches were proposed for 3D shape matching [17, 19, 28, 33] . While recent approaches demonstrate near-perfect matching accuracy without requiring ground truth annotations [8, 17], they are limited to 3D shapes represented as triangle meshes and strongly rely on clean data. Figure 1 . Left: Our method obtains accurate correspondences for triangle meshes, point clouds and even partially observed point clouds. Right: Proportion of correct keypoints (PCK) curves and mean geodesic errors (scores in the legend) on the SHREC’19 dataset [34] for meshes (solid lines) and point clouds (dashed lines) . Existing point cloud matching methods (DPC [26], green line), or mesh-based methods applied to point clouds (Deep Shells [17], red dashed line) are unable to meet the matching performance of mesh-based methods (solid lines) . In contrast, our method is multimodal and can process both meshes and point clouds, while enabling accurate shape matching with comparable performance for both modalities (blue lines) . Since point clouds are a common representation for realworld 3D data, many unsupervised learning approaches were specifically designed for point cloud matching [20, 26, 63] . These methods are often based on learning perpoint features, so that point-wise correspondences are obtained by comparing feature similarities. The learned features were shown to be robust under large shape deformations and severe noise. However, although point clouds commonly represent samples of a surface, respective topological relations are not explicitly available and thus cannot effectively be used during training. In turn, existing point cloud correspondence methods are unable to meet the matching performance of mesh-based methods, as can be seen in

FAQs

What problem does the paper address?

The paper addresses non-rigid 3D shape matching by finding correspondences between parts under large deformations, noise, and topological changes.

How does the proposed method combine meshes and point clouds?

It uses mesh functional map regularisation and a self-supervised contrastive loss that couples triangle meshes with corresponding point cloud data to learn consistent representations.

What kinds of correspondences can the method produce?

It can produce intramodal correspondences for triangle meshes and complete or partially observed point clouds, and it can also generate correspondences across these modalities.

Sep 11, 2026

Upload by Mabel