Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting

Oct 24, 2026

Upload by Franzy

This ICCV paper studies 2D sketch to 3D shape retrieval with an explicit goal of democratising the process. It removes reliance on large, specifically sourced 2D sketch and 3D shape datasets, and reduces constraints on users’ sketch quality and viewpoint angle. The proposed system is trainable on existing datasets and enables sketching without drawing-skill or view restrictions through feature pivoting and 3D-aware injection.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Abstract

Introduction

Motivation and problem setting

Pivoting across domains

Approach overview

This ICCV paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting Pinaki Nath Chowdhury 1,2 Ayan Kumar Bhunia 1 Aneeshan Sain 1,2 Subhadeep Koley 1,2 Tao Xiang 1,2 Yi-Zhe Song 1,2 1 SketchX, CVSSP, University of Surrey, United Kingdom. 2iFlyTek-Surrey Joint Research Centre on Artificial Intelligence. {p.chowdhury, a.bhunia, a.sain, s.koley, t.xiang, [y.song](y.song}@surrey.ac.uk)[}](y.song}@surrey.ac.uk)[@surrey.ac.uk](y.song}@surrey.ac.uk) Abstract This paper studies the problem of 2D sketch to 3D shape retrieval, but with a focus on democratising the process. We would like this democratisation to happen on two fronts:(i) to remove the need for large-scale specifically sourced 2D sketch and 3D shape datasets, and (ii) to remove restrictions on how well the user needs to sketch and from what viewpoints. The end result is a system that is trainable using existing datasets, and once trained allows users to sketch regardless of drawing skills and without restriction on view angle. We achieve all this via a clever use of pivoting, along with novel designs that injects 3D understanding of 2D sketches into the system. We perform pivoting using two existing datasets, each from a distant research domain to the other: 2D sketch and photo pairs from the sketch-based image retrieval field (SBIR), and 3D shapes from ShapeNet. It follows that the actual feature pivoting happens on photos from the former and 2D projections from the latter. Doing this already achieves most of our democratisation challenge – the level of 2D sketch abstraction embedded in SBIR dataset offers demoralization on drawing quality, and the whole thing works without a specifically sourced 2D sketch and 3D model pair. To further achieve democratisation on sketching viewpoint, we“lift” 2D sketches to 3D space using Blind Perspective-nPoints (BPnP) that injects 3D-aware information into the sketch encoder. Results show ours achieves competitive performance compared with fully-supervised baselines, while meeting all set democratisation goals. 1. Introduction Sketches are highly expressive [37] . This particular sketch strait has been explored to a large extend for image retrieval, especially under the fine-grained setting. The 3D literature followed a similar trend, starting with categorylevel retrieval of 3D shapes using sketches [45], but only very recently moved onto the fine-grained instance-level Using pa ired photos Figure 1 . Collecting 2D sketches for 3D shapes is difficult since the viewpoints often needs to be pre-defined that arguably makes sketch collection for 3D shapes an ill-posed problem to start with – there is no one good view that caters for everyone. We remove the need for large-scale 2D sketch and 3D shape datasets through a clever use of pivoting across two separate domains (i) paired 2D freehand sketches from well-studied FG-SBIR literature [91], and (ii) the 3D shape domain, where we have ample data at our dispoal (e.g., ShapeNet [11]) . Our pivoting happens on paired 2D photos from FG-SBIR and 2D projections from 3D shapes. setup [58] . The very reason behind such a time lag lies with that of data collection – just as how collection of 2D sketches for images was difficult [70, 18, 66, 43], sketches for 3D models had been proven to be even harder to collect [47, 58] . In particular, since one is collecting a 2D sketch for a 3D model, viewpoints often need to be pre-defined [95], which arguably makes sketch collection for 3D shapesan ill-posed problem to start with – there is no one good view that caters for everyone. There is however a strong voice in the 3D community stating the opposite – that different to images, sketches can be freely rendered for 3D models, other than collected by hand. T

FAQs

What is the core problem addressed in the paper?

The paper addresses 2D sketch to 3D shape retrieval, aiming to make the process more accessible and less constrained for users.

How does the proposed method achieve democratisation?

It removes the need for specially sourced 2D sketch and 3D shape pairs, and allows users to sketch with fewer restrictions on drawing skill and viewpoint angle.

What datasets are used for the pivoting strategy?

The method performs pivoting using paired 2D photo information from FG-SBIR-style sketch-and-photo pairs and 3D shapes from ShapeNet.

Oct 24, 2026

Upload by Franzy