Sketching without Worrying - Noise-Tolerant Sketch-Based Image Retrieval

Aug 20, 2026

Sketching enables image retrieval applications, but users often fail to sketch due to the fear-to-sketch problem (“I can't sketch”), especially in fine-grained SBIR where irrelevant details can strongly hurt ranking. This work introduces a noise-tolerant pre-processing module that helps users sketch without worry by detecting and removing noisy strokes. A stroke subset selector trained with reinforcement learning quantifies stroke importance via reward from retrieval ranking, yielding 8–10% gains over baselines. The selector integrates plug-and-play with existing sketch applications.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

1. Introduction

2. Method Overview

3. Noisy Stroke Analysis

4. Reinforcement Learning for Stroke Selection

5. Experimental Results and Performance Gains

6. Plug-and-Play Applications

Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval Ayan Kumar Bhunia 1 Subhadeep Koley 1,2 Abdullah Faiz Ur Rahman Khilji* Aneeshan Sain 1,2 Pinaki Nath Chowdhury 1,2 Tao Xiang 1,2 Yi-Zhe Song 1,2 1 SketchX, CVSSP, University of Surrey, United Kingdom. 2iFlyTek-Surrey Joint Research Centre on Artiﬁcial Intelligence. fa.bhunia, s.koley, a.sain, p.chowdhury, t.xiang, [y.song](y.songg@surrey.ac.uk)[g](y.songg@surrey.ac.uk)[@surrey.ac.uk](y.songg@surrey.ac.uk) Abstract Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., “I can't sketch”) has however proven to be fatal for its widespread adoption. This paper tackles this “fear” head on, and for the ﬁrst time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We ﬁrst conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the “I can't sketch”. We consequently design a stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantiﬁes the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve asigniﬁcant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible. 1. Introduction Thanks to the convenience of interactive touchscreen devices, sketch-based image retrieval (SBIR) [11, 12, 14, 38] has emerged as a practical means of image research that is complementary to the conventional text-based retrieval [25] . Although initially developed for a category-level setting [42, 36, 59], of late SBIR has undertaken a ﬁne-grained shift to better reﬂect the inherent ﬁne-grained characteristics (pose, appearance detail, etc) of sketches [46, 56, 7] . Despite great strides made [3, 33, 10], the fear-to-sketch has proven to be fatal for its omnipresence – a “I can't sketch” reply is often the end of it. This “fear” is predominant for ﬁne-grained SBIR (FG-SBIR), where the system dictates users to produce even more faithful and diligent *Interned with SketchX Figure 1: (a) While the average ranking percentile increases as the sketching proceeds from starting towards completion, unwanted sudden drops have been noticed for many individual sketches due to noisy/irrelevant strokes drawn. (b) The same thing is visualised with number of samples in the third axis to get an overall statistics on QMUL-Shoe-V2 dataset. sketches than that required for category-level retrieval [11] . In this paper, we tackle this “fear” head-on and propose for the ﬁrst time a pre-processing module for FG-SBIRthat essentially let the users sketch without the worry of “Ican't”. We ﬁrst experimentally show that, in most cases it is not about how bad a sketch is – most can sketch (even a rough outline)– the devil lies in the fact that users typically draw irrelevant (noisy) strokes that are detrimental to the overall retrieval performance (see Section 3) . This observation has largely inspired us to alleviate the “can't sketch” problem by eliminating the noisy strokes through selecting an optimal subset that can lead to effective retrieval. This problem might sound trivial enough – e.g., how about considering all possible stroke subsets as training samples to gain model invariance against noisy strokes? Albeit theoretically possible, the highly complex nature of this process (i.e., O(2N )) quickly renders this naive solution infeasible, especially when the number of strokes in freehand sketches can range from an avera

FAQs

What problem does the paper address in sketch-based image retrieval?

It addresses the fear-to-sketch problem, where users reply “I can't sketch,” limiting adoption. It targets fine-grained SBIR by reducing the negative impact of irrelevant/noisy strokes drawn by users.

How does the proposed method handle noisy strokes?

It introduces a stroke subset selector that detects noisy strokes and keeps only strokes that positively contribute to successful retrieval. The selector outputs a binary decision per stroke to form an optimal subset.

Why is reinforcement learning used for training the stroke selector?

Rasterization is non-differentiable, preventing direct gradient-based optimization. The method uses reinforcement learning (actor-critic PPO) with reward computed using a pre-trained retrieval model’s ranking.

Aug 20, 2026

Upload by Quinn Holloway