SketchDesc: Learning Local Sketch Descriptors for Multi-View Correspondence

Deng Yu $^1$ Lei Li $^2$ Youyi Zheng $^3$ Manfred Lau $^1$ Yi-Zhe Song $^4$ Chiew-Lan Tai $^2$ Hongbo Fu $^{1*}$

1 School of Creative Media, City University of Hong Kong

2 Department of Computer Science and Engineering, HKUST

3 State Key Lab of CAD&CG, Zhejiang University

4 SketchX, CVSSP, University of Surrey

* Corresponding author

Accepted by IEEE TCSVT

[Paper & Supplemental Material] [Dataset & Code]

teaser

Fig 1: Multi-view sketch correspondence matching by our SketchDesc on product sketches in OpenSketch dataset.

Abstract

In this paper, we study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object and predict as output the semantic correspondence among the sketches. This problem is challenging since the visual features of corresponding points at different views can be very different. To this end, we take a deep learning approach and learn a novel local sketch descriptor from data. We contribute a training dataset by generating the pixel-level correspondence for the multi-view line drawings synthesized from 3D shapes. To handle the sparsity and ambiguity of sketches, we design a novel multi-branch neural network that integrates a patch-based representation and a multi-scale strategy to learn the pixel-level correspondence among multi-view sketches. We demonstrate the effectiveness of our proposed approach with extensive experiments on hand-drawn sketches and multi-view line drawings rendered from multiple 3D shape datasets.

Network Pipeline

network

Fig. 2:SketchDesc $\times$ $\times$ $\times$ $\times$ $\times$ 32. Given the multi-scale patches, we design a multi-branch framework with shared weights to take as input these rescaled patches. The dashed lines represent the data flow from an input patch to an output descriptor. For t he kernel size and stride in our network, we adopt the same settings as L2Net. Finally, the output as a 128-D descriptor embeds features from the four scales by concatenation and full connection operations.

Multi-view Sketch Correspondence

opensketch

Fig. 3: Sketch correspondence in the OpenSketch dataset computed with SketchDesc descriptors.

Visualization of the Computed Distance Map

distance map

more matching_reults

Fig. 4: For a given pixel inside a sketched object under View 1, we find a corresponding point in the other sketch under View 2 by computing a distance map through our learned descriptor.

Applications:

Sketch Segmentation Transfer

segmentation_transfer

Fig. 5: Sketch segmentation transfer. The top row are the inputs: one segmented sketch and several unlabeled sketches. The bottom row are the outputs.

Multi-view Image-based Correspondence

image_correspondence

Fig. 6: Correspondence matching among multi-view images of different objects.

Citation


xxxxxxxxxx
@article{yu2020sketchdesc,
  title={SketchDesc: Learning local sketch descriptors for multi-view correspondence},
  author={Yu, Deng and Li, Lei and Zheng, Youyi and Lau, Manfred and Song, Yi-Zhe and Tai, Chiew-Lan and Fu, Hongbo},
  journal={IEEE Transactions on Circuits and Systems for Video Technology},
  volume={31},
  number={5},
  pages={1738--1750},
  year={2020},
  publisher={IEEE}
}