Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects

Deng Yu$^1$   Manfred Lau$^{1*}$    Lin Gao$^2$    Hongbo Fu$^{1*}$

1 School of Creative Media, City University of Hong Kong

2 Institute of Computing Technology, Chinese Academy of Science

* Corresponding author

Accepted by IEEE TVCG

[Paper & Supplemental Material]      [Dataset & Code]

Fig 1: We present a novel technique for beautifying freehand sketches of man-made objects. Each triplet contains an input sketch (left), the sketch after part beautification (middle), and the final result after structure refinement (right).

Abstract

We present a novel freehand sketch beautification method, which takes as input a freely drawn sketch of a man-made object and automatically beautifies it both geometrically and structurally. Beautifying a sketch is challenging because of its highly abstract and heavily diverse drawing manner. Existing methods are usually confined to the distribution of their limited training samples and thus cannot beautify freely drawn sketches with rich variations. To address this challenge, we adopt a divide-and-combine strategy. Specifically, we first parse an input sketch into semantic components, beautify individual components by a learned part beautification module based on part-level implicit manifolds, and then reassemble the beautified components through a structure beautification module. With this strategy, our method can go beyond the training samples and handle novel freehand sketches. We demonstrate the effectiveness of our system with extensive experiments and a perceptive study.

Interface

Fig 2: Our sketching interface.

Pipeline

Fig 3: System pipeline of sketch beautification. For an input sketch (a), we first parse it to individual part sketches (b), and then synthesize the corresponding part references (c) by retrieval and interpolation. After performing geometry beautification on the part sketches (b) towards part references (c), we obtain the new part sketches (d) with well-beautified geometry (see geometry differences between (b) and (d)). During the stage of structure beautification, we adjust the imperfect structure (notice the misalignment of chair arms) of the intermediate output (e) with the help of part-level bounding boxes (e) and generate the final beautified sketch (f). Different colors in beautified part sketches (d) indicate the different strokes. The colorful bounding boxes (e) denote the scales and spatial locations of different part sketches in the image space (256 × 256).

Interpolation with Our Sketch Implicit Representation

Fig 4: Smooth interpolation from the leftmost towards the rightmost samples with our sketch implicit representations.

Geometry Beautification

Fig 4: Pipeline for part-geometry beautification.

Structure Beautification

Fig 5: Pipeline for learning structure beautification of the part-deformed sketch.

Video

Citation

xxxxxxxxxx
@article{yu2023sketch,
  title={Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects},
  author={Yu, Deng and Lau, Manfred and Gao, Lin and Fu, Hongbo},
  journal={arXiv preprint arXiv:2306.05832},
  year={2023}
}