CLIPasso: Semantically Aware Object Sketching
SIGGRAPH• 2022
Abstract
Abstraction is at the heart of sketching due to the simple and minimal nature
of line drawings. Abstraction entails identifying the essential visual
properties of an object or scene, which requires semantic understanding and
prior knowledge of high-level concepts. Abstract depictions are therefore
challenging for artists, and even more so for machines. We present CLIPasso, an
object sketching method that can achieve different levels of abstraction,
guided by geometric and semantic simplifications. While sketch generation
methods often rely on explicit sketch datasets for training, we utilize the
remarkable ability of CLIP (Contrastive-Language-Image-Pretraining) to distill
semantic concepts from sketches and images alike. We define a sketch as a set
of B\'ezier curves and use a differentiable rasterizer to optimize the
parameters of the curves directly with respect to a CLIP-based perceptual loss.
The abstraction degree is controlled by varying the number of strokes. The
generated sketches demonstrate multiple levels of abstraction while maintaining
recognizability, underlying structure, and essential visual components of the
subject drawn.