2022

CLIPasso: Semantically-Aware Object Sketching

Vinker, Yael, Pajouheshgar, Ehsan, Bo, Jessica Y. et al.

Understand

Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings.

Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts.
Abstract depictions are therefore challenging for artists, and even more so for machines.
We present CLIPasso, an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications.

Built on

A computational approach to edge detection
John Canny · 1986
Earlier work this paper cites.
Example-based facial sketch generation with non-parametric sampling
Hong Chen, Ying-Qing Xu, Heung-Yeung Shum, Song-Chun Zhu, and Nan-Ning Zheng · 2001
Earlier work this paper cites.
A survey of stroke-based rendering
A. Hertzmann · 2003
Earlier work this paper cites.
Contour detection and hierarchical image segmentation
Pablo Arbeláez, Michael Maire, Charless Fowlkes, and Jitendra Malik · 2011
Earlier work this paper cites.
Xdog: An extended difference-of-gaussians compendium including advanced image stylization
Holger Winnemöller, Jan Eric Kyprianidis, and Sven C. Olsen · 2012
Earlier work this paper cites.
Style and abstraction in portrait sketching
Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins · 2013
Earlier work this paper cites.
Free-hand sketch synthesis with deformable stroke models
Original
Yi Li, Yi-Zhe Song, Timothy M. Hospedales, and Shaogang Gong · 2015
Earlier work this paper cites.
The genesis of errors in drawing
Rebecca Chamberlain and Johan Wagemans · 2016
Earlier work this paper cites.
The sketchy database: Learning to retrieve badly drawn bunnies
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays · 2016
Earlier work this paper cites.
Sketch-pix2seq: a model to generate sketches of multiple categories
Original
Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu · 2017
Earlier work this paper cites.
A neural representation of sketch drawings
Original
David Ha and Douglas Eck · 2017
Earlier work this paper cites.
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros · 2017
Earlier work this paper cites.
Benchmarking non-photorealistic rendering of portraits
Paul L. Rosin, David Mould, Itamar Berger, John P. Collomosse, Yu-Kun Lai, Chuan Li, Hua Li, Ariel Shamir, Michael Wand, T. Wang, and Holger Winnemöller · 2017
Earlier work this paper cites.
High-resolution image synthesis and semantic manipulation with conditional gans
Original
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro · 2017
Earlier work this paper cites.
Line drawings from 3d models
Original
Pierre Bénard and Aaron Hertzmann · 2018
Earlier work this paper cites.

Similar

Common object representations for visual production and recognition
Judith E. Fan, Daniel L. K. Yamins, and Nicholas B. Turk-Browne · 2018
Cited alongside, same era.
Synthesizing programs for images using reinforced adversarial learning
Original
Yaroslav Ganin, Tejas D. Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals · 2018
Cited alongside, same era.
Learning deep sketch abstraction
Original
Umar Riaz Muhammad, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales · 2018
Cited alongside, same era.
Learning to sketch with shortcut cycle consistency, 2018
Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales · 2018
Cited alongside, same era.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang · 2018
Cited alongside, same era.
Learning to sketch with deep q networks and demonstrated strokes
Original
Tao Zhou, Chen Fang, Zhaowen Wang, Jimei Yang, Byungmoon Kim, Zhili Chen, Jonathan Brandt, and Demetri Terzopoulos · 2018
Cited alongside, same era.
Opensketch: a richly-annotated dataset of product design sketches
Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia C. Pont, Frédo Durand, and Adrien Bousseau · 2019
Cited alongside, same era.
Photo-sketching: Inferring contour drawings from images, 2019
Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan · 2019
Cited alongside, same era.
Unsupervised doodling and painting with improved SPIRAL
Original
John F. J. Mellor, Eunbyung Park, Yaroslav Ganin, Igor Babuschkin, Tejas Kulkarni, Dan Rosenbaum, Andy Ballard, Theophane Weber, Oriol Vinyals, and S. M. Ali Eslami · 2019
Cited alongside, same era.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala · 2019
Cited alongside, same era.
Strokenet: A neural painting environment
N. Zheng, Yf Jiang, and Ding jiang Huang · 2019
Cited alongside, same era.
Pixelor: a competitive sketching ai agent. so you think you can sketch?
Ayan Kumar Bhunia, Ayan Das, Umar Riaz Muhammad, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song · 2020
Cited alongside, same era.

Then

Differentiable vector graphics rasterization for editing and learning
Tzu-Mao Li, Michal Lukáč, Gharbi Michaël, and Jonathan Ragan-Kelley · 2020
Later among the works it cites.
Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt
Hangyu Lin, Yanwei Fu, Yu-Gang Jiang, and X. Xue · 2020
Later among the works it cites.
Sketchformer: Transformer-based representation for sketched structure
Leo Sampaio Ferraz Ribeiro, Tu Bui, John P. Collomosse, and Moacir Antonelli Ponti · 2020
Later among the works it cites.
Deep learning for free-hand sketch: A survey and a toolbox, 2020
Peng Xu, Timothy M. Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang · 2020
Later among the works it cites.
Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers
Hila Chefer, Shir Gur, and Lior Wolf · 2021
Later among the works it cites.
Clipdraw: Exploring text-to-drawing synthesis through language-image encoders
Original
Kevin Frans, Lisa B. Soros, and Olaf Witkowski · 2021
Later among the works it cites.
Multimodal neurons in artificial neural networks
Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah · 2021
Later among the works it cites.
The role of edges in line drawing perception
Aaron Hertzmann · 2021
Later among the works it cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai · 2021
Later among the works it cites.
Differentiable drawing and sketching
Original
Daniela Mihai and Jonathon S. Hare · 2021
Later among the works it cites.
General virtual sketching framework for vector line art
Haoran Mo, Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei Wang · 2021
Later among the works it cites.
Sketchlattice: Latticed representation for sketch manipulation
Original
Yonggang Qi, Guoyao Su, Pinaki Nath Chowdhury, Mingkang Li, and Yi-Zhe Song · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever · 2021
Later among the works it cites.
Modern evolution strategies for creativity: Fitting concrete images and abstract concepts
Original
Yingtao Tian and David Ha · 2021
Later among the works it cites.
Teaching gans to sketch in vector format
V Varshaneya, Sangeetha Balasubramanian, and Vineeth N. Balasubramanian · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…