2023

Word-As-Image for Semantic Typography

Iluz, Shir, Vinker, Yael, Hertz, Amir et al.

Understand

A word-as-image is a semantic typography technique where a word illustration presents a visualization of the meaning of the word, while also preserving its readability.

We present a method to create word-as-image illustrations automatically.
This task is highly challenging as it requires semantic understanding of the word and a creative idea of where and how to depict these semantics in a visually pleasing and legible manner.
We rely on the remarkable ability of recent large pretrained language-vision models to distill textual concepts visually.

Built on

Trick or TReAT: Thematic Reinforcement for Artistic Typography
Purva Tendulkar, Kalpesh Krishna, Ramprasaath R. Selvaraju, and Devi Parikh. 2019 · 1903
Earlier work this paper cites.
Shape skeletonization by identifying discrete local symmetries
Ju Jia Zou, Hung-Hsin Chang, and Hong Yan. 2001 · 1905
Earlier work this paper cites.
Sur la sphere vide
Boris Delaunay et al · 1934
Earlier work this paper cites.
Adobe Type 1 Font Format
Adobe Systems Inc. 1990 · 1990
Earlier work this paper cites.
A History of TrueType
Laurence Penney. 1996 · 1996
Earlier work this paper cites.
Morphological analysis of shapes
Lakshman Prasad. 1997 · 1997
Earlier work this paper cites.
MIPS: An efficient global parametrization method
Kai Hormann and Günther Greiner. 2000 · 2000
Earlier work this paper cites.
Denoising Diffusion Probabilistic Models
Original
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020 · 2006
Earlier work this paper cites.
Calligraphic Packing. In Proceedings of Graphics Interface 2007 on - GI ’07 . ACM Press, Montreal, Canada, 43
Jie Xu and Craig S. Kaplan. 2007 · 2007
Earlier work this paper cites.
FreeType library
FreeType. 2009 · 2009
Earlier work this paper cites.
Example-Based Automatic Font Generation
Rapee Suveeranont and Takeo Igarashi. 2010 · 2010
Earlier work this paper cites.
Word As Image
Ji Lee. 2011 · 2011
Earlier work this paper cites.
Learning a Manifold of Fonts
Neill DF Campbell and Jan Kautz. 2014 · 2014
Earlier work this paper cites.
Flexyfont: Learning Transferring Rules for Flexible Typeface Synthesis
Huy Quoc Phan, Hongbo Fu, and Antoni B Chan. 2015 · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention . Springer, 234–241
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015 · 2015
Earlier work this paper cites.
Legible Compact Calligrams
Changqing Zou, Junjie Cao, Warunika Ranaweera, Ibraheem Alhashim, Ping Tan, Alla Sheffer, and Hao Zhang. 2016 · 2016
Earlier work this paper cites.
Awesome Typography: Statistics-Based Text Effects Transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Shuai Yang, Jiaying Liu, Zhouhui Lian, and Zongming Guo. 2017 · 2017
Earlier work this paper cites.
Synthesizing Ornamental Typefaces: Synthesizing Ornamental Typefaces
Junsong Zhang, Yu Wang, Weiyi Xiao, and Zhenshan Luo. 2017 · 2017
Earlier work this paper cites.
Multi-Content GAN for Few-Shot Font Style Transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, Salt Lake City, UT, USA, 7564–7573
Samaneh Azadi, Matthew Fisher, Vladimir G. Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. 2018 · 2018
Earlier work this paper cites.
A Neural Representation of Sketch Drawings. In Sixth International Conference on Learning Representations (ICLR)
Original
David Ha and Douglas Eck. 2018 · 2018
Earlier work this paper cites.

Similar

EasyFont: A style learning-based system to easily build your large-scale handwriting fonts
Zhouhui Lian, Bo Zhao, Xudong Chen, and Jianguo Xiao. 2018 · 2018
Cited alongside, same era.
Context-Aware Unsupervised Text Stylization. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM ’18) . Association for Computing Machinery, New York, NY, USA, 1688–1696
Shuai Yang, Jiaying Liu, Wenhan Yang, and Zongming Guo. 2018 · 2018
Cited alongside, same era.
Learning a Stroke-Based Representation for Fonts
Elena Balashova, Amit H. Bermano, Vladimir G. Kim, Stephen DiVerdi, Aaron Hertzmann, and Thomas Funkhouser. 2019 · 2019
Cited alongside, same era.
SCFont: Structure-Guided Chinese Font Generation via Deep Stacked Networks
Yue Jiang, Zhouhui Lian, Yingmin Tang, and Jianguo Xiao. 2019 · 2019
Cited alongside, same era.
A Learned Representation for Scalable Vector Graphics. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. 2019 · 2019
Cited alongside, same era.
Typography With Decor: Intelligent Text Style Transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Wenjing Wang, Jiaying Liu, Shuai Yang, and Zongming Guo. 2019 · 2019
Cited alongside, same era.
SketchPatch: Sketch Stylization via Seamless Patch-Level Synthesis
Noa Fish, Lilach Perry, Amit Bermano, and Daniel Cohen-Or. 2020 · 2020
Cited alongside, same era.
Differentiable Vector Graphics Rasterization for Editing and Learning
Tzu-Mao Li, Michal Lukáč, Gharbi Michaël, and Jonathan Ragan-Kelley. 2020 · 2020
Cited alongside, same era.
SegDiff: Image Segmentation with Diffusion Probabilistic Models
Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. 2021 · 2021
Cited alongside, same era.
Transformer Interpretability Beyond Attention Visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 782–791
Hila Chefer, Shir Gur, and Lior Wolf. 2021 · 2021
Cited alongside, same era.
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Original
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. 2021 · 2021
Cited alongside, same era.
Clipdraw: Exploring text-to-drawing synthesis through language-image encoders
Original
Kevin Frans, Lisa B Soros, and Olaf Witkowski. 2021 · 2021
Cited alongside, same era.

Then

Blended Diffusion for Text-Driven Editing of Natural Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 18208–18218
Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022 · 2022
Later among the works it cites.
StrokeStyles: Stroke-Based Segmentation and Stylization of Fonts
Daniel Berio, Frederic Fol Leymarie, Paul Asente, and Jose Echevarria. 2022 · 2022
Later among the works it cites.
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
Prompt-to-prompt image editing with cross attention control
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models
Original
Ajay Jain, Amber Xie, and Pieter Abbeel. 2022 · 2022
Later among the works it cites.
Towards Layer-wise Image Vectorization
Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. 2022 · 2022
Later among the works it cites.
Intelligent Typography: Artistic Text Style Transfer for Complex Texture and Structure
Wendong Mao, Shuai Yang, Huihong Shi, Jiaying Liu, and Zhongfeng Wang. 2022 · 2022
Later among the works it cites.
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In International Conference on Learning Representations
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. 2022 · 2022
Later among the works it cites.
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
Dreamfusion: Text-to-3d using 2d diffusion
Original
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. 2022 · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022 · 2022
Later among the works it cites.
DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2022 · 2022
Later among the works it cites.
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022 · 2022
Later among the works it cites.
Diffusion Guided Domain Adaptation of Image Generators
Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris Metaxas, and Ahmed Elgammal. 2022 · 2022
Later among the works it cites.
Motionclip: Exposing human motion generation to clip space. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII . Springer, 358–374
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
CLIPasso: Semantically-Aware Object Sketching
Yael Vinker, Ehsan Pajouheshgar, Jessica Y. Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. 2022b · 2022
Later among the works it cites.
Diffusers: State-of-the-art diffusion models
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. 2022 · 2022
Later among the works it cites.
Aesthetic Text Logo Synthesis via Content-Aware Layout Inferring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2436–2445
Yizhi Wang, Guo Pu, Wenhan Luo, Yexin Wang, Pengfei Xiong, Hongwen Kang, and Zhouhui Lian. 2022 · 2022
Later among the works it cites.
Designing an Encoder for Fast Personalization of Text-to-Image Models
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…