2023

Mesh Neural Cellular Automata

Pajouheshgar, Ehsan, Xu, Yitao, Mordvintsev, Alexander et al.

Understand

Texture modeling and synthesis are essential for enhancing the realism of virtual environments.

Methods that directly synthesize textures in 3D offer distinct advantages to the UV-mapping-based methods as they can create seamless textures and align more closely with the ways textures form in nature.
We propose Mesh Neural Cellular Automata (MeshNCA), a method that directly synthesizes dynamic textures on 3D meshes without requiring any UV maps.
MeshNCA is a generalized type of cellular automata that can operate on a set of cells arranged on non-grid structures such as the vertices of a 3D mesh.

Built on

The Chemical Basis of Morphogenesis
AM Turing. 1952 · 1952
Earlier work this paper cites.
Theory of self-reproducing automata
John von Neumann. 1966 · 1966
Earlier work this paper cites.
Solid texturing of complex surfaces. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques . 279–286
Darwyn R Peachey. 1985 · 1985
Earlier work this paper cites.
An image synthesizer
Ken Perlin. 1985 · 1985
Earlier work this paper cites.
Generating textures on arbitrary surfaces using reaction-diffusion
Greg Turk. 1991 · 1991
Earlier work this paper cites.
Cellular texture generation. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques . 239–248
Kurt W Fleischer, David H Laidlaw, Bena L Currin, and Alan H Barr. 1995 · 1995
Earlier work this paper cites.
Texture synthesis by non-parametric sampling. In Proceedings of the Seventh IEEE International Conference on Computer Vision , Vol. 2. IEEE, 1033–1038
Alexei A Efros and Thomas K Leung. 1999 · 1999
Earlier work this paper cites.
3D surface cellular automata and their applications
Stéphane Gobron and Norishige Chiba. 1999 · 1999
Earlier work this paper cites.
Texture synthesis on surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques . 347–354
Greg Turk. 2001 · 2001
Earlier work this paper cites.
Texture synthesis over arbitrary manifold surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques . 355–360
Li-Yi Wei and Marc Levoy. 2001 · 2001
Earlier work this paper cites.
Texture and shape synthesis on surfaces. In Rendering Techniques 2001: Proceedings of the Eurographics Workshop in London, United Kingdom, June 25–27, 2001 12 . Springer, 301–312
Lexing Ying, Aaron Hertzmann, Henning Biermann, and Denis Zorin. 2001 · 2001
Earlier work this paper cites.
Spherical harmonic lighting: The gritty details. In Archives of the game developers conference , Vol. 56. 4
Robin Green. 2003 · 2003
Earlier work this paper cites.
Synthesis of progressively-variant textures on arbitrary surfaces
Jingdan Zhang, Kun Zhou, Luiz Velho, Baining Guo, and Heung-Yeung Shum. 2003 · 2003
Earlier work this paper cites.
Fast example-based surface texture synthesis via discrete optimization
Jianwei Han, Kun Zhou, Li-Yi Wei, Minmin Gong, Hujun Bao, Xinming Zhang, and Baining Guo. 2006 · 2006
Earlier work this paper cites.
Appearance-Space Texture Synthesis
Sylvain Lefebvre and Hugues Hoppe. 2006 · 2006
Earlier work this paper cites.
Solid Texture Synthesis from 2D Exemplars
Johannes Kopf, Chi-Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, and Tien-Tsin Wong. 2007 · 2007
Earlier work this paper cites.
A survey of aging and weathering phenomena in computer graphics
Stéphane Mérillou and Djamchid Ghazanfarpour. 2008 · 2008
Earlier work this paper cites.
Spectral geometry processing with manifold harmonics. In Computer Graphics Forum , Vol. 27. Wiley Online Library, 251–260
Bruno Vallet and Bruno Lévy. 2008 · 2008
Earlier work this paper cites.
A database and evaluation methodology for optical flow
Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. 2011 · 2011
Earlier work this paper cites.
Markov random fields for super-resolution and texture synthesis
WT Freeman and Ce Liu. 2011 · 2011
Earlier work this paper cites.
ShapeNet: An Information-Rich 3D Model Repository
Original
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015 · 2015
Earlier work this paper cites.

Similar

Texture synthesis using convolutional neural networks
Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015 · 2015
Cited alongside, same era.
Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Yoshua Bengio and Yann LeCun (Eds.)
Original
Karen Simonyan and Andrew Zisserman. 2015 · 2015
Cited alongside, same era.
Learn OpenGL - PBR
Joey de Vries. 2017 · 2017
Cited alongside, same era.
Single-image svbrdf capture with a rendering-aware deep network
Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018 · 2018
Cited alongside, same era.
Differentiable image parameterizations
Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. 2018 · 2018
Cited alongside, same era.
Two-stream convolutional networks for dynamic texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6703–6712
Matthew Tesfaldet, Marcus A Brubaker, and Konstantinos G Derpanis. 2018 · 2018
Cited alongside, same era.
Learning to predict 3d objects with an interpolation-based differentiable renderer
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019 · 2019
Cited alongside, same era.
A novel framework for inverse procedural texture modeling
Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019 · 2019
Cited alongside, same era.
Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10051–10060
Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019 · 2019
Cited alongside, same era.
Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 4531–4540
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. 2019 · 2019
Cited alongside, same era.
Free seamless PBR textures with Diffuse, Normal, Displacement, Occlusion and Roughness Maps
Joao Paulo. 2019 · 2019
Cited alongside, same era.
Rethinking texture mapping. In Computer graphics forum , Vol. 38. Wiley Online Library, 535–551
Cem Yuksel, Sylvain Lefebvre, and Marco Tarini. 2019 · 2019
Cited alongside, same era.

Then

PyMeshLab
Alessandro Muntoni and Paolo Cignoni. 2021 · 2021
Later among the works it cites.
Self-organising textures
Eyvind Niklasson, Alexander Mordvintsev, Ettore Randazzo, and Michael Levin. 2021 · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision. In International conference on machine learning . PMLR, 8748–8763
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Later among the works it cites.
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs
Original
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021 · 2021
Later among the works it cites.
StyleGAN-NADA: CLIP-guided domain adaptation of image generators
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
MatFormer: A Generative Model for Procedural Materials
Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomír Mĕch, Tamy Boubekeur, and Niloy J Mitra. 2022 · 2022
Later among the works it cites.
ReLU Fields: The Little Non-Linearity That Could. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22) . Association for Computing Machinery, New York, NY, USA, Article 27, 9 pages
Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. 2022 · 2022
Later among the works it cites.
Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition
Jiabao Lei, Yabin Zhang, Kui Jia, et al · 2022
Later among the works it cites.
Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13492–13502
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2022 · 2022
Later among the works it cites.
Differentiable Programming of Reaction-Diffusion Patterns. In ALIFE 2022: The 2022 Conference on Artificial Life . MIT Press
Alexander Mordvintsev, Ettore Randazzo, and Eyvind Niklasson. 2021 · 2022
Later among the works it cites.
Laion-5b: An open large-scale dataset for training next generation image-text models
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al · 2022
Later among the works it cites.
Texturify: Generating textures on 3d shape surfaces. In European Conference on Computer Vision . Springer, 72–88
Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. 2022 · 2022
Later among the works it cites.
CLIPTexture: Text-Driven Texture Synthesis. In Proceedings of the 30th ACM International Conference on Multimedia . 5468–5476
Yiren Song. 2022 · 2022
Later among the works it cites.
Generating procedural 3D materials from images using neural networks. In Proceedings of the 2022 4th International Conference on Image, Video and Signal Processing . 32–40
Lyne P Tchapmi, Trishiet Ray, Micael Tchapmi, Bokui Shen, Roberto Martin-Martin, and Silvio Savarese. 2022 · 2022
Later among the works it cites.
Mesh2Tex: Generating Mesh Textures from Image Queries
Original
Alexey Bokhovkin, Shubham Tulsiani, and Angela Dai. 2023 · 2023
Closest in time.
Texture Generation Using a Graph Generative Adversarial Network and Differentiable Rendering. In Image and Vision Computing , Wei Qi Yan, Minh Nguyen, and Martin Stommel (Eds.). Springer Nature Switzerland, Cham, 388–401
K. C. Dharma, Clayton T. Morrison, and Bradley Walls. 2023 · 2023
Closest in time.
Generating Procedural Materials from Text or Image Prompts. In ACM SIGGRAPH 2023 Conference Proceedings . 1–11
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. 2023 · 2023
Closest in time.
X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . 2749–2760
Yiwei Ma, Xiaoqing Zhang, Xiaoshuai Sun, Jiayi Ji, Haowei Wang, Guannan Jiang, Weilin Zhuang, and Rongrong Ji. 2023 · 2023
Closest in time.
DyNCA: Real-time Dynamic Texture Synthesis Using Neural Cellular Automata. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 20742–20751
Ehsan Pajouheshgar, Yitao Xu, Tong Zhang, and Sabine Süsstrunk. 2023 · 2023
Closest in time.
Texture: Text-guided texturing of 3d shapes
Original
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…