2023

Text-To-4D Dynamic Scene Generation

Singer, Uriel, Sheynin, Shelly, Polyak, Adam et al.

Understand

We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions.

Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency by querying a Text-to-Video (T2V) diffusion-based model.
The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment.
MAV3D does not require any 3D or 4D data and the T2V model is trained only on Text-Image pairs and unlabeled videos.

Built on

Natural language input for scene generation
Adorni, G. and Di Manzo, M · 1983
Earlier work this paper cites.
Wordseye: An automatic text-to-scene conversion system
Coyne, B. and Sproat, R · 2001
Earlier work this paper cites.
Free-viewpoint video of human actors
Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P · 2003
Earlier work this paper cites.
3d video and free viewpoint video-technologies, applications and mpeg standards
Smolic, A., Mueller, K., Merkle, P., Fehn, C., Kauff, P., Eisert, P., and Wiegand, T · 2006
Earlier work this paper cites.
Learning spatial knowledge for text to 3d scene generation
Chang, A., Savva, M., and Manning, C. D · 2014
Earlier work this paper cites.
High-quality streamable free-viewpoint video
Collet, A., Chuang, M., Sweeney, P., Gillett, D., Evseev, D., Calabrese, D., Hoppe, H., Kirk, A., and Sullivan, S · 2015
Earlier work this paper cites.
Understanding deep image representations by inverting them
Mahendran, A. and Vedaldi, A · 2015
Earlier work this paper cites.
Structure-from-motion revisited
Schönberger, J. L. and Frahm, J.-M · 2016
Earlier work this paper cites.
Xatlas algorithm, 2016
Young, J · 2016
Earlier work this paper cites.
Text2shape: Generating shapes from natural language by learning joint embeddings
Chen, K., Choy, C. B., Savva, M., Chang, A. X., Funkhouser, T., and Savarese, S · 2018
Earlier work this paper cites.
Diffusion models beat gans on image synthesis
Dhariwal, P. and Nichol, A · 2021
Earlier work this paper cites.
Neural radiance flow for 4d view synthesis and video processing
Du, Y., Zhang, Y., Yu, H.-X., Tenenbaum, J. B., and Wu, J · 2021
Earlier work this paper cites.
Taming transformers for high-resolution image synthesis
Esser, P., Rombach, R., and Ommer, B · 2021
Earlier work this paper cites.
Dynamic view synthesis from dynamic monocular video
Gao, C., Saraf, A., Kopf, J., and Huang, J.-B · 2021
Earlier work this paper cites.
Clipmatrix: Text-controlled creation of 3d textured meshes
Original
Jetchev, N · 2021
Earlier work this paper cites.
Neural scene flow fields for space-time view synthesis of dynamic scenes
Li, Z., Niklaus, S., Snavely, N., and Wang, O · 2021
Earlier work this paper cites.
NeRF in the wild: Neural radiance fields for unconstrained photo collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S., Barron, J. T., Dosovitskiy, A., and Duckworth, D · 2021
Earlier work this paper cites.

Similar

NeRF: Representing scenes as neural radiance fields for view synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R · 2021
Cited alongside, same era.
D-NeRF: Neural radiance fields for dynamic scenes
Pumarola, A., Corona, E., Pons-Moll, G., and Moreno-Noguer, F · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I · 2021
Cited alongside, same era.
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I · 2021
Cited alongside, same era.
Neural geometric level of detail: Real-time rendering with implicit 3D shapes
Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahrai, D., Jacobson, A., McGuire, M., and Fidler, S · 2021
Cited alongside, same era.
Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video
Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Lassner, C., and Theobalt, C · 2021
Cited alongside, same era.
Space-time neural irradiance fields for free-viewpoint video
Xian, W., Huang, J.-B., Kopf, J., and Kim, C · 2021
Cited alongside, same era.
pixelnerf: Neural radiance fields from one or few images
Yu, A., Ye, V., Tancik, M., and Kanazawa, A · 2021
Cited alongside, same era.
Mip-NeRF 360: Unbounded anti-aliased neural radiance fields
Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., and Hedman, P · 2022
Cited alongside, same era.
Efficient geometry-aware 3d generative adversarial networks
Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., et al · 2022
Cited alongside, same era.
TensoRF: Tensorial radiance fields
Chen, A., Xu, Z., Geiger, A., Yu, J., and Su, H · 2022
Cited alongside, same era.
Fast dynamic radiance fields with time-aware neural voxels
Fang, J., Yi, T., Wang, X., Xie, L., Zhang, X., Liu, W., Nießner, M., and Tian, Q · 2022
Cited alongside, same era.

Then

Instant neural graphics primitives with a multiresolution hash encoding
Müller, T., Evans, A., Schied, C., and Keller, A · 2022
Later among the works it cites.
RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs
Niemeyer, M., Barron, J. T., Mildenhall, B., Sajjadi, M. S., Geiger, A., and Radwan, N · 2022
Later among the works it cites.
DreamFusion: Text-to-3d using 2d diffusion
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M · 2022
Later among the works it cites.
Clip-forge: Towards zero-shot text-to-shape generation
Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., and Malekshan, K. R · 2022
Later among the works it cites.
Knn-diffusion: Image generation via large-scale retrieval
Original
Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y · 2022
Later among the works it cites.
Make-a-video: Text-to-video generation without text-video data
Original
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al · 2022
Later among the works it cites.
Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction
Sun, C., Sun, M., and Chen, H.-T · 2022
Later among the works it cites.
Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022
Tang, J · 2022
Later among the works it cites.
Phenaki: Variable length video generation from open domain textual description, 2022
Original
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D · 2022
Later among the works it cites.
Clip-NeRF: Text-and-image driven manipulation of neural radiance fields
Wang, C., Chai, M., He, M., Chen, D., and Liao, J · 2022
Later among the works it cites.
Scaling autoregressive models for content-rich text-to-image generation
Original
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., and Wu, Y · 2022
Later among the works it cites.
HexPlane: a fast representation for dynamic scenes
Original
Cao, A. and Johnson, J · 2023
Closest in time.
K-planes: Explicit radiance fields in space, time, and appearance
Original
Fridovich-Keil, S., Meanti, G., Warburg, F., Recht, B., and Kanazawa, A · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…