Fetching the paper…

Reading the bibliography…

2024

LLMs Meet Multimodal Generation and Editing: A Survey

He, Yingqing, Liu, Zhaoyang, Chen, Jingye et al.

Original
PDF
Post on X

Understand

With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning.

Previous surveys of multimodal large language models (MLLMs) mainly focus on multimodal understanding.
This survey elaborates on multimodal generation and editing across various domains, comprising image, video, 3D, and audio.
Specifically, we summarize the notable advancements with milestone works in these fields and categorize these studies into LLM-based and CLIP/T5-based methods.

LLMs Meet Multimodal Generation and Editing: A Survey · Around

Built on

N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 1921–1930
1930
Earlier work this paper cites.
D. Kirk and J. Arvo, “Unbiased sampling techniques for image synthesis,” ACM SIGGRAPH Computer Graphics , vol. 25, no. 4, pp. 153–156, 1991
1991
Earlier work this paper cites.
M. F. Cohen and J. R. Wallace, Radiosity and realistic image synthesis . Morgan Kaufmann, 1993
1993
Earlier work this paper cites.
J. Rothstein, MIDI: A comprehensive introduction . AR Editions, Inc., 1995, vol. 7
1995
Earlier work this paper cites.
T. Vetter and T. Poggio, “Linear object classes and image synthesis from a single example image,” IEEE Transactions on pattern analysis and machine intelligence , vol. 19, no. 7, pp. 733–742, 1997
1997
Earlier work this paper cites.
E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in ISMIR . Citeseer, 2009, pp. 387–392
2009
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255
2009
Earlier work this paper cites.
V. Ordonez, G. Kulkarni, and T. Berg, “Im2text: Describing images using 1 million captioned photographs,” Advances in neural information processing systems , vol. 24, 2011
2011
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer, 2014, pp. 740–755
2014
Earlier work this paper cites.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International conference on machine learning . PMLR, 2015, pp. 2256–2265
2015
Earlier work this paper cites.
F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365 , 2015
Original
2015
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2015, pp. 5206–5210
2015
Earlier work this paper cites.
J. Xu, T. Mei, T. Yao, and Y. Rui, “Msr-vtt: A large video description dataset for bridging video and language,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 5288–5296, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:206594535
2016
Earlier work this paper cites.
A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. J. Pal, H. Larochelle, A. C. Courville, and B. Schiele, “Movie description,” International Journal of Computer Vision , vol. 123, pp. 94 – 120, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:18217052
2016
Earlier work this paper cites.
L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,” arXiv preprint arXiv:1605.08803 , 2016
Original
2016
Earlier work this paper cites.
R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense-captioning events in videos,” 2017 IEEE International Conference on Computer Vision (ICCV) , pp. 706–715, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:1026139
2017
Earlier work this paper cites.
J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2017, pp. 776–780
2017
Earlier work this paper cites.
R. Jia and P. Liang, “Adversarial examples for evaluating reading comprehension systems,” arXiv preprint arXiv:1707.07328 , 2017
Original
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Earlier work this paper cites.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh et al. , “Mixed precision training,” arXiv preprint arXiv:1710.03740 , 2017
Original
2017
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
Z. Liu, P. Luo, X. Wang, and X. Tang, “Large-scale celebfaces attributes (celeba) dataset,” Retrieved August , vol. 15, no. 2018, p. 11, 2018
2018
Earlier work this paper cites.
R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, and F. Metze, “How2: A large-scale dataset for multimodal language understanding,” ArXiv , vol. abs/1811.00347, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:53186236
Original
2018
Earlier work this paper cites.
C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the maestro dataset,” arXiv preprint arXiv:1810.12247 , 2018
Original
2018
Earlier work this paper cites.
P. Korshunov and S. Marcel, “Deepfakes: A new threat to face recognition? assessment and detection. arxiv 2018,” arXiv preprint arXiv:1812.08685
Original
2018
Earlier work this paper cites.
B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 2704–2713
2018
Earlier work this paper cites.
D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122 , 2018
Original
2018
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
2019
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
2019
Earlier work this paper cites.
J. Lin, C. Gan, and S. Han, “Tsm: Temporal shift module for efficient video understanding,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 7083–7093
2019
Earlier work this paper cites.
C.-Y. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick, “Long-term feature banks for detailed video understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 284–293
2019
Earlier work this paper cites.
X. E. Wang, J. Wu, J. Chen, L. Li, Y. fang Wang, and W. Y. Wang, “Vatex: A large-scale, high-quality multilingual dataset for video-and-language research,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 4580–4590, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:102352148
2019
Earlier work this paper cites.
A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, “Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 2630–2640, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:182952863
2019
Earlier work this paper cites.
H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “Libritts: A corpus derived from librispeech for text-to-speech,” arXiv preprint arXiv:1904.02882 , 2019
Original
2019
Earlier work this paper cites.
D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic music tagging.” ICML, 2019
2019
Earlier work this paper cites.
E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh, “Universal adversarial triggers for attacking and analyzing nlp,” arXiv preprint arXiv:1908.07125 , 2019
Original
2019
Earlier work this paper cites.
Y. Choukroun, E. Kravchik, F. Yang, and P. Kisilev, “Low-bit quantization of neural networks for efficient inference,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . IEEE, 2019, pp. 3009–3018
2019
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020
2020
Earlier work this paper cites.
Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558 , 2020
Original
2020
Earlier work this paper cites.
P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” arXiv preprint arXiv:2005.00341 , 2020
Original
2020
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020
2020
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM , vol. 63, no. 11, pp. 139–144, 2020
2020
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840–6851, 2020
2020
Earlier work this paper cites.
Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16 . Springer, 2020, pp. 776–794
2020
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020
2020
Earlier work this paper cites.
J. C. Stroud, D. A. Ross, C. Sun, J. Deng, R. Sukthankar, and C. Schmid, “Learning video representations from textual web supervision,” ArXiv , vol. abs/2007.14937, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:220845567
Original
2020
Earlier work this paper cites.
J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen et al. , “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7669–7673
2020
Earlier work this paper cites.
H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 721–725
2020
Earlier work this paper cites.
L. Verdoliva, “Media forensics and deepfakes: an overview,” IEEE Journal of Selected Topics in Signal Processing , vol. 14, no. 5, pp. 910–932, 2020
2020
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020
Original
2020
Earlier work this paper cites.
M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 1708–1718, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:232478955
2021
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763
2021
Earlier work this paper cites.
P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 12 873–12 883
2021
Earlier work this paper cites.
C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in International conference on machine learning . PMLR, 2021, pp. 4904–4916
2021
Earlier work this paper cites.
S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 3558–3568
2021
Earlier work this paper cites.
K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork, “Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2021, pp. 2443–2449
2021
Earlier work this paper cites.
C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” arXiv preprint arXiv:2111.02114 , 2021
Original
2021
Earlier work this paper cites.
C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “Sdedit: Guided image synthesis and editing with stochastic differential equations,” arXiv preprint arXiv:2108.01073 , 2021
Original
2021
Earlier work this paper cites.
B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan, “Star: A benchmark for situated reasoning in real-world videos,” in Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2) , 2021
2021
Earlier work this paper cites.
Z. Liu, L. Wang, W. Wu, C. Qian, and T. Lu, “Tam: Temporal adaptive module for video recognition,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 13 708–13 718
2021
Earlier work this paper cites.
M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1728–1738
2021
Earlier work this paper cites.
G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” in ICML , vol. 2, no. 3, 2021, p. 4
2021
Earlier work this paper cites.
R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi, “Merlot: Multimodal neural script knowledge models,” in Neural Information Processing Systems , 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:235352775
2021
Earlier work this paper cites.
H. Xue, T. Hang, Y. Zeng, Y. Sun, B. Liu, H. Yang, J. Fu, and B. Guo, “Advancing high-resolution video-language representation with large-scale video transcriptions,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 5026–5035, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:244462849
2021
Earlier work this paper cites.
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International Conference on Machine Learning . PMLR, 2021, pp. 8821–8831
2021
Earlier work this paper cites.
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” arXiv preprint arXiv:2112.10741 , 2021
Original
2021
Earlier work this paper cites.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM , vol. 65, no. 1, pp. 99–106, 2021
2021
Earlier work this paper cites.
P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” arXiv preprint arXiv:2106.10689 , 2021
Original
2021
Earlier work this paper cites.
T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler, “Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis,” Advances in Neural Information Processing Systems , vol. 34, pp. 6087–6101, 2021
2021
Earlier work this paper cites.
J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny, “Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 10 901–10 911
2021
Earlier work this paper cites.
Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” arXiv preprint arXiv:2104.01778 , 2021
Original
2021
Earlier work this paper cites.
P. Żelasko, S. Joshi, Y. Shao, J. Villalba, J. Trmal, N. Dehak, and S. Khudanpur, “Adversarial attacks and defenses for speech recognition systems,” arXiv preprint arXiv:2103.17122 , 2021
Original
2021
Earlier work this paper cites.
Z. Chen, L. Xie, S. Pang, Y. He, and Q. Tian, “Appending adversarial frames for universal video attack,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2021, pp. 3199–3208
2021
Earlier work this paper cites.
S.-Y. Lo and V. M. Patel, “Defending against multiple and unforeseen adversarial videos,” IEEE Transactions on Image Processing , vol. 31, pp. 962–973, 2021
2021
Earlier work this paper cites.
Y. Mirsky and W. Lee, “The creation and detection of deepfakes: A survey,” ACM computing surveys (CSUR) , vol. 54, no. 1, pp. 1–41, 2021
2021
Earlier work this paper cites.
D. Wodajo and S. Atnafu, “Deepfake video detection using convolutional vision transformer,” arXiv preprint arXiv:2102.11126 , 2021
Original
2021
Earlier work this paper cites.
S. Hussain, P. Neekhara, M. Jere, F. Koushanfar, and J. McAuley, “Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2021, pp. 3348–3357
2021
Earlier work this paper cites.
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” Advances in Neural Information Processing Systems , vol. 35, pp. 8633–8646, 2022
2022
Earlier work this paper cites.
Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen, “Latent video diffusion models for high-fidelity long video generation,” arXiv preprint arXiv:2211.13221 , 2022
Original
2022
Earlier work this paper cites.
D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “Magicvideo: Efficient video generation with latent diffusion models,” arXiv preprint arXiv:2211.11018 , 2022
Original
2022
Earlier work this paper cites.
U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni et al. , “Make-a-video: Text-to-video generation without text-video data,” arXiv preprint arXiv:2209.14792 , 2022
Original
2022
Earlier work this paper cites.
J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet et al. , “Imagen video: High definition video generation with diffusion models,” arXiv preprint arXiv:2210.02303 , 2022
Original
2022
Earlier work this paper cites.
R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan, “Phenaki: Variable length video generation from open domain textual descriptions,” in International Conference on Learning Representations , 2022
2022
Earlier work this paper cites.
A. Sanghi, H. Chu, J. G. Lambourne, Y. Wang, C.-Y. Cheng, M. Fumero, and K. R. Malekshan, “Clip-forge: Towards zero-shot text-to-shape generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 603–18 613
2022
Earlier work this paper cites.
N. Mohammad Khalid, T. Xie, E. Belilovsky, and T. Popa, “Clip-mesh: Generating textured meshes from text using pretrained image-text models,” in SIGGRAPH Asia 2022 conference papers , 2022, pp. 1–8
2022
Earlier work this paper cites.
O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka, “Text2mesh: Text-driven neural stylization for meshes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 13 492–13 502
2022
Earlier work this paper cites.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 684–10 695
2022
Earlier work this paper cites.
B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text-to-3d using 2d diffusion,” arXiv preprint arXiv:2209.14988 , 2022
Original
2022
Earlier work this paper cites.
F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, “Audiogen: Textually guided audio generation,” arXiv preprint arXiv:2209.15352 , 2022
Original
2022
Earlier work this paper cites.
S. Forsgren and H. Martiros, “Riffusion-stable diffusion for real-time music generation, 2022,” URL https://riffusion. com/about , vol. 6, 2022
2022
Earlier work this paper cites.
Z. Shi, S. Peng, Y. Xu, A. Geiger, Y. Liao, and Y. Shen, “Deep generative models on 3d representations: A survey,” arXiv preprint arXiv:2210.15663 , 2022
Original
2022
Earlier work this paper cites.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125 , vol. 1, no. 2, p. 3, 2022
Original
2022
Earlier work this paper cites.
C. Wang, M. Chai, M. He, D. Chen, and J. Liao, “Clip-nerf: Text-and-image driven manipulation of neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3835–3844
2022
Earlier work this paper cites.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al. , “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862 , 2022
Original
2022
Earlier work this paper cites.
Y. Zheng, H. Yang, T. Zhang, J. Bao, D. Chen, Y. Huang, L. Yuan, D. Chen, M. Zeng, and F. Wen, “General facial representation learning in a visual-linguistic manner,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 697–18 709
2022
Earlier work this paper cites.
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022
2022
Earlier work this paper cites.
LAION-COCO, “https://laion.ai/blog/laion-coco/,” 2022. [Online]. Available: https://laion.ai/blog/laion-coco/
2022
Earlier work this paper cites.
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman et al. , “Laion-5b: An open large-scale dataset for training next generation image-text models,” Advances in Neural Information Processing Systems , vol. 35, pp. 25 278–25 294, 2022
2022
Earlier work this paper cites.
Coyo-700M, “https://huggingface.co/datasets/kakaobrain/coyo-700m,” 2022. [Online]. Available: https://huggingface.co/datasets/kakaobrain/coyo-700m
2022
Earlier work this paper cites.
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al. , “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in Neural Information Processing Systems , vol. 35, pp. 36 479–36 494, 2022
2022
Earlier work this paper cites.
K. Frans, L. Soros, and O. Witkowski, “Clipdraw: Exploring text-to-drawing synthesis through language-image encoders,” Advances in Neural Information Processing Systems , vol. 35, pp. 5207–5218, 2022
2022
Earlier work this paper cites.
S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 696–10 706
2022
Earlier work this paper cites.
Z. Wang, W. Liu, Q. He, X. Wu, and Z. Yi, “Clip-gen: Language-free training of a text-to-image generator with clip,” arXiv preprint arXiv:2203.00386 , 2022
Original
2022
Earlier work this paper cites.
Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang, “An empirical study of gpt-3 for few-shot knowledge-based vqa,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 3, 2022, pp. 3081–3089
2022
Earlier work this paper cites.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9
2022
Earlier work this paper cites.
G. Kim, T. Kwon, and J. C. Ye, “Diffusionclip: Text-guided diffusion models for robust image manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 2426–2435
2022
Earlier work this paper cites.
G. Couairon, J. Verbeek, H. Schwenk, and M. Cord, “Diffedit: Diffusion-based semantic image editing with mask guidance,” arXiv preprint arXiv:2210.11427 , 2022
Original
2022
Earlier work this paper cites.
A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” arXiv preprint arXiv:2208.01626 , 2022
Original
2022
Earlier work this paper cites.
T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” arXiv preprint arXiv:2211.09800 , 2022
Original
2022
Earlier work this paper cites.
M. Zhao, B. Li, J. Wang, W. Li, W. Zhou, L. Zhang, S. Xuyang, Z. Yu, X. Yu, G. Li et al. , “Towards video text visual question answering: benchmark and baseline,” Advances in Neural Information Processing Systems , vol. 35, pp. 35 549–35 562, 2022
2022
Earlier work this paper cites.
A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 124–141, 2022
2022
Earlier work this paper cites.
——, “Learning to answer visual questions from web videos,” arXiv preprint arXiv:2205.05019 , 2022
Original
2022
Earlier work this paper cites.
K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y. Lu, and L. Wang, “Swinbert: End-to-end transformers with sparse attention for video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 949–17 958
2022
Earlier work this paper cites.
A. Nagrani, P. H. Seo, B. Seybold, A. Hauth, S. Manén, C. Sun, and C. Schmid, “Learning audio-video modalities from image captions,” in European Conference on Computer Vision , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:247939759
2022
Earlier work this paper cites.
A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen, “Point-e: A system for generating 3d point clouds from complex prompts,” arXiv preprint arXiv:2212.08751 , 2022
Original
2022
Earlier work this paper cites.
J. Lei, Y. Zhang, K. Jia et al. , “Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition,” Advances in Neural Information Processing Systems , vol. 35, pp. 30 923–30 936, 2022
2022
Earlier work this paper cites.
A. Jain, B. Mildenhall, J. T. Barron, P. Abbeel, and B. Poole, “Zero-shot text-guided object generation with dream fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 867–876
2022
Earlier work this paper cites.
G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or, “Motionclip: Exposing human motion generation to clip space,” in European Conference on Computer Vision . Springer, 2022, pp. 358–374
2022
Earlier work this paper cites.
F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, and Z. Liu, “Avatarclip: Zero-shot text-driven generation and animation of 3d avatars,” arXiv preprint arXiv:2205.08535 , 2022
Original
2022
Earlier work this paper cites.
B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng et al. , “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 6182–6186
2022
Earlier work this paper cites.
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma et al. , “Scaling instruction-finetuned language models,” arXiv preprint arXiv:2210.11416 , 2022
Original
2022
Earlier work this paper cites.
Y. Gong, A. Rouditchenko, A. H. Liu, D. Harwath, L. Karlinsky, H. Kuehne, and J. Glass, “Contrastive audio-visual masked autoencoder,” arXiv preprint arXiv:2210.07839 , 2022
Original
2022
Earlier work this paper cites.
A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022
Original
2022
Earlier work this paper cites.
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self-generated instructions,” arXiv preprint arXiv:2212.10560 , 2022
Original
2022
Earlier work this paper cites.
A. Parisi, Y. Zhao, and N. Fiedel, “Talm: Tool augmented language models,” arXiv preprint arXiv:2205.12255 , 2022
Original
2022
Earlier work this paper cites.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 24 824–24 837, 2022
2022
Earlier work this paper cites.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022
2022
Earlier work this paper cites.
S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, “Peft: State-of-the-art parameter-efficient fine-tuning methods,” https://github.com/huggingface/peft , 2022
2022
Earlier work this paper cites.
F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” arXiv preprint arXiv:2211.09527 , 2022
Original
2022
Earlier work this paper cites.
M. Ding, W. Zheng, W. Hong, and J. Tang, “Cogview2: Faster and better text-to-image generation via hierarchical transformers,” Advances in Neural Information Processing Systems , vol. 35, pp. 16 890–16 902, 2022
2022
Earlier work this paper cites.
R. Or-El, X. Luo, M. Shan, E. Shechtman, J. J. Park, and I. Kemelmacher-Shlizerman, “Stylesdf: High-resolution 3d-consistent image and geometry generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 13 503–13 513
2022
Earlier work this paper cites.
R. Liu, D. Garrette, C. Saharia, W. Chan, A. Roberts, S. Narang, I. Blok, R. Mical, M. Norouzi, and N. Constant, “Character-aware models improve visual text rendering,” arXiv preprint arXiv:2212.10562 , 2022
Original
2022
Earlier work this paper cites.
T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,” Advances in Neural Information Processing Systems , vol. 35, pp. 30 318–30 332, 2022
2022
Earlier work this paper cites.
Y. LeCun, “A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27,” Open Review , vol. 62, no. 1, 2022
2022
Earlier work this paper cites.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
Original
2023
Earlier work this paper cites.
OpenAI, “Chatgpt: A language model for conversational ai,” OpenAI, Tech. Rep., 2023. [Online]. Available: https://www.openai.com/research/chatgpt
2023
Earlier work this paper cites.
Y. Li, C. Wang, and J. Jia, “Llama-vid: An image is worth 2 tokens in large language models,” arXiv preprint arXiv:2311.17043 , 2023
Original
2023
Earlier work this paper cites.
P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al. , “Llama-adapter v2: Parameter-efficient visual instruction model,” arXiv preprint arXiv:2304.15010 , 2023
Original
2023
Earlier work this paper cites.
D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” arXiv preprint arXiv:2307.01952 , 2023
Original
2023
Earlier work this paper cites.
J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu et al. , “Pixart- α \alpha : Fast training of diffusion transformer for photorealistic text-to-image synthesis,” arXiv preprint arXiv:2310.00426 , 2023
Original
2023
Earlier work this paper cites.
H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang et al. , “Videocrafter1: Open diffusion models for high-quality video generation,” arXiv preprint arXiv:2310.19512 , 2023
Original
2023
Earlier work this paper cites.
Y. Guo, C. Yang, A. Rao, Y. Wang, Y. Qiao, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to-image diffusion models without specific tuning,” arXiv preprint arXiv:2307.04725 , 2023
Original
2023
Earlier work this paper cites.
R. Girdhar, M. Singh, A. Brown, Q. Duval, S. Azadi, S. S. Rambhatla, A. Shah, X. Yin, D. Parikh, and I. Misra, “Emu video: Factorizing text-to-video generation by explicit image conditioning,” arXiv preprint arXiv:2311.10709 , 2023
Original
2023
Earlier work this paper cites.
C. Wang, R. Jiang, M. Chai, M. He, D. Chen, and J. Liao, “Nerf-art: Text-driven neural radiance fields stylization,” IEEE Transactions on Visualization and Computer Graphics , 2023
2023
Earlier work this paper cites.
B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics , vol. 42, no. 4, July 2023. [Online]. Available: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
2023
Earlier work this paper cites.
T. Yi, J. Fang, J. Wang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang, “Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models,” arXiv preprint arXiv:2310.08529 , 2023
Original
2023
Earlier work this paper cites.
J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” arXiv preprint arXiv:2309.16653 , 2023
Original
2023
Earlier work this paper cites.
L. Höllein, A. Cao, A. Owens, J. Johnson, and M. Nießner, “Text2room: Extracting textured 3d meshes from 2d text-to-image models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 7909–7920
2023
Earlier work this paper cites.
Y. Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y. Chen, “LucidDreamer: Towards high-fidelity text-to-3d generation via interval score matching,” https://arxiv.org/abs/2311.11284 , 2023
Original
2023
Earlier work this paper cites.
X. Yu, Y.-C. Guo, Y. Li, D. Liang, S.-H. Zhang, and X. Qi, “Text-to-3d with classifier score distillation,” https://arxiv.org/abs/2310.19415 , 2023
Original
2023
Earlier work this paper cites.
W. Li, R. Chen, X. Chen, and P. Tan, “SweetDreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d,” https://arxiv.org/abs/2310.02596 , 2023
Original
2023
Earlier work this paper cites.
Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” arXiv preprint arXiv:2305.16213 , 2023
Original
2023
Earlier work this paper cites.
J. Lorraine, K. Xie, X. Zeng, C.-H. Lin, T. Takikawa, N. Sharp, T.-Y. Lin, M.-Y. Liu, S. Fidler, and J. Lucas, “Att3d: Amortized text-to-3d object synthesis,” in International Conference on Computer Vision ICCV , 2023
2023
Earlier work this paper cites.
J. Xu, X. Wang, W. Cheng, Y.-P. Cao, Y. Shan, X. Qie, and S. Gao, “Dream3D: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models,” https://arxiv.org/abs/2212.14704 , 2023
Original
2023
Earlier work this paper cites.
J. Zhu and P. Zhuang, “HiFA: High-fidelity text-to-3d with advanced diffusion guidance,” https://arxiv.org/abs/2305.18766 , 2023
Original
2023
Earlier work this paper cites.
R. Chen, Y. Chen, N. Jiao, and K. Jia, “Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023
2023
Earlier work this paper cites.
C. Tsalicoglou, F. Manhardt, A. Tonioni, M. Niemeyer, and F. Tombari, “Textmesh: Generation of realistic 3d meshes from text prompts,” arXiv preprint arXiv:2304.12439 , 2023
Original
2023
Earlier work this paper cites.
C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3d: High-resolution text-to-3d content creation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 300–309
2023
Earlier work this paper cites.
J. Seo, W. Jang, M.-S. Kwak, J. Ko, H. Kim, J. Kim, J.-H. Kim, J. Lee, and S. Kim, “Let 2d diffusion model know 3d-consistency for robust text-to-3d generation,” arXiv preprint arXiv:2303.07937 , 2023
Original
2023
Earlier work this paper cites.
H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” arXiv preprint arXiv:2301.12503 , 2023
Original
2023
Earlier work this paper cites.
H. Liu, Q. Tian, Y. Yuan, X. Liu, X. Mei, Q. Kong, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” arXiv preprint arXiv:2308.05734 , 2023
Original
2023
Earlier work this paper cites.
A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al. , “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325 , 2023
Original
2023
Earlier work this paper cites.
K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” arXiv preprint arXiv:2304.09116 , 2023
Original
2023
Earlier work this paper cites.
C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al. , “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111 , 2023
Original
2023
Earlier work this paper cites.
Z. Jiang, J. Liu, Y. Ren, J. He, C. Zhang, Z. Ye, P. Wei, C. Wang, X. Yin, Z. Ma et al. , “Mega-tts 2: Zero-shot text-to-speech with arbitrary length speech prompts,” arXiv preprint arXiv:2307.07218 , 2023
Original
2023
Earlier work this paper cites.
Y. Ge, Y. Ge, Z. Zeng, X. Wang, and Y. Shan, “Planting a seed of vision in large language model,” arXiv preprint arXiv:2307.08041 , 2023
Original
2023
Earlier work this paper cites.
L. Zeqiang, Z. Xizhou, D. Jifeng, Q. Yu, and W. Wenhai, “Mini-dalle3: Interactive text to image by prompting large language models,” arXiv preprint arXiv:2310.07653 , 2023
Original
2023
Earlier work this paper cites.
Z. Tang, Z. Yang, M. Khademi, Y. Liu, C. Zhu, and M. Bansal, “Codi-2: In-context, interleaved, and interactive any-to-any generation,” arXiv preprint arXiv:2311.18775 , 2023
Original
2023
Earlier work this paper cites.
Y. Ge, S. Zhao, Z. Zeng, Y. Ge, C. Li, X. Wang, and Y. Shan, “Making llama see and draw with seed tokenizer,” arXiv preprint arXiv:2310.01218 , 2023
Original
2023
Earlier work this paper cites.
Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang et al. , “Generative multimodal models are in-context learners,” arXiv preprint arXiv:2312.13286 , 2023
Original
2023
Earlier work this paper cites.

Similar

X. Zhao, B. Liu, Q. Liu, G. Shi, and X.-M. Wu, “Making multimodal generation easier: When diffusion models meet llms,” arXiv preprint arXiv:2310.08949 , 2023
Original
2023
Cited alongside, same era.
J. Chen, Y. Huang, T. Lv, L. Cui, Q. Chen, and F. Wei, “Textdiffuser-2: Unleashing the power of language models for text rendering,” arXiv preprint arXiv:2311.16465 , 2023
Original
2023
Cited alongside, same era.
L. Lian, B. Li, A. Yala, and T. Darrell, “Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models,” arXiv preprint arXiv:2305.13655 , 2023
Original
2023
Cited alongside, same era.
W. Feng, W. Zhu, T.-j. Fu, V. Jampani, A. Akula, X. He, S. Basu, X. E. Wang, and W. Y. Wang, “Layoutgpt: Compositional visual planning and generation with large language models,” arXiv preprint arXiv:2305.15393 , 2023
Original
2023
Cited alongside, same era.

Then

Z. Liu, Y. Li, Y. Lin, X. Yu, S. Peng, Y.-P. Cao, X. Qi, X. Huang, D. Liang, and W. Ouyang, “Unidream: Unifying diffusion priors for relightable text-to-3d generation,” 2023
2023
Later among the works it cites.
L. Qiu, G. Chen, X. Gu, Q. zuo, M. Xu, Y. Wu, W. Yuan, Z. Dong, L. Bo, and X. Han, “Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d,” arXiv preprint arXiv:2311.16918 , 2023
Original
2023
Later among the works it cites.
J. Li, H. Tan, K. Zhang, Z. Xu, F. Luan, Y. Xu, Y. Hong, K. Sunkavalli, G. Shakhnarovich, and S. Bi, “Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model,” arXiv preprint arXiv:2311.06214 , 2023
Original
2023
Later among the works it cites.
H. Jun and A. Nichol, “Shap-e: Generating conditional 3d implicit functions,” arXiv preprint arXiv:2305.02463 , 2023
Original
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

T. Zhang, Y. Zhang, V. Vineet, N. Joshi, and X. Wang, “Controllable text-to-image generation with gpt-4,” arXiv preprint arXiv:2305.18583 , 2023

2023

Cited alongside, same era.

L. Qu, S. Wu, H. Fei, L. Nie, and T.-S. Chua, “Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 643–654

2023

Cited alongside, same era.

Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee, “Gligen: Open-set grounded text-to-image generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 511–22 521

2023

Cited alongside, same era.

J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo et al. , “Improving image generation with better captions,” Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2023

2023

Cited alongside, same era.

D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, R. Hornung, H. Adam, H. Akbari, Y. Alon, V. Birodkar et al. , “Videopoet: A large language model for zero-shot video generation,” arXiv preprint arXiv:2312.14125 , 2023

2023

Cited alongside, same era.

L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann et al. , “Language model beats diffusion–tokenizer is key to visual generation,” arXiv preprint arXiv:2310.05737 , 2023

2023

Cited alongside, same era.

H. Fei, S. Wu, W. Ji, H. Zhang, and T.-S. Chua, “Empowering dynamics-aware text-to-video diffusion with large language models,” arXiv preprint arXiv:2308.13812 , 2023

2023

Cited alongside, same era.

H. Lin, A. Zala, J. Cho, and M. Bansal, “Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning,” arXiv preprint arXiv:2309.15091 , 2023

2023

Cited alongside, same era.

R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang, and D. Lin, “Pointllm: Empowering large language models to understand point clouds,” arXiv preprint arXiv:2308.16911 , 2023

2023

Later among the works it cites.

Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan, “3d-llm: Injecting the 3d world into large language models,” arXiv preprint arXiv:2307.12981 , 2023

2023

Later among the works it cites.

O. Gordon, O. Avrahami, and D. Lischinski, “Blended-nerf: Zero-shot object generation and blending in existing neural radiance fields,” arXiv preprint arXiv:2306.12760 , 2023

2023

Later among the works it cites.

W. Gao, N. Aigerman, T. Groueix, V. Kim, and R. Hanocka, “Textdeformer: Geometry manipulation using text guidance,” in ACM SIGGRAPH 2023 Conference Proceedings , 2023, pp. 1–11

2023

Later among the works it cites.

C. Bao, Y. Zhang, B. Yang, T. Fan, Z. Yang, H. Bao, G. Zhang, and Z. Cui, “Sine: Semantic-driven image-based nerf editing with prior-guided editing field,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 20 919–20 929

2023

Later among the works it cites.

A. Mikaeili, O. Perel, M. Safaee, D. Cohen-Or, and A. Mahdavi-Amiri, “Sked: Sketch-guided text-based 3d editing,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 14 607–14 619

2023

Later among the works it cites.

J. Zhuang, C. Wang, L. Lin, L. Liu, and G. Li, “Dreameditor: Text-driven 3d scene editing with neural fields,” in SIGGRAPH Asia 2023 Conference Papers , 2023, pp. 1–10

2023

Later among the works it cites.

A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa, “Instruct-nerf2nerf: Editing 3d scenes with instructions,” arXiv preprint arXiv:2303.12789 , 2023

2023

Later among the works it cites.

D. Decatur, I. Lang, K. Aberman, and R. Hanocka, “3d paintbrush: Local stylization of 3d shapes with cascaded score distillation,” arXiv preprint arXiv:2311.09571 , 2023

2023

Later among the works it cites.

G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-or, and A. H. Bermano, “Human motion diffusion model,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=SJ1kSyO2jwu

2023

Later among the works it cites.

R. Chen, Y. Chen, N. Jiao, and K. Jia, “Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation,” arXiv preprint arXiv:2303.13873 , 2023

Original

2023

Later among the works it cites.

T. Huang, Y. Zeng, Z. Zhang, W. Xu, H. Xu, S. Xu, R. W. Lau, and W. Zuo, “Dreamcontrol: Control-based text-to-3d generation with 3d self-prior,” arXiv preprint arXiv:2312.06439 , 2023

2023

Later among the works it cites.

Y. Chen, C. Zhang, X. Yang, Z. Cai, G. Yu, L. Yang, and G. Lin, “It3d: Improved text-to-3d generation with explicit view synthesis,” 2023

2023

Later among the works it cites.

M. Zhao, C. Zhao, X. Liang, L. Li, Z. Zhao, Z. Hu, C. Fan, and X. Yu, “Efficientdreamer: High-fidelity and robust 3d creation via orthogonal-view diffusion prior,” arXiv preprint arXiv:2308.13223 , 2023

2023

Later among the works it cites.

Z. Chen, F. Wang, and H. Liu, “Text-to-3d using gaussian splatting,” arXiv preprint arXiv:2309.16585 , 2023

2023

Later among the works it cites.

Y. Ma, Y. Fan, J. Ji, H. Wang, X. Sun, G. Jiang, A. Shu, and R. Ji, “X-dreamer: Creating high-quality 3d content by bridging the domain gap between text-to-2d and text-to-3d generation,” arXiv preprint arXiv:2312.00085 , 2023

2023

Later among the works it cites.

X. Yang, Y. Chen, C. Chen, C. Zhang, Y. Xu, X. Yang, F. Liu, and G. Lin, “Learn to optimize denoising scores for 3d generation: A unified and improved diffusion prior on nerf and 3d gaussian splatting,” arXiv preprint arXiv:2312.04820 , 2023

2023

Later among the works it cites.

F. Liu, D. Wu, Y. Wei, Y. Rao, and Y. Duan, “Sherpa3d: Boosting high-fidelity text-to-3d generation via coarse 3d prior,” 2023

2023

Later among the works it cites.

Y. Yang, F.-Y. Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, J. Wu, N. Haber, R. Krishna, L. Liu, C. Callison-Burch, M. Yatskar, A. Kembhavi, and C. Clark, “Holodeck: Language guided generation of 3d embodied ai environments,” arXiv preprint arXiv:2312.09067 , 2023

2023

Later among the works it cites.

H. Song, S. Choi, H. Do, C. Lee, and T. Kim, “Blending-nerf: Text-driven localized editing in neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 14 383–14 393

2023

Later among the works it cites.

R. He, S. Huang, X. Nie, T. Hui, L. Liu, J. Dai, J. Han, G. Li, and S. Liu, “Customize your nerf: Adaptive source driven 3d scene editing via local-global iterative training,” arXiv preprint arXiv:2312.01663 , 2023

2023

Later among the works it cites.

X. Zeng, X. Chen, Z. Qi, W. Liu, Z. Zhao, Z. Wang, B. FU, Y. Liu, and G. Yu, “Paint3d: Paint anything 3d with lighting-less texture diffusion models,” 2023

2023

Later among the works it cites.

H. Hao, L. Zhou, S. Liu, J. Li, S. Hu, R. Wang, and F. Wei, “Boosting large language model for speech synthesis: An empirical study,” arXiv preprint arXiv:2401.00246 , 2023

2023

Later among the works it cites.

J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi, “Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action,” arXiv preprint arXiv:2312.17172 , 2023

2023

Later among the works it cites.

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023

2023

Later among the works it cites.

J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu et al. , “On decoder-only architecture for speech-to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2023, pp. 1–8

2023

Later among the works it cites.

W. Yu, C. Tang, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Connecting speech encoder and large language model for asr,” arXiv preprint arXiv:2309.13963 , 2023

2023

Later among the works it cites.

S. Wang, C.-H. H. Yang, J. Wu, and C. Zhang, “Can whisper perform speech-based in-context learning,” arXiv preprint arXiv:2309.07081 , 2023

2023

Later among the works it cites.

R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al. , “Palm 2 technical report,” arXiv preprint arXiv:2305.10403 , 2023

2023

Later among the works it cites.

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning . PMLR, 2023, pp. 28 492–28 518

2023

Later among the works it cites.

S. Kakouros, J. Šimko, M. Vainio, and A. Suni, “Investigating the utility of surprisal from large language models for speech synthesis prosody,” arXiv preprint arXiv:2306.09814 , 2023

2023

Later among the works it cites.

Z. Deng, Y. Ma, Y. Liu, R. Guo, G. Zhang, W. Chen, W. Huang, and E. Benetos, “Musilingo: Bridging music and text with pre-trained language models for music captioning and query response,” arXiv preprint arXiv:2309.08730 , 2023

2023

Later among the works it cites.

Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Lin, A. Ragni, E. Benetos, N. Gyenge et al. , “Mert: Acoustic music understanding model with large-scale self-supervised training,” arXiv preprint arXiv:2306.00107 , 2023

2023

Later among the works it cites.

Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al. , “Toolllm: Facilitating large language models to master 16000+ real-world apis,” arXiv preprint arXiv:2307.16789 , 2023

2023

Later among the works it cites.

Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, and L. Sun, “Toolalpaca: Generalized tool learning for language models with 3000 simulated cases,” arXiv preprint arXiv:2306.05301 , 2023

2023

Later among the works it cites.

T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” arXiv preprint arXiv:2302.04761 , 2023

2023

Later among the works it cites.

N. Farn and R. Shin, “Tooltalk: Evaluating tool-usage in a conversation setting,” arXiv preprint arXiv:2311.10775 , 2023

2023

Later among the works it cites.

S. Hao, T. Liu, Z. Wang, and Z. Hu, “Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings,” arXiv preprint arXiv:2305.11554 , 2023

2023

Later among the works it cites.

C.-Y. Hsieh, S.-A. Chen, C.-L. Li, Y. Fujii, A. Ratner, C.-Y. Lee, R. Krishna, and T. Pfister, “Tool documentation enables zero-shot tool-usage with large language models,” arXiv preprint arXiv:2308.00675 , 2023

2023

Later among the works it cites.

J. Ruan, Y. Chen, B. Zhang, Z. Xu, T. Bao, G. Du, S. Shi, H. Mao, X. Zeng, and R. Zhao, “Tptu: Task planning and tool usage of large language model-based ai agents,” arXiv preprint arXiv:2308.03427 , 2023

2023

Later among the works it cites.

J. Zhang, “Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt,” arXiv preprint arXiv:2304.11116 , 2023

2023

Later among the works it cites.

Y. Zhuang, X. Chen, T. Yu, S. Mitra, V. Bursztyn, R. A. Rossi, S. Sarkhel, and C. Zhang, “Toolchain*: Efficient action space navigation in large language models with a* search,” arXiv preprint arXiv:2310.13227 , 2023

2023

Later among the works it cites.

Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen, “Critic: Large language models can self-correct with tool-interactive critiquing,” arXiv preprint arXiv:2305.11738 , 2023

2023

Later among the works it cites.

Q. Jin, Y. Yang, Q. Chen, and Z. Lu, “Genegpt: Augmenting large language models with domain tools for improved access to biomedical information,” ArXiv , 2023

2023

Later among the works it cites.

B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro, “Art: Automatic multi-step reasoning and tool-use for large language models,” arXiv preprint arXiv:2303.09014 , 2023

2023

Later among the works it cites.

Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen et al. , “Tora: A tool-integrated reasoning agent for mathematical problem solving,” arXiv preprint arXiv:2309.17452 , 2023

2023

Later among the works it cites.

Y. Song, W. Xiong, D. Zhu, C. Li, K. Wang, Y. Tian, and S. Li, “Restgpt: Connecting large language models with real-world applications via restful apis,” arXiv preprint arXiv:2306.06624 , 2023

2023

Later among the works it cites.

S. Qiao, H. Gui, H. Chen, and N. Zhang, “Making language models better tool learners with execution feedback,” arXiv preprint arXiv:2305.13068 , 2023

2023

Later among the works it cites.

K. Zhang, H. Chen, L. Li, and W. Wang, “Syntax error-free and generalizable tool use for llms via finite-state decoding,” arXiv preprint arXiv:2310.07075 , 2023

2023

Later among the works it cites.

T. Gupta and A. Kembhavi, “Visual programming: Compositional visual reasoning without training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 953–14 962

2023

Later among the works it cites.

D. Surís, S. Menon, and C. Vondrick, “Vipergpt: Visual inference via python execution for reasoning,” Proceedings of IEEE International Conference on Computer Vision (ICCV) , 2023

2023

Later among the works it cites.

Z. Gao, Y. Du, X. Zhang, X. Ma, W. Han, S.-C. Zhu, and Q. Li, “Clova: A closed-loop visual assistant with tool usage and update,” arXiv preprint arXiv:2312.10908 , 2023

2023

Later among the works it cites.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR) , 2023

2023

Later among the works it cites.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al. , “Segment anything,” arXiv preprint arXiv:2304.02643 , 2023

2023

Later among the works it cites.

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” arXiv preprint arXiv:2305.14314 , 2023

2023

Later among the works it cites.

X. Fu, Z. Wang, S. Li, R. K. Gupta, N. Mireshghallah, T. Berg-Kirkpatrick, and E. Fernandes, “Misusing tools in large language models with visual adversarial examples,” arXiv preprint arXiv:2310.03185 , 2023

2023

Later among the works it cites.

L. Bailey, E. Ong, S. Russell, and S. Emmons, “Image hijacks: Adversarial images can control generative models at runtime,” arXiv preprint arXiv:2309.00236 , 2023

2023

Later among the works it cites.

A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043 , 2023

2023

Later among the works it cites.

E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt, “Automatically auditing large language models via discrete optimization,” in International Conference on Machine Learning . PMLR, 2023, pp. 15 307–15 329

2023

Later among the works it cites.

H. Liu, W. Zhou, D. Chen, H. Fang, H. Bian, K. Liu, W. Zhang, and N. Yu, “Coherent adversarial deepfake video generation,” Signal Processing , vol. 203, p. 108790, 2023

2023

Later among the works it cites.

H. J. Lee and Y. M. Ro, “Defending video recognition model against adversarial perturbations via defense patterns,” IEEE Transactions on Dependable and Secure Computing , 2023

2023

Later among the works it cites.

Y. Wu, X. Li, Y. Liu, P. Zhou, and L. Sun, “Jailbreaking gpt-4v via self-adversarial attacks with system prompts,” arXiv preprint arXiv:2311.09127 , 2023

2023

Later among the works it cites.

Y. Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, “Defending chatgpt against jailbreak attack via self-reminders,” Nature Machine Intelligence , vol. 5, no. 12, pp. 1486–1496, 2023

2023

Later among the works it cites.

Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and Y. Liu, “Prompt injection attack against llm-integrated applications,” arXiv preprint arXiv:2306.05499 , 2023

2023

Later among the works it cites.

N. Carlini, M. Jagielski, C. A. Choquette-Choo, D. Paleka, W. Pearce, H. Anderson, A. Terzis, K. Thomas, and F. Tramèr, “Poisoning web-scale training datasets is practical,” arXiv preprint arXiv:2302.10149 , 2023

2023

Later among the works it cites.

M.-H. Van and X. Wu, “Detecting and correcting hate speech in multimodal memes with large visual language model,” arXiv preprint arXiv:2311.06737 , 2023

2023

Later among the works it cites.

Z. Wei, Y. Wang, and Y. Wang, “Jailbreak and guard aligned language models with only few in-context demonstrations,” arXiv preprint arXiv:2310.06387 , 2023

2023

Later among the works it cites.

A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,” arXiv preprint arXiv:2310.03684 , 2023

2023

Later among the works it cites.

X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li, “Better aligning text-to-image models with human preference,” arXiv preprint arXiv:2303.14420 , 2023

2023

Later among the works it cites.

H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang, “Raft: Reward ranked finetuning for generative foundation model alignment,” arXiv preprint arXiv:2304.06767 , 2023

2023

Later among the works it cites.

M. Masood, M. Nawaz, K. M. Malik, A. Javed, A. Irtaza, and H. Malik, “Deepfakes generation and detection: State-of-the-art, open challenges, countermeasures, and way forward,” Applied intelligence , vol. 53, no. 4, pp. 3974–4026, 2023

2023

Later among the works it cites.

D. Wodajo, S. Atnafu, and Z. Akhtar, “Deepfake video detection using generative convolutional vision transformer,” arXiv preprint arXiv:2307.07036 , 2023

2023

Later among the works it cites.

W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, “Detecting pretraining data from large language models,” arXiv preprint arXiv:2310.16789 , 2023

2023

Later among the works it cites.

S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry, “Trak: Attributing model behavior at scale,” arXiv preprint arXiv:2303.14186 , 2023

2023

Later among the works it cites.

J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, “A watermark for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 17 061–17 084

2023

Later among the works it cites.

Y. Cui, J. Ren, H. Xu, P. He, H. Liu, L. Sun, and J. Tang, “Diffusionshield: A watermark for copyright protection against generative diffusion models,” arXiv preprint arXiv:2306.04642 , 2023

2023

Later among the works it cites.

P. Fernandez, G. Couairon, H. Jégou, M. Douze, and T. Furon, “The stable signature: Rooting watermarks in latent diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 22 466–22 477

2023

Later among the works it cites.

Z. Zhang, L. Lei, L. Wu, R. Sun, Y. Huang, C. Long, X. Liu, X. Lei, J. Tang, and M. Huang, “Safetybench: Evaluating the safety of large language models with multiple choice questions,” arXiv preprint arXiv:2309.07045 , 2023

2023

Later among the works it cites.

X. Wang, X. Yi, H. Jiang, S. Zhou, Z. Wei, and X. Xie, “Tovilag: Your visual-language generative model is also an evildoer,” arXiv preprint arXiv:2312.11523 , 2023

2023

Later among the works it cites.

Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang, “Figstep: Jailbreaking large vision-language models via typographic visual prompts,” arXiv preprint arXiv:2311.05608 , 2023

2023

Later among the works it cites.

X. Liu, Y. Zhu, Y. Lan, C. Yang, and Y. Qiao, “Query-relevant images jailbreak large multi-modal models,” arXiv preprint arXiv:2311.17600 , 2023

2023

Later among the works it cites.

Y. He, S. Yang, H. Chen, X. Cun, M. Xia, Y. Zhang, X. Wang, R. He, Q. Chen, and Y. Shan, “Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models,” in The Twelfth International Conference on Learning Representations , 2023

2023

Later among the works it cites.

S. Zhou, P. Yang, J. Wang, Y. Luo, and C. C. Loy, “Upscale-a-video: Temporal-consistent diffusion model for real-world video super-resolution,” arXiv preprint arXiv:2312.06640 , 2023

2023

Later among the works it cites.

R. S. Roman, Y. Adi, A. Deleforge, R. Serizel, G. Synnaeve, and A. Défossez, “From discrete tokens to high-fidelity audio using multi-band diffusion,” arXiv preprint arXiv:2308.02560 , 2023

2023

Later among the works it cites.

Y. Yao, P. Li, B. Chen, and A. Wang, “Jen-1 composer: A unified framework for high-fidelity multi-track music generation,” arXiv preprint arXiv:2310.19180 , 2023

2023

Later among the works it cites.

X. Huang, W. Li, J. Hu, H. Chen, and Y. Wang, “Refsr-nerf: Towards high fidelity and super resolution view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 8244–8253

2023

Later among the works it cites.

F.-Y. Wang, W. Chen, G. Song, H.-J. Ye, Y. Liu, and H. Li, “Gen-l-video: Multi-text to long video generation via temporal co-denoising,” arXiv preprint arXiv:2305.18264 , 2023

Original

2023

Later among the works it cites.

J. Yoo, S. Kim, D. Lee, C. Kim, and S. Hong, “Towards end-to-end generative modeling of long videos with memory-efficient bidirectional transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 888–22 897

2023

Later among the works it cites.

L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3836–3847

2023

Later among the works it cites.

M. Zhao, R. Wang, F. Bao, C. Li, and J. Zhu, “Controlvideo: Adding conditional control for one shot text-to-video editing,” arXiv preprint arXiv:2305.17098 , 2023

2023

Later among the works it cites.

J. Ma, M. Zhao, C. Chen, R. Wang, D. Niu, H. Lu, and X. Lin, “Glyphdraw: Learning to draw chinese characters in image synthesis models coherently,” arXiv preprint arXiv:2303.17870 , 2023

2023

Later among the works it cites.

S. Woo, B. Park, H. Go, J.-Y. Kim, and C. Kim, “Harmonyview: Harmonizing consistency and diversity in one-image-to-3d,” arXiv preprint arXiv:2312.15980 , 2023

2023

Later among the works it cites.

J. Ye, P. Wang, K. Li, Y. Shi, and H. Wang, “Consistent-1-to-3: Consistent image to 3d view synthesis via geometry-aware diffusion models,” arXiv preprint arXiv:2310.03020 , 2023

2023

Later among the works it cites.

P. Wang, S. Wang, J. Lin, S. Bai, X. Zhou, J. Zhou, X. Wang, and C. Zhou, “One-peace: Exploring one general representation model toward unlimited modalities,” arXiv preprint arXiv:2305.11172 , 2023

2023

Later among the works it cites.

C. Boletsis, A. Lie, O. Prillard, K. Husby, and J. Li, “The invizar project: Augmented reality visualization for non-destructive testing data from jacket platforms,” 2023

2023

Later among the works it cites.

X. Wu, C. Li, R. Y. Aminabadi, Z. Yao, and Y. He, “Understanding int4 quantization for language models: latency speedup, composability, and failure cases,” in International Conference on Machine Learning . PMLR, 2023, pp. 37 524–37 539

2023

Later among the works it cites.

Y. Qu, X. Shen, X. He, M. Backes, S. Zannettou, and Y. Zhang, “Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , 2023, pp. 3403–3417

2023

Later among the works it cites.

C. Min, D. Zhao, L. Xiao, Y. Nie, and B. Dai, “Uniworld: Autonomous driving pre-training via world models,” arXiv preprint arXiv:2308.07234 , 2023

2023

Later among the works it cites.

D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,” arXiv preprint arXiv:2301.04104 , 2023

2023

Later among the works it cites.

OpenAI, “Video generation models as world simulators,” OpenAI, Tech. Rep., 2024. [Online]. Available: https://openai.com/research/video-generation-models-as-world-simulators

2024

Closest in time.

O. Bar-Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, Y. Li, T. Michaeli et al. , “Lumiere: A space-time diffusion model for video generation,” arXiv preprint arXiv:2401.12945 , 2024

2024

Closest in time.

H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan, “Videocrafter2: Overcoming data limitations for high-quality video diffusion models,” arXiv preprint arXiv:2401.09047 , 2024

2024

Closest in time.

J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, “Simple and controllable music generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He et al. , “Naturalspeech: End-to-end text-to-speech synthesis with human-level quality,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

2024

Closest in time.

Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang et al. , “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” arXiv preprint arXiv:2403.03100 , 2024

2024

Closest in time.

F. Long, Z. Qiu, T. Yao, and T. Mei, “Videodrafter: Content-consistent multi-scene video generation with llm,” arXiv preprint arXiv:2401.01256 , 2024

2024

Closest in time.

T. Wu, G. Yang, Z. Li, K. Zhang, Z. Liu, L. Guibas, D. Lin, and G. Wetzstein, “Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024

2024

Closest in time.

S. Liu, A. S. Hussain, C. Sun, and Y. Shan, “Music understanding llama: Advancing text-to-music generation with question answering and captioning,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 286–290

2024

Closest in time.

R. Yuan, H. Lin, Y. Wang, Z. Tian, S. Wu, T. Shen, G. Zhang, Y. Wu, C. Liu, Z. Zhou et al. , “Chatmusician: Understanding and generating music intrinsically with llm,” arXiv preprint arXiv:2402.16153 , 2024

2024

Closest in time.

S. Ding, Z. Liu, X. Dong, P. Zhang, R. Qian, C. He, D. Lin, and J. Wang, “Songcomposer: A large language model for lyric and melody composition in song generation,” arXiv preprint arXiv:2402.17645 , 2024

2024

Closest in time.

S.-L. Wu, X. Chang, G. Wichern, J.-w. Jung, F. Germain, J. Le Roux, and S. Watanabe, “Improving audio captioning models with fine-grained audio features, text embedding supervision, and llm mix-up augmentation,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 316–320

2024

Closest in time.

S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196 , 2024

2024

Closest in time.

Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, W. Ye, Y. Zhang, Y. Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,” ACM Trans. Intell. Syst. Technol. , vol. 15, no. 3, mar 2024. [Online]. Available: https://doi.org/10.1145/3641289

2024

Closest in time.

S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

Y. Xing, Y. He, Z. Tian, X. Wang, and Q. Chen, “Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,” arXiv preprint arXiv:2402.17723 , 2024

2024

Closest in time.

X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu, “Ella: Equip diffusion models with llm for enhanced semantic alignment,” arXiv preprint arXiv:2403.05135 , 2024

2024

Closest in time.

S. Zhao, S. Hao, B. Zi, H. Xu, and K.-Y. K. Wong, “Bridging different language models and generative vision models for text-to-image generation,” arXiv preprint arXiv:2403.07860 , 2024

2024

Closest in time.

Y. Lu, X. Yang, X. Li, X. E. Wang, and W. Y. Wang, “Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra et al. , “Language is not all you need: Aligning perception with language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

W. Zhu, J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi, “Multimodal c4: An open, billion-scale corpus of images interleaved with text,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

2024

Closest in time.

S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang et al. , “Datacomp: In search of the next generation of multimodal datasets,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

J. Chen, Y. Huang, T. Lv, L. Cui, Q. Chen, and F. Wei, “Textdiffuser: Diffusion models as text painters,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

Y. Yang, D. Gui, Y. Yuan, W. Liang, H. Ding, H. Hu, and K. Chen, “Glyphcontrol: Glyph conditional control for visual text generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

J. Qin, J. Wu, W. Chen, Y. Ren, H. Li, H. Wu, X. Xiao, R. Wang, and S. Wen, “Diffusiongpt: Llm-driven text-to-image generation system,” arXiv preprint arXiv:2401.10061 , 2024

2024

Closest in time.

L. Yang, Z. Yu, C. Meng, M. Xu, S. Ermon, and B. Cui, “Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms,” arXiv preprint arXiv:2401.11708 , 2024

2024

Closest in time.

——, “Diffeditor: Boosting accuracy and flexibility on diffusion-based image editing,” arXiv preprint arXiv:2402.02583 , 2024

2024

Closest in time.

Y. Huang, L. Xie, X. Wang, Z. Yuan, X. Cun, Y. Ge, J. Zhou, C. Dong, R. Huang, R. Zhang et al. , “Smartedit: Exploring complex instruction-based image editing with multimodal large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024

2024

Closest in time.

Y. Ma, Y. He, X. Cun, X. Wang, S. Chen, X. Li, and Q. Chen, “Follow your pose: Pose-guided text-to-video generation using pose-free videos,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 5, 2024, pp. 4117–4125

2024

Closest in time.

Y. He, H. Liu, H. Chen, X. Cun, X. Wang, Y. Shan et al. , “Make-your-video: Customized video generation using textual and structural guidance.” IEEE Transactions on Visualization and Computer Graphics , 2024

2024

Closest in time.

Y. Ma, Y. He, H. Wang, A. Wang, C. Qi, C. Cai, X. Li, Z. Li, H.-Y. Shum, W. Liu et al. , “Follow-your-click: Open-domain regional image animation via short prompts,” arXiv preprint arXiv:2403.08268 , 2024

2024

Closest in time.

T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H.-w. Chao, B. E. Jeon, Y. Fang, H.-Y. Lee, J. Ren, M.-H. Yang et al. , “Panda-70m: Captioning 70m videos with multiple cross-modality teachers,” arXiv preprint arXiv:2402.19479 , 2024

2024

Closest in time.

X. Li, Q. Zhang, D. Kang, W. Cheng, Y. Gao, J. Zhang, Z. Liang, J. Liao, Y.-P. Cao, and Y. Shan, “Advances in 3d generation: A survey,” arXiv preprint arXiv:2401.17807 , 2024

2024

Closest in time.

M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre et al. , “Objaverse-xl: A universe of 10m+ 3d objects,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen, “Motiongpt: Human motion as a foreign language,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

B. Yang, W. Dong, L. Ma, W. Hu, X. Liu, Z. Cui, and Y. Ma, “Dreamspace: Dreaming your room space with text-driven panoramic texture propagation,” in 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR) . IEEE, 2024, pp. 650–660

2024

Closest in time.

D. Wang, H. Meng, Z. Cai, Z. Shao, Q. Liu, L. Wang, M. Fan, Y. Shan, X. Zhan, and Z. Wang, “Headevolver: Text to head avatars via locally learnable mesh deformation,” arXiv preprint arXiv:2403.09326 , 2024

2024

Closest in time.

Y. Kant, Z. Wu, M. Vasilkovsky, G. Qian, J. Ren, R. A. Guler, B. Ghanem, S. Tulyakov, I. Gilitschenski, and A. Siarohin, “Spad: Spatially aware multiview diffusers,” arXiv preprint arXiv:2402.05235 , 2024

2024

Closest in time.

J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu, “Lgm: Large multi-view gaussian model for high-resolution 3d content creation,” arXiv preprint arXiv:2402.05054 , 2024

2024

Closest in time.

X. Yinghao, S. Zifan, Y. Wang, C. Hansheng, Y. Ceyuan, P. Sida, S. Yujun, and W. Gordon, “Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation,” 2024

2024

Closest in time.

Z. Hu, A. Iscen, A. Jain, T. Kipf, Y. Yue, D. A. Ross, C. Schmid, and A. Fathi, “Scenecraft: An llm agent for synthesizing 3d scene as blender code,” arXiv preprint arXiv:2403.01248 , 2024

2024

Closest in time.

Z. Pan, J. Lu, X. Zhu, and L. Zhang, “Enhancing high-resolution 3d generation through pixel-wise gradient clipping,” in International Conference on Learning Representations (ICLR) , 2024

2024

Closest in time.

G. Qian, J. Cao, A. Siarohin, Y. Kant, C. Wang, M. Vasilkovsky, H.-Y. Lee, Y. Fang, I. Skorokhodov, P. Zhuang et al. , “Atom: Amortized text-to-mesh using 2d diffusion,” arXiv preprint arXiv:2402.00867 , 2024

2024

Closest in time.

Z. Wu, P. Zhou, X. Yi, X. Yuan, and H. Zhang, “Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior,” arXiv preprint arXiv:2401.09050 , 2024

2024

Closest in time.

J. Wu, X. Gao, X. Liu, Z. Shen, C. Zhao, H. Feng, J. Liu, and E. Ding, “Hd-fusion: Detailed text-to-3d generation leveraging multiple noise estimation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2024, pp. 3202–3211

2024

Closest in time.

Y. Lin, R. Clark, and P. Torr, “Dreampolisher: Towards high-quality text-to-3d generation via geometric diffusion,” arXiv preprint arXiv:2403.17237 , 2024

2024

Closest in time.

W. Kang, X. Yang, Z. Yao, F. Kuang, Y. Yang, L. Guo, L. Lin, and D. Povey, “Libriheavy: a 50,000 hours asr corpus with punctuation casing and context,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 10 991–10 995

2024

Closest in time.

J. Zhan, J. Dai, J. Ye, Y. Zhou, D. Zhang, Z. Liu, X. Zhang, R. Yuan, G. Zhang, L. Li et al. , “Anygpt: Unified multimodal llm with discrete sequence modeling,” arXiv preprint arXiv:2402.12226 , 2024

2024

Closest in time.

Q. Deng, Q. Yang, R. Yuan, Y. Huang, Y. Wang, X. Liu, Z. Tian, J. Pan, G. Zhang, H. Lin et al. , “Composerx: Multi-agent symbolic music composition with llms,” arXiv preprint arXiv:2404.18081 , 2024

2024

Closest in time.

R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

W. Shen, C. Li, H. Chen, M. Yan, X. Quan, H. Chen, J. Zhang, and F. Huang, “Small llms are weak tool learners: A multi-llm agent,” arXiv preprint arXiv:2401.07324 , 2024

2024

Closest in time.

J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,” arXiv preprint arXiv:2401.16158 , 2024

2024

Closest in time.

C. Wang, W. Luo, Q. Chen, H. Mai, J. Guo, S. Dong, X. M. Xuan, Z. Li, L. Ma, and S. Gao, “Mllm-tool: A multimodal large language model for tool agent learning,” arXiv preprint arXiv:2401.10727 , 2024

2024

Closest in time.

J. Chen, X. Li, X. Ye, C. Li, Z. Fan, and H. Zhao, “Idea-2-3d: Collaborative lmm agents enable 3d model generation from interleaved multimodal inputs,” arXiv preprint arXiv:2404.04363 , 2024

2024

Closest in time.

R. Liu, A. Khakzar, J. Gu, Q. Chen, P. Torr, and F. Pizzati, “Latent guard: a safety framework for text-to-image generation,” arXiv preprint arXiv:2404.08031 , 2024

2024

Closest in time.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

R. Pi, T. Han, W. Xiong, J. Zhang, R. Liu, R. Pan, and T. Zhang, “Strengthening multimodal large language model with bootstrapped preference optimization,” arXiv preprint arXiv:2403.08730 , 2024

2024

Closest in time.

Z. Wang, C. Chen, Y. Zeng, L. Lyu, and S. Ma, “Where did i come from? origin attribution of ai-generated images,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

H. Lin, Z. Luo, B. Wang, R. Yang, and J. Ma, “Goat-bench: Safety insights to large multimodal models through meme-based social abuse,” arXiv preprint arXiv:2401.01523 , 2024

2024

Closest in time.

A. Avetisyan, C. Xie, H. Howard-Jenkins, T.-Y. Yang, S. Aroudj, S. Patra, F. Zhang, D. Frost, L. Holland, C. Orme, J. Engel, E. Miller, R. Newcombe, and V. Balntas, “Scenescript: Reconstructing scenes with an autoregressive structured language model,” 2024

2024

Closest in time.

L. Guo, Y. He, H. Chen, M. Xia, X. Cun, Y. Wang, S. Huang, Y. Zhang, X. Wang, Q. Chen et al. , “Make a cheap scaling: A self-cascade diffusion model for higher-resolution adaptation,” arXiv preprint arXiv:2402.10491 , 2024

2024

Closest in time.

Y. Xu, T. Park, R. Zhang, Y. Zhou, E. Shechtman, F. Liu, J.-B. Huang, and D. Liu, “Videogigagan: Towards detail-rich video super-resolution,” arXiv preprint arXiv:2404.12388 , 2024

2024

Closest in time.

Y. Zhang, Y. Wei, X. Lin, Z. Hui, P. Ren, X. Xie, X. Ji, and W. Zuo, “Videoelevator: Elevating video generation quality with versatile text-to-image diffusion models,” arXiv preprint arXiv:2403.05438 , 2024

2024

Closest in time.

R. Henschel, L. Khachatryan, D. Hayrapetyan, H. Poghosyan, V. Tadevosyan, Z. Wang, S. Navasardyan, and H. Shi, “Streamingt2v: Consistent, dynamic, and extendable long video generation from text,” arXiv preprint arXiv:2403.14773 , 2024

2024

Closest in time.

L. Lin, G. Xia, Y. Zhang, and J. Jiang, “Arrange, inpaint, and refine: Steerable long-term music audio generation and editing via content-based controls,” 2024

2024

Closest in time.

C. Chen, X. Yang, F. Yang, C. Feng, Z. Fu, C.-S. Foo, G. Lin, and F. Liu, “Sculpt3d: Multi-view consistent text-to-3d generation with sparse 3d prior,” arXiv preprint arXiv:2403.09140 , 2024

2024

Closest in time.

Q. Zuo, X. Gu, L. Qiu, Y. Dong, Z. Zhao, W. Yuan, R. Peng, S. Zhu, Z. Dong, L. Bo et al. , “Videomv: Consistent multi-view generation based on large video generative model,” arXiv preprint arXiv:2403.12010 , 2024

2024

Closest in time.

S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, and J. Liu, “Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

H. Liu, W. Yan, M. Zaharia, and P. Abbeel, “World model on million-length video and language with ringattention,” arXiv preprint arXiv:2402.08268 , 2024

2024

Closest in time.