DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models

V. Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” in Proceedings of the 26th annual conference on Computer graphics and interactive techniques , 1999, pp. 187–194

1999

Earlier work this paper cites.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing , vol. 13, no. 4, pp. 600–612, 2004

2004

Earlier work this paper cites.

L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008

2008

Earlier work this paper cites.

N. D. Narvekar and L. J. Karam, “A no-reference perceptual image sharpness metric based on a cumulative probability of blur detection,” in 2009 International Workshop on Quality of Multimedia Experience . IEEE, 2009, pp. 87–91

2009

Earlier work this paper cites.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in neural information processing systems , vol. 27, 2014

2014

Earlier work this paper cites.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML , 2015

2015

Earlier work this paper cites.

J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in Asian conference on computer vision . Springer, 2016, pp. 251–263

2016

Earlier work this paper cites.

S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing obama: learning lip sync from audio,” ACM Transactions on Graphics (ToG) , vol. 36, no. 4, pp. 1–13, 2017

2017

Earlier work this paper cites.

J. S. Chung, A. Jamaludin, and A. Zisserman, “You said that?” arXiv preprint arXiv:1705.02966 , 2017

Original

2017

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

2017

Earlier work this paper cites.

Y. Song, J. Zhu, D. Li, X. Wang, and H. Qi, “Talking face generation by conditional recurrent adversarial network,” arXiv preprint arXiv:1804.04786 , 2018

Original

2018

Earlier work this paper cites.

L. Chen, Z. Li, R. K. Maddox, Z. Duan, and C. Xu, “Lip movements generation at a glance,” in ECCV , 2018, pp. 520–535

2018

Earlier work this paper cites.

J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” arXiv preprint arXiv:1806.05622 , 2018

Original

2018

Earlier work this paper cites.

S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” PloS one , vol. 13, no. 5, p. e0196391, 2018

2018

Earlier work this paper cites.

O. Fried, A. Tewari, M. Zollhöfer, A. Finkelstein, E. Shechtman, D. B. Goldman, K. Genova, Z. Jin, C. Theobalt, and M. Agrawala, “Text-based editing of talking-head video,” ACM Transactions on Graphics (TOG) , vol. 38, no. 4, pp. 1–14, 2019

2019

Earlier work this paper cites.

N. Sadoughi and C. Busso, “Speech-driven expressive talking lips with conditional sequential generative adversarial networks,” IEEE Transactions on Affective Computing , vol. 12, no. 4, pp. 1031–1044, 2019

2019

Earlier work this paper cites.

K. Vougioukas, S. Petridis, and M. Pantic, “Realistic speech-driven facial animation with gans,” International Journal of Computer Vision , pp. 1–16, 2019

2019

Earlier work this paper cites.

H. Zhou, Y. Liu, Z. Liu, P. Luo, and X. Wang, “Talking face generation by adversarially disentangled audio-visual representation,” in AAAI , vol. 33, no. 01, 2019, pp. 9299–9306

2019

Earlier work this paper cites.

L. Chen, R. K. Maddox, Z. Duan, and C. Xu, “Hierarchical cross-modal talking face generation with dynamic pixel-wise loss,” in CVPR , 2019, pp. 7832–7841

2019

Earlier work this paper cites.

J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in CVPR , 2019, pp. 4690–4699

2019

Earlier work this paper cites.

D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. J. Black, “Capture, learning, and synthesis of 3d speaking styles,” in CVPR , 2019, pp. 10 101–10 111

2019

Earlier work this paper cites.

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851

2020

Earlier work this paper cites.

D. Das, S. Biswas, S. Sinha, and B. Bhowmick, “Speech-driven facial animation using cascaded gans for learning of motion and texture,” in ECCV . Springer, 2020, pp. 408–424

2020

Earlier work this paper cites.

K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y. Qiao, and C. C. Loy, “Mead: A large-scale audio-visual dataset for emotional talking-face generation,” in ECCV . Springer, 2020, pp. 700–717

2020

Earlier work this paper cites.

R. Yi, Z. Ye, J. Zhang, H. Bao, and Y.-J. Liu, “Audio-driven talking face video generation with learning-based personalized head pose,” arXiv preprint arXiv:2002.10137 , 2020

Original

2020

Earlier work this paper cites.

J. Thies, M. Elgharib, A. Tewari, C. Theobalt, and M. Nießner, “Neural voice puppetry: Audio-driven facial reenactment,” in ECCV . Springer, 2020, pp. 716–731

2020

Earlier work this paper cites.

L. Song, W. Wu, C. Qian, R. He, and C. C. Loy, “Everybody’s talkin’: Let me talk as you want,” arXiv preprint arXiv:2001.05201 , 2020

Original

2020

Earlier work this paper cites.

K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 484–492

2020

Earlier work this paper cites.

Y. Zhou, X. Han, E. Shechtman, J. Echevarria, E. Kalogerakis, and D. Li, “Makelttalk: speaker-aware talking-head animation,” ACM Transactions on Graphics (TOG) , vol. 39, no. 6, pp. 1–15, 2020

2020

Earlier work this paper cites.

L. Chen, G. Cui, C. Liu, Z. Li, Z. Kou, Y. Xu, and C. Xu, “Talking-head generation with rhythmic head motion,” in ECCV . Springer, 2020, pp. 35–51

2020

Earlier work this paper cites.

P. Safari, M. India, and J. Hernando, “Self-attention encoding and pooling for speaker recognition,” arXiv preprint arXiv:2008.01077 , 2020

Original

2020

Earlier work this paper cites.

J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 , 2020

Original

2020

Earlier work this paper cites.

P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” NeurIPS , 2021

2021

Earlier work this paper cites.

X. Ji, H. Zhou, K. Wang, W. Wu, C. C. Loy, X. Cao, and F. Xu, “Audio-driven emotional video portraits,” in CVPR , 2021, pp. 14 080–14 089

2021

Earlier work this paper cites.

Y. Lu, J. Chai, and X. Cao, “Live speech portraits: real-time photorealistic talking-head animation,” ACM Transactions on Graphics (TOG) , vol. 40, no. 6, pp. 1–17, 2021

2021

Earlier work this paper cites.

A. Lahiri, V. Kwatra, C. Frueh, J. Lewis, and C. Bregler, “Lipsync3d: Data-efficient learning of personalized 3d talking faces from video using pose and lighting normalization,” in CVPR , 2021, pp. 2755–2764

2021

Earlier work this paper cites.

C. Zhang, S. Ni, Z. Fan, H. Li, M. Zeng, M. Budagavi, and X. Guo, “3d talking face with personalized pose dynamics,” IEEE Transactions on Visualization and Computer Graphics , 2021

2021

Cited alongside, same era.

C. Zhang, Y. Zhao, Y. Huang, M. Zeng, S. Ni, M. Budagavi, and X. Guo, “Facial: Synthesizing dynamic talking face with implicit attribute learning,” in ICCV , 2021, pp. 3867–3876

2021

Cited alongside, same era.

Y. Guo, K. Chen, S. Liang, Y. Liu, H. Bao, and J. Zhang, “Ad-nerf: Audio driven neural radiance fields for talking head synthesis,” arXiv preprint arXiv:2103.11078 , 2021

Original

2021

Cited alongside, same era.

Z. Zhang, L. Li, Y. Ding, and C. Fan, “Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,” in CVPR , 2021, pp. 3661–3670

2021

Cited alongside, same era.

J. Wang, K. Zhao, Y. Ma, S. Zhang, Y. Zhang, Y. Shen, D. Zhao, and J. Zhou, “Facecomposer: A unified model for versatile facial content creation,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023

2023

Closest in time.

Z. Ye, Z. Jiang, Y. Ren, J. Liu, J. He, and Z. Zhao, “Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis,” arXiv preprint arXiv:2301.13430 , 2023

Original

2023

Closest in time.

W. Zhang, X. Cun, X. Wang, Y. Zhang, X. Shen, Y. Guo, Y. Shan, and F. Wang, “Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 8652–8661

2023

Closest in time.

Z. Sheng, L. Nie, M. Liu, Y. Wei, and Z. Gao, “Towards fine-grained talking face generation,” IEEE Transactions on Image Processing , 2023

2023

Closest in time.