2021

Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure

Aristidou, Andreas, Yiannakidis, Anastasios, Aberman, Kfir et al.

Understand

Synthesizing human motion with a global structure, such as a choreography, is a challenging task.

Existing methods tend to concentrate on local smooth pose transitions and neglect the global context or the theme of the motion.
In this work, we present a music-driven motion synthesis framework that generates long-term sequences of human motions which are synchronized with the input beats, and jointly form a global structure that respects a specific dance genre.
In addition, our framework enables generation of diverse motions that are controlled by the content of the music, and not only by the beat.

Built on

S. H. Fraleigh, Dance and the Lived Body: A Descriptive Aesthetics . University of Pittsburgh Press, 1987
1987
Earlier work this paper cites.
L. Kovar, M. Gleicher, and F. Pighin, “Motion graphs,” ACM Trans. Graph. , vol. 21, no. 3, pp. 473–482, Jul. 2002
2002
Earlier work this paper cites.
J. Lee, J. Chai, P. S. A. Reitsma, J. K. Hodgins, and N. S. Pollard, “Interactive control of avatars animated with human motion data,” ACM Trans. Graph. , vol. 21, no. 3, p. 491–500, Jul. 2002
2002
Earlier work this paper cites.
T.-h. Kim, S. I. Park, and S. Y. Shin, “Rhythmic-motion synthesis based on motion-beat analysis,” ACM Trans. Graph. , vol. 22, no. 3, pp. 392–401, Jul. 2003
2003
Earlier work this paper cites.
M. Mckinney and J. Breebaart, “Features for audio and music classification,” in Proceedings of the International Symposium on Music Information Retrieval , ser. ISMIR’03, 2003, pp. 151–158
2003
Earlier work this paper cites.
K. Grochow, S. L. Martin, A. Hertzmann, and Z. Popovic̀, “Style-based inverse kinematics,” ACM Trans. Graph. , vol. 23, no. 3, p. 522–531, Aug. 2004
2004
Earlier work this paper cites.
C. K. Liu, A. Hertzmann, and Z. Popovic̀, “Learning physics-based motion style with nonlinear inverse optimization,” in ACM SIGGRAPH 2005 Papers , ser. SIGGRAPH ’05. NY, USA: ACM, 2005, p. 1071–1081
2005
Earlier work this paper cites.
J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M. B. Sandler, “A tutorial on onset detection in music signals,” IEEE Transactions on Speech and Audio Processing , vol. 13, no. 5, pp. 1035–1047, Sep. 2005
2005
Earlier work this paper cites.
T. Shiratori, A. Nakazawa, and K. Ikeuchi, “Dancing-to-music character animation,” Computer Graphics Forum , vol. 25, no. 3, pp. 449–458, September 2006
2006
Earlier work this paper cites.
A. Safonova and J. K. Hodgins, “Construction and optimal search of interpolated motion graphs,” ACM Trans. Graph. , vol. 26, no. 3, Jul. 2007
2007
Earlier work this paper cites.
A. Treuille, Y. Lee, and Z. Popovic̀, “Near-optimal character animation with continuous control,” ACM Trans. Graph. , vol. 26, no. 3, p. 7–es, Jul. 2007
2007
Earlier work this paper cites.
K. Yin, K. Loken, and M. van de Panne, “Simbicon: Simple biped locomotion control,” ACM Trans. Graph. , vol. 26, no. 3, p. 105–es, Jul. 2007
2007
Earlier work this paper cites.
M. F. McKinney, D. Moelants, M. Davies, and A. Klapuri, “Evaluation of audio beat tracking and music tempo extraction algorithms,” Journal of New Music Research , vol. 36, no. 1, pp. 1–16, 2007
2007
Earlier work this paper cites.
D. P. Ellis, “Beat tracking by dynamic programming,” Journal of New Music Research , vol. 36, no. 1, pp. 51–60, 2007
2007
Earlier work this paper cites.
M. Müller, Information Retrieval for Music and Motion . Berlin, Heidelberg: Springer-Verlag, 2007
2007
Earlier work this paper cites.
P. Beaudoin, S. Coros, M. van de Panne, and P. Poulin, “Motion-motif graphs,” in Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation , ser. SCA ’08. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association, 2008, pp. 117–126
2008
Earlier work this paper cites.
H. P. H. Shum, T. Komura, M. Shiraishi, and S. Yamazaki, “Interaction patches for multi-character animation,” ACM Trans. Graph. , vol. 27, no. 5, Dec. 2008
2008
Earlier work this paper cites.
D. Sauer and Y.-H. Yang, “Music-driven character animation,” ACM Trans. Multimedia Comput. Commun. Appl. , vol. 5, no. 4, pp. 27:1–27:16, Nov. 2009
2009
Earlier work this paper cites.
S. L. Foster, Choreographing empathy: kinesthesia in performance . Routledge, 2011
2011
Earlier work this paper cites.
A. Schulz and L. Velho, “Choreographics: An authoring environment for dance shows,” in ACM SIGGRAPH 2011 Posters . NY, USA: ACM, 2011
2011
Earlier work this paper cites.
F. Ofli, E. Erzin, Y. Yemez, and A. M. Tekalp, “Learn2dance: Learning statistical music-to-dance mappings for choreography synthesis,” Trans. Multi. , vol. 14, no. 3, pp. 747–759, Jun. 2012
2012
Earlier work this paper cites.
J. Min and J. Chai, “Motion graphs++: A compact generative model for semantic motion analysis and synthesis,” ACM Trans. Graph. , vol. 31, no. 6, pp. 153:1–153:12, Nov. 2012
2012
Earlier work this paper cites.
S. Levine, J. M. Wang, A. Haraux, Z. Popović, and V. Koltun, “Continuous character control with low-dimensional embeddings,” ACM Trans. Graph. , vol. 31, no. 4, pp. 28:1–28:10, Jul. 2012
2012
Earlier work this paper cites.
T. Geijtenbeek and N. Pronost, “Interactive character animation using simulated physics: A state-of-the-art review,” Comput. Graph. Forum , vol. 31, no. 8, p. 2492–2515, Dec. 2012
2012
Earlier work this paper cites.
S. Ha, Y. Ye, and C. K. Liu, “Falling and landing motion control for character animation,” ACM Trans. Graph. , vol. 31, no. 6, Nov. 2012
2012
Earlier work this paper cites.
R. Fan, S. Xu, and W. Geng, “Example-based automatic music-driven conventional dance motion synthesis,” IEEE Transactions on Visualization and Computer Graphics , vol. 18, no. 3, p. 501–515, Mar. 2012
2012
Earlier work this paper cites.
Y.-H. Yang and H. H. Chen, “Machine recognition of music emotion: A review,” ACM Trans. Intell. Syst. Technol. , vol. 3, no. 3, May 2012
2012
Earlier work this paper cites.
T. Geijtenbeek, M. van de Panne, and A. F. van der Stappen, “Flexible muscle-based locomotion for bipedal creatures,” ACM Trans. Graph. , vol. 32, no. 6, Nov. 2013
2013
Earlier work this paper cites.
M. Lee, K. Lee, and J. Park, “Music similarity-based approach to generating dance motion sequence,” Mult. Tools Appl. , vol. 62, no. 3, p. 895–912, Feb. 2013
2013
Earlier work this paper cites.
T. R. Langlois and D. L. James, “Inverse-foley animation: Synchronizing rigid-body motions to sound,” ACM Trans. Graph. , vol. 33, no. 4, Jul. 2014
2014
Earlier work this paper cites.
S. Fukayama and M. Goto, “Automated choreography synthesis using a gaussian process leveraging consumer-generated dance motions,” in Proceedings of the 11th Conference on Advances in Computer Entertainment Technology , ser. ACE ’14. NY, USA: ACM, 2014
2014
Earlier work this paper cites.

Similar

P. Hämäläinen, J. Rajamäki, and C. K. Liu, “Online control of simulated humanoids using particle belief propagation,” ACM Trans. Graph. , vol. 34, no. 4, Jul. 2015
2015
Cited alongside, same era.
A. Clegg, J. Tan, G. Turk, and C. K. Liu, “Animating human dressing,” ACM Trans. Graph. , vol. 34, no. 4, Jul. 2015
2015
Cited alongside, same era.
K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik, “Recurrent network models for human dynamics,” in Proceedings of the IEEE International Conference on Computer Vision , ser. ICCV’15. Washington, DC, USA: IEEE Computer Society, 2015, pp. 4346–4354
2015
Cited alongside, same era.
I.-K. Chiang, I. Spiro, S. Lee, A. Lees, J. Liu, C. Bregler, and Y. Liu, “Dancing with turks,” in Proceedings of the 23rd ACM International Conference on Multimedia , ser. MM ’15. NY, USA: ACM, 2015, pp. 241–250
2015
Cited alongside, same era.
B. McFee, C. Raffel, D. Liang, D. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “Librosa: Audio and music signal analysis in python,” in Proceedings of the 14th Python in Science Conference . SciPy, 2015
2015
Cited alongside, same era.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proceedings of the 3rd International Conference on Learning Representations , ser. ICLR’15, Y. Bengio and Y. LeCun, Eds., 2015
2015
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the 3rd International Conference on Learning Representations , ser. ICLR’15, Y. Bengio and Y. LeCun, Eds., 2015
2015
Cited alongside, same era.
L. Liu, M. V. D. Panne, and K. Yin, “Guided learning of control graphs for physics-based characters,” ACM Trans. Graph. , vol. 35, no. 3, May 2016
2016
Cited alongside, same era.
D. Holden, J. Saito, and T. Komura, “A deep learning framework for character motion synthesis and editing,” ACM Trans. Graph. , vol. 35, no. 4, pp. 138:1–138:11, Jul. 2016
2016
Cited alongside, same era.
A. Jain, A. R. Zamir, S. Savarese, and A. Saxena, “Structural-rnn: Deep learning on spatio-temporal graphs.” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , ser. CVPR ’16. IEEE, 2016, p. 5308–5317
2016
Cited alongside, same era.
A. Soga, Y. Yazaki, B. Umino, and M. Hirayama, “Body-part motion synthesis system for contemporary dance creation,” in ACM SIGGRAPH 2016 Posters . NY, USA: ACM, 2016
2016
Cited alongside, same era.
J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Proceedings of the European conference on Computer Vision , ser. ECCV’16. Springer, 2016, pp. 694–711
2016
Cited alongside, same era.

Then

H. Zhang, S. Starke, T. Komura, and J. Saito, “Mode-adaptive neural networks for quadruped motion control,” ACM Trans. Graph. , vol. 37, no. 4, pp. 145:1–145:11, Jul. 2018
2018
Later among the works it cites.
D. Pavllo, D. Grangier, and M. Auli, “Quaternet: A quaternion-based recurrent model for human motion,” in Proceedings of the British Machine Vision Conference , ser. BMVC ’18, Newcastle upon Tyne, UK, 2018
2018
Later among the works it cites.
L.-Y. Gui, Y. Wang, X. Liang, and J. M. F. Moura, “Adversarial geometry-aware human motion prediction,” in European Conference on Computer Vision , ser. ECCV ’18. Springer, October 2018
2018
Later among the works it cites.
X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne, “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills,” ACM Trans. Graph. , vol. 37, no. 4, pp. 143:1–143:14, Jul. 2018
2018
Later among the works it cites.
L. Liu and J. K. Hodgins, “Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning,” ACM Trans. Graph. , vol. 37, no. 4, pp. 142:1–142:14, Jul. 2018
2018
Later among the works it cites.
Y. Zhou, Z. Xu, C. Landreth, E. Kalogerakis, S. Maji, and K. Singh, “Visemenet: Audio-driven animator-centric speech animation,” ACM Trans. Graph. , vol. 37, no. 4, pp. 161:1–161:10, Jul. 2018
2018
Later among the works it cites.
N. Iwamoto, T. Kato, H. P. H. Shum, R. Kakitsuka, K. Hara, and S. Morishima, “DanceDJ: A 3d dance animation authoring system for live performance,” in Advances in Computer Entertainment Technology , A. D. Cheok, M. Inami, and T. Romão, Eds. Springer International, 2018, pp. 653–670
2018
Later among the works it cites.
A. Davis and M. Agrawala, “Visual rhythm and beat,” ACM Trans. Graph. , vol. 37, no. 4, Jul. 2018
2018
Later among the works it cites.
H.-Y. Lee, X. Yang, M.-Y. Liu, T.-C. Wang, Y.-D. Lu, M.-H. Yang, and J. Kautz, “Dancing to music,” in Advances in Neural Information Processing Systems , ser. NeurIPS, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 3586–3596
2019
Later among the works it cites.
H. Wang, E. S. L. Ho, H. P. H. Shum, and Z. Zhu, “Spatio-temporal manifold learning for human motions via long-horizon modeling,” IEEE Transactions on Visualization and Computer Graphics , 2019
2019
Later among the works it cites.
T. Kucherenko, D. Hasegawa, G. E. Henter, N. Kaneko, and H. Kjellström, “Analyzing input and output representations for speech-driven gesture generation,” in Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents , ser. IVA ’19. NY, USA: ACM, 2019, p. 97–104
2019
Later among the works it cites.
S. Lee, M. Park, K. Lee, and J. Lee, “Scalable muscle-actuated human simulation and control,” ACM Trans. Graph. , vol. 38, no. 4, pp. 73:1–73:13, Jul. 2019
2019
Later among the works it cites.
A. Aristidou, A. Shamir, and Y. Chrysanthou, “Digital dance ethnography: Organizing large dance collections,” ACM J. Comput. Cult. Herit. , vol. 12, no. 4, Nov. 2019
2019
Later among the works it cites.
J. Lee, S. Kim, and K. Lee, “Automatic choreography generation with convolutional encoder-decoder network,” in Proceedings of the 20th International Society for Music Information Retrieval Conference , ser. ISMIR ’19, 2019, pp. 894–899
2019
Later among the works it cites.
G. E. Henter, S. Alexanderson, and J. Beskow, “Moglow: Probabilistic and controllable motion synthesis using normalising flows,” ACM Trans. Graph. , vol. 39, no. 6, Nov. 2020
2020
Later among the works it cites.
S. Alexanderson, G. E. Henter, T. Kucherenko, and J. Beskow, “Style-controllable speech-driven gesture synthesis using normalising flows,” Computer Graphics Forum , vol. 39, no. 2, pp. 487–496, 2020
2020
Later among the works it cites.
H. Y. Ling, F. Zinno, G. Cheng, and M. Van De Panne, “Character controllers using motion vaes,” ACM Trans. Graph. , vol. 39, no. 4, Jul. 2020
2020
Later among the works it cites.
S. Ghorbani, C. Wloka, A. Etemad, M. A. Brubaker, and N. F. Troje, “Probabilistic character motion synthesis using a hierarchical deep latent variable model,” Comput. Graph. Forum , 2020
2020
Later among the works it cites.
S. Starke, Y. Zhao, T. Komura, and K. Zaman, “Local motion phases for learning multi-contact character movements,” ACM Trans. Graph. , vol. 39, no. 4, Jul. 2020
2020
Later among the works it cites.
H.-K. Kao and L. Su, “Temporally guided music-to-body-movement generation,” in Proceedings of the 28th ACM International Conference on Multimedia , ser. MM ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 147–155. [Online]. Available: https://doi.org/10.1145/3394171.3413848
2020
Later among the works it cites.
W. Zhuang, C. Wang, S. Xia, J. Chai, and Y. Wang, “Music2dance: Music-driven dance generation using wavenet,” 2020
2020
Later among the works it cites.
H. Ahn, J. Kim, K. Kim, and S. Oh, “Generative autoregressive networks for 3d dancing move synthesis from music,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3501–3508, 2020
2020
Later among the works it cites.
W. Zhuang, Y. Wang, J. Robinson, C. Wang, M. Shao, Y. Fu, and S. Xia, “Towards 3d dance motion synthesis and control,” 2020
2020
Later among the works it cites.
J. Li, Y. Yin, H. Chu, Y. Zhou, T. Wang, S. Fidler, and H. Li, “Learning to generate diverse dance motions with transformer,” 2020
2020
Later among the works it cites.
Z. Ye, H. Wu, J. Jia, Y. Bu, W. Chen, F. Meng, and Y. Wang, “ChoreoNet: Towards music to dance synthesis with choreographic action unit,” in Proceedings of the 28th ACM International Conference on Multimedia , ser. MM ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 744–752
2020
Later among the works it cites.
X. Ren, H. Li, Z. Huang, and Q. Chen, “Self-supervised dance video synthesis conditioned on music,” in Proceedings of the 28th ACM International Conference on Multimedia , ser. MM ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 46–54
2020
Later among the works it cites.
Wikipedia, “Dance: https://en.wikipedia.org/wiki/Dance . accessed online: 20/07/2021,” 2021
2021
Closest in time.
Z. Wang, J. Chai, and S. Xia, “Combining recurrent neural networks and adversarial training for human motion synthesis and control,” IEEE Transactions on Visualization and Computer Graphics , vol. 27, no. 1, p. 14–28, Jan. 2021
2021
Closest in time.
R. Huang, H. Hu, W. Wu, K. Sawada, and M. Zhang, “Dance revolution: Long sequence dance generation with music via curriculum learning,” in Proceedings of the International Conference on Learning Representations , ser. ICLR’21, 2021
2021
Closest in time.
K. Chen, Z. Tan, J. Lei, S.-H. Zhang, Y.-C. Guo, W. Zhang, and S.-M. Hu, “Choreomaster: Choreography-oriented music-driven dance synthesis,” ACM Trans. Graph. , vol. 40, no. 4, Jul. 2021
2021
Closest in time.
R. Li, S. Yang, D. A. Ross, and A. Kanazawa, “Learn to dance with AIST++: Music conditioned 3d dance generation,” 2021
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…