2022

GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech

Huang, Rongjie, Ren, Yi, Liu, Jinglin et al.

Understand

Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data.

This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice.
GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization.
Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity.

Built on

Multispeech: Multi-speaker text to speech with transformer
Original
Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., Qin, T., and Liu, T.-Y · 2006
Earlier work this paper cites.
Unsupervised domain adaptation by backpropagation
Ganin, Y. and Lempitsky, V · 2015
Earlier work this paper cites.
Librispeech: an asr corpus based on public domain audio books
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Instance normalization: The missing ingredient for fast stylization
Original
Ulyanov, D., Vedaldi, A., and Lempitsky, V · 2016
Earlier work this paper cites.
Deeper, broader and artier domain generalization
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M · 2017
Earlier work this paper cites.
Voxceleb: a large-scale speaker identification dataset
Original
Nagrani, A., Chung, J. S., and Zisserman, A · 2017
Earlier work this paper cites.
Neural discrete representation learning
van den Oord, A., Vinyals, O., and Kavukcuoglu, K · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Earlier work this paper cites.
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R., Wang, Q., Shen, J., Ren, F., Nguyen, P., Pang, R., Lopez Moreno, I., Wu, Y., et al · 2018
Earlier work this paper cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P · 2018
Earlier work this paper cites.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al · 2018
Earlier work this paper cites.
Towards end-to-end prosody transfer for expressive speech synthesis with tacotron
Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., and Saurous, R. A · 2018
Earlier work this paper cites.

Similar

Feature-critic networks for heterogeneous domain generalization
Li, Y., Yang, Y., Zhou, W., and Hospedales, T · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al · 2019
Cited alongside, same era.
Libritts: A corpus derived from librispeech for text-to-speech
Original
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y · 2019
Cited alongside, same era.
wav2vec 2.0: A framework for self-supervised learning of speech representations
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M · 2020
Cited alongside, same era.
Glow-tts: A generative flow for text-to-speech via monotonic alignment search
Kim, J., Kim, S., Kong, J., and Yoon, S · 2020
Cited alongside, same era.
Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis
Original
Kong, J., Kim, J., and Bae, J · 2020
Cited alongside, same era.
Speaker conditional wavernn: Towards universal neural vocoder for unseen speaker and recording conditions
Original
Paul, D., Pantazis, Y., and Stylianou, Y · 2020
Cited alongside, same era.
Fastspeech 2: Fast and high-quality end-to-end text to speech
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y · 2020
Cited alongside, same era.
Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens
Valle, R., Li, J., Prenger, R., and Catanzaro, B · 2020
Cited alongside, same era.
Domain generalization with mixstyle
Zhou, K., Yang, Y., Qiao, Y., and Xiang, T · 2020
Cited alongside, same era.
Sc-glowtts: an efficient zero-shot multi-speaker text-to-speech model
Original
Casanova, E., Shulby, C., Gölge, E., Müller, N. M., de Oliveira, F. S., Junior, A. C., Soares, A. d. S., Aluisio, S. M., and Ponti, M. A · 2021
Cited alongside, same era.
Singgan: Generative adversarial network for high-fidelity singing voice generation
Original
Chen, F., Huang, R., Cui, C., Ren, Y., Liu, J., and Zhao, Z · 2021
Cited alongside, same era.

Then

Emovie: A mandarin emotion speech dataset with a simple emotional text-to-speech model
Original
Cui, C., Ren, Y., Liu, J., Chen, F., Huang, R., Lei, M., and Zhao, Z · 2021
Later among the works it cites.
Fedspeech: Federated text-to-speech with continual learning
Original
Jiang, Z., Ren, Y., Lei, M., and Zhao, Z · 2021
Later among the works it cites.
Style normalization and restitution for domain generalization and adaptation
Jin, X., Lan, C., Zeng, W., and Chen, Z · 2021
Later among the works it cites.
Bilateral denoising diffusion models
Original
Lam, M. W., Wang, J., Huang, R., Su, D., and Yu, D · 2021
Later among the works it cites.
Styler: Style factor modeling with rapidity and robustness via speech decomposition for expressive and controllable neural text to speech
Original
Lee, K., Park, K., and Kim, D · 2021
Later among the works it cites.
Towards multi-scale style control for expressive speech synthesis
Original
Li, X., Song, C., Li, J., Wu, Z., Jia, J., and Meng, H · 2021
Later among the works it cites.
Meta-stylespeech: Multi-speaker adaptive text-to-speech generation
Min, D., Lee, D. B., Yang, E., and Hwang, S. J · 2021
Later among the works it cites.
Latent video transformer
Rakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., and Burnaev, E · 2021
Later among the works it cites.
Portaspeech: Portable and high-quality generative text-to-speech
Ren, Y., Liu, J., and Zhao, Z · 2021
Later among the works it cites.
Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset
Zhou, K., Sisman, B., Liu, R., and Li, H · 2021
Later among the works it cites.
Dict-tts: Learning to pronounce with prior dictionary knowledge for text-to-speech
Original
Jiang, Z., Zhe, S., Zhao, Z., Yang, Q., Ren, Y., Liu, J., and Ye, Z · 2022
Closest in time.
Diffsinger: Singing voice synthesis via shallow diffusion mechanism
Liu, J., Li, C., Ren, Y., Chen, F., and Zhao, Z · 2022
Closest in time.
Hifidenoise: High-fidelity denoising text to speech with adversarial networks
Zhang, L., Ren, Y., Deng, L., and Zhao, Z · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…