2020

Low-resource expressive text-to-speech using data augmentation

Huybrechts, Goeric, Merritt, Thomas, Comini, Giulia et al.

Understand

While recent neural text-to-speech (TTS) systems perform remarkably well, they typically require a substantial amount of recordings from the target speaker reading in the desired speaking style.

In this work, we present a novel 3-step methodology to circumvent the costly operation of recording large amounts of target data in order to build expressive style voices with as little as 15 minutes of such recordings.
First, we augment data via voice conversion by leveraging recordings in the desired speaking style from other speakers.
Next, we use that synthetic data on top of the available recordings to train a TTS model.

Built on

“1534-1, method for the subjective assessment of intermediate quality levels of coding systems (mushra),”,”
RB ITU-R, · 2003
Earlier work this paper cites.
“Auto-encoding variational bayes,”
Original
Diederik P Kingma et al., · 2013
Earlier work this paper cites.
“Audio augmentation for speech recognition,”
Tom Ko et al., · 2015
Earlier work this paper cites.
“Wavenet: A generative model for raw audio,”
Original
Aaron van den Oord et al., · 2016
Earlier work this paper cites.
“Tacotron: Towards end-to-end speech synthesis,”
Original
Yuxuan Wang et al., · 2017
Earlier work this paper cites.
“Char2wav: End-to-end speech synthesis,”
Jose Sotelo et al., · 2017
Earlier work this paper cites.
“Deep voice 2: Multi-speaker neural text-to-speech,”
Andrew Gibiansky et al., · 2017
Earlier work this paper cites.
“The effectiveness of data augmentation in image classification using deep learning,”
Original
Luis Perez et al., · 2017
Earlier work this paper cites.
“A study on data augmentation of reverberant speech for robust speech recognition,”
Tom Ko et al., · 2017
Earlier work this paper cites.
“An overview of voice conversion systems,”
Seyed Hamidreza Mohammadi et al., · 2017
Earlier work this paper cites.

Similar

“Voice conversion from unaligned corpora using variational autoencoding wasserstein generative adversarial networks,”
Original
Chin-Cheng Hsu et al., · 2017
Cited alongside, same era.
“Towards end-to-end prosody transfer for expressive speech synthesis with tacotron,”
Original
RJ Skerry-Ryan et al., · 2018
Cited alongside, same era.
“Efficient neural audio synthesis,”
Original
Nal Kalchbrenner et al., · 2018
Cited alongside, same era.
“Parallel wavenet: Fast high-fidelity speech synthesis,”
Aaron Oord et al., · 2018
Cited alongside, same era.
“Transfer learning from speaker verification to multispeaker text-to-speech synthesis,”
Ye Jia et al., · 2018
Cited alongside, same era.
“Data augmentation for improving deep learning in image classification problem,”
Agnieszka Mikołajczyk et al., · 2018
Cited alongside, same era.
“The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods,”
Original
Jaime Lorenzo-Trueba et al., · 2018
Cited alongside, same era.
“Semi-supervised training for improving data efficiency in end-to-end speech synthesis,”
Yu-An Chung et al., · 2019
Cited alongside, same era.
“Exploring transfer learning for low resource emotional tts,”
Noé Tits et al., · 2019
Cited alongside, same era.
“Effect of data reduction on sequence-to-sequence neural tts,”
Javier Latorre et al., · 2019
Cited alongside, same era.

Then

“A survey on image data augmentation for deep learning,”
Connor Shorten et al., · 2019
Later among the works it cites.
“Specaugment: A simple data augmentation method for automatic speech recognition,”
Original
Daniel S Park et al., · 2019
Later among the works it cites.
“Cyclegan-vc2: Improved cyclegan-based non-parallel voice conversion,”
Takuhiro Kaneko et al., · 2019
Later among the works it cites.
“Learning latent representations for style control and transfer in end-to-end speech synthesis,”
Ya-Jie Zhang et al., · 2019
Later among the works it cites.
“Dynamic prosody generation for speech synthesis using linguistics-driven acoustic embedding selection,”
Original
Shubhi Tyagi et al., · 2019
Later among the works it cites.
“Unsupervised learning for sequence-to-sequence text-to-speech for low-resource languages,”
Original
Haitong Zhang et al., · 2020
Closest in time.
“Lrspeech: Extremely low-resource speech synthesis and recognition,”
Jin Xu et al., · 2020
Closest in time.
“Copycat: Many-to-many fine-grained prosody transfer for neural text-to-speech,”
Original
Sri Karlapati et al., · 2020
Closest in time.
“End-to-end text-to-speech for low-resource languages by cross-lingual transfer learning.,”
Yuan-Jui Chen et al., · 2079
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…