DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modeling

Understand

While neural-based text to speech (TTS) models can synthesize natural and intelligible voice, they usually require high-quality speech data, which is costly to collect.

• In many scenarios, only noisy speech of a target speaker is available, which presents challenges for TTS model training for this speaker.
• Previous works usually address the challenge using two methods: 1) training the TTS model using the speech denoised with an enhancement model; 2) taking a single noise embedding as input when training with noisy speech.
• However, they usually cannot handle speech with real-world complicated noise such as those with high variations along time.