Fetching the paper…

Reading the bibliography…

2024

Autoregressive Diffusion Transformer for Text-to-Speech Synthesis

Liu, Zhijun, Wang, Shuai, Inoue, Sho et al.

Original
PDF
Post on X

Understand

Audio language models have recently emerged as a promising approach for various audio generation tasks, relying on audio tokenizers to encode waveforms into sequences of discrete symbols.

Audio tokenization often poses a necessary compromise between code bitrate and reconstruction accuracy.
When dealing with low-bitrate audio codes, language models are constrained to process only a subset of the information embedded in the audio, which in turn restricts their generative capabilities.
To circumvent these issues, we propose encoding audio as vector sequences in continuous space $\mathbb R^d$ and autoregressively generating these sequences using a decoder-only diffusion transformer (ARDiT).

Autoregressive Diffusion Transformer for Text-to-Speech Synthesis · Around

Built on

G. E. Hinton and D. van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in Annual Conference Computational Learning Theory , 1993
1993
Earlier work this paper cites.
C. M. Bishop, “Mixture density networks,” 1994
1994
Earlier work this paper cites.
P. Vincent, “A connection between score matching and denoising autoencoders,” Neural Computation , 2011
2011
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
S. Kraft and U. Zölzer, “BeaqleJS: HTML5 and JavaScript based framework for the subjective evaluation of audio quality,” in Linux Audio Conference , 2014. [Online]. Available: https://github.com/HSU-ANT/beaqlejs
2014
Earlier work this paper cites.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML , 2015
2015
Earlier work this paper cites.
A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” in ICML , 2016
2016
Earlier work this paper cites.
E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” arXiv preprint arXiv:1611.01144 , 2016
Original
2016
Earlier work this paper cites.
X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel, “Variational lossy autoencoder,” arXiv preprint arXiv:1611.02731 , 2016
Original
2016
Earlier work this paper cites.
A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” NeurIPS , 2017
2017
Earlier work this paper cites.
X. Wang, S. Takaki, and J. Yamagishi, “An autoregressive recurrent mixture density network for parametric speech synthesis,” ICASSP , 2017
2017
Earlier work this paper cites.
T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma, “PixelCNN++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications,” arXiv preprint arXiv:1701.05517 , 2017
Original
2017
Earlier work this paper cites.
Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al. , “Tacotron: Towards end-to-end speech synthesis,” arXiv preprint arXiv:1703.10135 , 2017
Original
2017
Earlier work this paper cites.
Z. Jin, G. J. Mysore, S. Diverdi, J. Lu, and A. Finkelstein, “Voco: Text-based insertion and replacement in audio narration,” TOG , 2017
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS , 2017
2017
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR , 2017
2017
Earlier work this paper cites.
——, “The perception-distortion tradeoff,” in CVPR , 2018
2018
Earlier work this paper cites.
D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” NeurIPS , 2018
2018
Earlier work this paper cites.
A. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. Driessche, E. Lockhart, L. Cobo, F. Stimberg et al. , “Parallel Wavenet: Fast high-fidelity speech synthesis,” in ICML , 2018
2018
Earlier work this paper cites.
Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y. Wu et al. , “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” NeurIPS , 2018
2018
Earlier work this paper cites.
Y. Blau and T. Michaeli, “Rethinking lossy compression: The rate-distortion-perception tradeoff,” in ICML , 2019
2019
Earlier work this paper cites.
A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in ICLR , 2019
2019
Earlier work this paper cites.
B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, “On variational bounds of mutual information,” in ICML , 2019
2019
Earlier work this paper cites.
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” NeurIPS , 2020
2020
Earlier work this paper cites.
R. Valle, K. Shih, R. Prenger, and B. Catanzaro, “Flowtron: An autoregressive flow-based generative network for text-to-speech synthesis,” arXiv preprint arXiv:2005.05957 , 2020
Original
2020
Earlier work this paper cites.
W. Ping, K. Peng, K. Zhao, and Z. Song, “WaveFlow: A compact flow-based model for raw audio,” in ICML , 2020
2020
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM , 2020
2020
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS , 2020
2020
Earlier work this paper cites.
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456 , 2020
Original
2020
Earlier work this paper cites.
J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 , 2020
Original
2020
Earlier work this paper cites.
K. Peng, W. Ping, Z. Song, and K. Zhao, “Non-autoregressive neural text-to-speech,” in ICML , 2020
2020
Earlier work this paper cites.
L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” 2020
2020
Earlier work this paper cites.
J. Thienpondt, B. Desplanques, and K. Demuynck, “The IDLAB VoxSRC-20 submission: Large margin fine-tuning and quality-aware score calibration in dnn based speaker verification,” arXiv preprint arXiv:2010.11255 , 2020
Original
2020
Earlier work this paper cites.
K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed et al. , “On generative spoken language modeling from raw audio,” TACL , 2021
2021
Earlier work this paper cites.
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” TASLP , 2021
2021
Earlier work this paper cites.
A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” arXiv preprint arXiv:2104.00355 , 2021
Original
2021
Earlier work this paper cites.
P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in CVPR , 2021
2021
Earlier work this paper cites.
G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference,” JMLR , 2021
2021
Earlier work this paper cites.
R. J. Weiss, R. Skerry-Ryan, E. Battenberg, S. Mariooryad, and D. P. Kingma, “Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis,” in ICASSP , 2021
2021
Earlier work this paper cites.
C. Du and K. Yu, “Phone-level prosody modelling with gmm-based mdn for diverse and controllable speech synthesis,” TASLP , 2021
2021
Earlier work this paper cites.
M. Morrison, R. Kumar, K. Kumar, P. Seetharaman, A. Courville, and Y. Bengio, “Chunked autoregressive gan for conditional waveform synthesis,” arXiv preprint arXiv:2110.10139 , 2021
Original
2021
Earlier work this paper cites.
D. Tan, L. Deng, Y. T. Yeung, X. Jiang, X. Chen, and T. Lee, “EditSpeech: A text based speech editing system using partial inference and bidirectional fusion,” in ASRU , 2021
2021
Earlier work this paper cites.

Similar

J. Tae, H. Kim, and T. Kim, “EdiTTS: Score-based editing for controllable text-to-speech,” arXiv preprint arXiv:2110.02584 , 2021
Original
2021
Cited alongside, same era.
E. Luhman and T. Luhman, “Knowledge distillation in iterative generative models for improved sampling speed,” arXiv preprint arXiv:2101.02388 , 2021
Original
2021
Cited alongside, same era.
T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” in ICLR , 2021
2021
Cited alongside, same era.
S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” arXiv preprint arXiv:2110.13900 , 2021
Original
2021
Cited alongside, same era.
F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, “AudioGen: Textually guided audio generation,” arXiv preprint arXiv:2209.15352 , 2022

Then

Y. Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang, Y. He, J. Yu, P. Yang et al. , “LAVIE: High-quality video generation with cascaded latent diffusion models,” arXiv preprint arXiv:2309.15103 , 2023
Original
2023
Later among the works it cites.
H. Lu, G. Yang, N. Fei, Y. Huo, Z. Lu, P. Luo, and M. Ding, “VDT: General-purpose video diffusion transformers via mask modeling,” in ICLR , 2023
2023
Later among the works it cites.
T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park, “One-step diffusion with distribution matching distillation,” arXiv preprint arXiv:2311.18828 , 2023
Original
2023
Later among the works it cites.
Z. Zhang, L. Zhou, C. Wang, S. Chen, Y. Wu, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al. , “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,” arXiv preprint arXiv:2303.03926 , 2023
Original
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2022

Cited alongside, same era.

H. Inaguma, S. Popuri, I. Kulikov, P.-J. Chen, C. Wang, Y.-A. Chung, Y. Tang, A. Lee, S. Watanabe, and J. Pino, “Unity: Two-pass direct speech-to-speech translation with discrete units,” arXiv preprint arXiv:2212.08055 , 2022

2022

Cited alongside, same era.

M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen, “Efficient training of language models to fill in the middle,” arXiv preprint arXiv:2207.14255 , 2022

2022

Cited alongside, same era.

J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” in NeurIPS , 2022

2022

Cited alongside, same era.

T. Wang, J. Yi, L. Deng, R. Fu, J. Tao, and Z. Wen, “Context-aware mask prediction network for end-to-end text-based speech editing,” in ICASSP , 2022

2022

Cited alongside, same era.

H. Bai, R. Zheng, J. Chen, M. Ma, X. Li, and L. Huang, “A 3 T: Alignment-aware acoustic and text pretraining for speech synthesis and editing,” in ICML , 2022

2022

Cited alongside, same era.

Z. Borsos, M. Sharifi, and M. Tagliasacchi, “SpeechPainter: Text-conditioned speech inpainting,” in Interspeech , 2022

2022

Cited alongside, same era.

D. Yin, C. Tang, Y. Liu, X. Wang, Z. Zhao, Y. Zhao, Z. Xiong, S. Zhao, and C. Luo, “RetrieverTTS: Modeling decomposed factors for text-based speech insertion,” arXiv preprint arXiv:2206.13865 , 2022

2022

Cited alongside, same era.

Z. Jiang, Y. Ren, Z. Ye, J. Liu, C. Zhang, Q. Yang, S. Ji, R. Huang, C. Wang, X. Yin et al. , “Mega-TTS: Zero-shot text-to-speech at scale with intrinsic inductive bias,” arXiv preprint arXiv:2306.03509 , 2023

2023

Later among the works it cites.

Z. Jiang, J. Liu, Y. Ren, J. He, C. Zhang, Z. Ye, P. Wei, C. Wang, X. Yin, Z. Ma et al. , “Mega-TTS 2: Zero-shot text-to-speech with arbitrary length speech prompts,” arXiv preprint arXiv:2307.07218 , 2023

2023

Later among the works it cites.

K. Shen, Z. Ju, X. Tan, E. Liu, Y. Leng, L. He, T. Qin, J. Bian et al. , “NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” in ICLR , 2023

2023

Later among the works it cites.

Z. Jiang, Q. Yang, J. Zuo, Z. Ye, R. Huang, Y. Ren, and Z. Zhao, “FluentSpeech: Stutter-oriented automatic speech editing with context-aware diffusion models,” in ACL , 2023

2023

Later among the works it cites.

Z. Liu, Y. Guo, and K. Yu, “DiffVoice: Text-to-speech with latent diffusion,” in ICASSP , 2023

2023

Later among the works it cites.

Y. Gao, N. Morioka, Y. Zhang, and N. Chen, “E3 TTS: Easy end-to-end diffusion-based text to speech,” arXiv preprint arXiv:2311.00945 , 2023

2023

Later among the works it cites.

Y. Song, P. Dhariwal, M. Chen, and I. Sutskever, “Consistency models,” in ICML , 2023

2023

Later among the works it cites.

D. Berthelot, A. Autef, J. Lin, D. A. Yap, S. Zhai, S. Hu, D. Zheng, W. Talbott, and E. Gu, “TRACT: Denoising diffusion models with transitive closure time-distillation,” arXiv preprint arXiv:2303.04248 , 2023

2023

Later among the works it cites.

J. Gu, S. Zhai, Y. Zhang, L. Liu, and J. Susskind, “BOOT: Data-free distillation of denoising diffusion models with bootstrapping,” arXiv preprint arXiv:2306.05544 , 2023

2023

Later among the works it cites.

Z. Ye, W. Xue, X. Tan, J. Chen, Q. Liu, and Y. Guo, “CoMoSpeech: One-step speech and singing voice synthesis via consistency model,” in ACM Multimedia , 2023

2023

Later among the works it cites.

M. S. Albergo, N. M. Boffi, and E. Vanden-Eijnden, “Stochastic interpolants: A unifying framework for flows and diffusions,” arXiv preprint arXiv:2303.08797 , 2023

2023

Later among the works it cites.

W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in ICCV , 2023

2023

Later among the works it cites.

S.-H. Lee, H.-Y. Choi, S.-B. Kim, and S.-W. Lee, “HierSpeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis,” arXiv preprint arXiv:2311.12454 , 2023

2023

Later among the works it cites.

H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y. Deng, and Y. Qian, “WeSpeaker: A research and production oriented speaker embedding learning toolkit,” in ICASSP , 2023

2023

Later among the works it cites.

J. Lovelace, S. Ray, K. Kim, K. Q. Weinberger, and F. Wu, “Simple-TTS: End-to-end text-to-speech synthesis with latent diffusion,” 2023. [Online]. Available: https://openreview.net/forum?id=m4mwbPjOwb

2023

Later among the works it cites.

H. Wu, X. Chen, Y.-C. Lin, K.-W. Chang, H.-L. Chung, A. H. Liu, and H. yi Lee, “Towards audio language modeling - an overview,” arXiv preprint arXiv:2402.13236 , 2024

2024

Closest in time.

M. Hassid, T. Remez, T. A. Nguyen, I. Gat, A. Conneau, F. Kreuk, J. Copet, A. Defossez, G. Synnaeve, E. Dupoux et al. , “Textually pretrained speech language models,” NeurIPS , 2024

2024

Closest in time.

R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” NeurIPS , 2024

2024

Closest in time.

Z. Du, S. Zhang, K. Hu, and S. Zheng, “FunCodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,” in ICASSP , 2024

2024

Closest in time.

P. Peng, P.-Y. Huang, A. Mohamed, and D. Harwath, “VoiceCraft: Zero-shot speech editing and text-to-speech in the wild,” arXiv preprint arXiv:2403.16973 , 2024

2024

Closest in time.

J. Kim, K. Lee, S. Chung, and J. Cho, “CLaM-TTS: Improving neural codec language model for zero-shot text-to-speech,” arXiv preprint arXiv:2404.02781 , 2024

2024

Closest in time.

Z. Wang, L. Min, and G. Xia, “Whole-song hierarchical generation of symbolic music using cascaded diffusion models,” arXiv preprint arXiv:2405.09901 , 2024

2024

Closest in time.

S. Xu, G. Chen, Y.-X. Guo, J. Yang, C. Li, Z. Zang, Y. Zhang, X. Tong, and B. Guo, “VASA-1: Lifelike audio-driven talking faces generated in real time,” arXiv preprint arXiv:2404.10667 , 2024

2024

Closest in time.

W. Luo, T. Hu, S. Zhang, J. Sun, Z. Li, and Z. Zhang, “Diff-Instruct: A universal approach for transferring knowledge from pre-trained diffusion models,” NeurIPS , 2024

2024

Closest in time.

J.-Y. Franceschi, M. Gartrell, L. Dos Santos, T. Issenhuth, E. de Bézenac, M. Chen, and A. Rakotomamonjy, “Unifying GANs and score-based diffusion as generative particle models,” NeurIPS , 2024

2024

Closest in time.

Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu, “ProlificDreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” NeurIPS , 2024

2024

Closest in time.

M. Zhou, H. Zheng, Z. Wang, M. Yin, and H. Huang, “Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation,” arXiv preprint arXiv:2404.04057 , 2024

2024

Closest in time.

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomputing , 2024

2024

Closest in time.

D. Ruhe, J. Heek, T. Salimans, and E. Hoogeboom, “Rolling diffusion models,” arXiv preprint arXiv:2402.09470 , 2024

2024

Closest in time.

C. Du, Y. Guo, F. Shen, Z. Liu, Z. Liang, X. Chen, S. Wang, H. Zhang, and K. Yu, “UniCATS: A unified context-aware text-to-speech framework with contextual vq-diffusion and vocoding,” in AAAI , 2024

2024

Closest in time.

Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang et al. , “NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” arXiv preprint arXiv:2403.03100 , 2024

2024

Closest in time.

M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar et al. , “VoiceBox: Text-guided multilingual universal speech generation at scale,” NeurIPS , 2024

2024

Closest in time.

S. Dieleman, “The paradox of diffusion distillation,” 2024. [Online]. Available: https://sander.ai/2024/02/28/paradox.html

2024

Closest in time.

J. Zheng, M. Hu, Z. Fan, C. Wang, C. Ding, D. Tao, and T.-J. Cham, “Trajectory consistency distillation,” arXiv preprint arXiv:2402.19159 , 2024

2024

Closest in time.

W. Guan, Q. Su, H. Zhou, S. Miao, X. Xie, L. Li, and Q. Hong, “Reflow-TTS: A rectified flow model for high-fidelity text-to-speech,” in ICASSP , 2024

2024

Closest in time.

N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden, and S. Xie, “SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers,” arXiv preprint arXiv:2401.08740 , 2024

2024

Closest in time.

Y. A. Li, C. Han, V. Raghavan, G. Mischler, and N. Mesgarani, “StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,” NeurIPS , 2024

2024

Closest in time.

G. Cámbara, P. L. Tobing, M. Babianski, R. Vipperla, D. W. R. Shmelkin, G. Coccia, O. Angelini, A. Joly, M. Lajszczak, and V. Pollet, “Mapache: Masked parallel transformer for advanced speech editing and synthesis,” in ICASSP , 2024

2024

Closest in time.