2022

SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers

Yuan, Hongyi, Yuan, Zheng, Tan, Chuanqi et al.

Understand

Diffusion model, a new generative modelling paradigm, has achieved great success in image, audio, and video generation.

However, considering the discrete categorical nature of text, it is not trivial to extend continuous diffusion models to natural language, and text diffusion models are less studied.
Sequence-to-sequence text generation is one of the essential natural language processing topics.
In this work, we apply diffusion models to approach sequence-to-sequence text generation, and explore whether the superiority generation performance of diffusion model can transfer to natural language domain.

Built on

Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.-J. Bleu: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, Pennsylvania, USA, 2002; pp 311–318
2002
Earlier work this paper cites.
Koehn, P. Statistical Significance Tests for Machine Translation Evaluation. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Barcelona, Spain, 2004; pp 388–395
2004
Earlier work this paper cites.
Koehn, P.; Hoang, H.; Birch, A.; Callison-Burch, C.; Federico, M.; Bertoldi, N.; Cowan, B.; Shen, W.; Moran, C.; Zens, R.; Dyer, C.; Bojar, O.; Constantin, A.; Herbst, E. Moses: Open Source Toolkit for Statistical Machine Translation. Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions. Prague, Czech Republic, 2007; pp 177–180
Original
2007
Earlier work this paper cites.
Kingma, D. P.; Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013
Original
2013
Earlier work this paper cites.
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. Advances in Neural Information Processing Systems. 2014
2014
Earlier work this paper cites.
Hendrycks, D.; Gimpel, K. Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units. ArXiv 2016
Original
2016
Earlier work this paper cites.
Sennrich, R.; Haddow, B.; Birch, A. Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, 2016; pp 1715–1725
2016
Earlier work this paper cites.
DataCanary,; hilfialkaff,; Jiang, L.; Risdal, M.; Dandekar, N.; tomtung, Quora Question Pairs. 2017; https://kaggle.com/competitions/quora-question-pairs
2017
Earlier work this paper cites.
Dhingra, B.; Mazaitis, K.; Cohen, W. W. Quasar: Datasets for Question Answering by Search and Reading. arXiv preprint arXiv:1707.03904 2017
Original
2017
Earlier work this paper cites.
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. 2017
2017
Earlier work this paper cites.
Zhou, H.; Young, T.; Huang, M.; Zhao, H.; Xu, J.; Zhu, X. Commonsense Knowledge Aware Conversation Generation with Graph Attention. Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018; p 4623–4629
2018
Earlier work this paper cites.
Post, M. A Call for Clarity in Reporting BLEU Scores. Proceedings of the Third Conference on Machine Translation: Research Papers. Brussels, Belgium, 2018; pp 186–191
2018
Earlier work this paper cites.
Deshpande, A.; Aneja, J.; Wang, L.; Schwing, A. G.; Forsyth, D. A. Fast, Diverse and Accurate Image Captioning Guided by Part-Of-Speech. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2018
2018
Earlier work this paper cites.
Lin, Y.; Ji, H.; Liu, Z.; Sun, M. Denoising Distantly Supervised Open-Domain Question Answering. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia, 2018; pp 1736–1745
2018
Earlier work this paper cites.
Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.; Salakhutdinov, R. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy, 2019; pp 2978–2988
2019
Earlier work this paper cites.

Similar

Gu, J.; Wang, C.; Zhao, J. Levenshtein Transformer. Neural Information Processing Systems. 2019
2019
Cited alongside, same era.
Ghazvininejad, M.; Levy, O.; Liu, Y.; Zettlemoyer, L. Mask-Predict: Parallel Decoding of Conditional Masked Language Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China, 2019; pp 6112–6121
2019
Cited alongside, same era.
Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota, 2019; pp 4171–4186
2019
Cited alongside, same era.
Ott, M.; Edunov, S.; Baevski, A.; Fan, A.; Gross, S.; Ng, N.; Grangier, D.; Auli, M. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). Minneapolis, Minnesota, 2019; pp 48–53
2019
Cited alongside, same era.
Ho, J.; Jain, A.; Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 2020
2020
Cited alongside, same era.
Kong, Z.; Ping, W.; Huang, J.; Zhao, K.; Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 2020
Original
2020
Cited alongside, same era.
Liu, Y.; Gu, J.; Goyal, N.; Li, X.; Edunov, S.; Ghazvininejad, M.; Lewis, M.; Zettlemoyer, L. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics 2020
2020
Cited alongside, same era.
Jiang, C.; Maddela, M.; Lan, W.; Zhong, Y.; Xu, W. Neural CRF Model for Sentence Alignment in Text Simplification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020; pp 7943–7960
2020
Cited alongside, same era.
Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; Artzi, Y. BERTScore: Evaluating Text Generation with BERT. International Conference on Learning Representations. 2020
2020
Cited alongside, same era.
Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. 2021; pp 10674–10685
2021
Cited alongside, same era.
Kobyzev, I.; Prince, S. J.; Brubaker, M. A. Normalizing Flows: An Introduction and Review of Current Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 2021
2021
Cited alongside, same era.
Nichol, A. Q.; Dhariwal, P. Improved Denoising Diffusion Probabilistic Models. 2021; https://openreview.net/forum?id=-NEXDKk8gZ
2021
Cited alongside, same era.

Then

Austin, J.; Johnson, D. D.; Ho, J.; Tarlow, D.; van den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems 2021
2021
Later among the works it cites.
Kingma, D. P.; Salimans, T.; Poole, B.; Ho, J. On Density Estimation with Diffusion Models. Advances in Neural Information Processing Systems. 2021
2021
Later among the works it cites.
Song, J.; Meng, C.; Ermon, S. Denoising Diffusion Implicit Models. International Conference on Learning Representations. 2021
2021
Later among the works it cites.
Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; Fleet, D. J. Video Diffusion Models. ArXiv 2022
Original
2022
Closest in time.
Austin, E.; Zaïane, O. R.; Largeron, C. Community Topic: Topic Model Inference by Consecutive Word Community Discovery. Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea, 2022; pp 971–983
2022
Closest in time.
Li, X. L.; Thickstun, J.; Gulrajani, I.; Liang, P.; Hashimoto, T. B. Diffusion-LM Improves Controllable Text Generation. Advances in Neural Information Processing Systems 2022
2022
Closest in time.
Gong, S.; Li, M.; Feng, J.; Wu, Z.; Kong, L. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933 2022
Original
2022
Closest in time.
Chen, T.; Zhang, R.; Hinton, G. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning. 2022; https://arxiv.org/abs/2208.04202
Original
2022
Closest in time.
He, Z.; Sun, T.; Wang, K.; Huang, X.; Qiu, X. DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models. arXiv preprint arXiv:2211.15029 2022
Original
2022
Closest in time.
Yu, P.; Xie, S.; Ma, X.; Jia, B.; Pang, B.; Gao, R.; Zhu, Y.; Zhu, S.-C.; Wu, Y. N. Latent Diffusion Energy-Based Model for Interpretable Text Modeling. 2022; https://arxiv.org/abs/2206.05895
Original
2022
Closest in time.
Strudel, R.; Tallec, C.; Altch’e, F.; Du, Y.; Ganin, Y.; Mensch, A.; Grathwohl, W.; Savinov, N.; Dieleman, S.; Sifre, L.; Leblond, R. Self-conditioned Embedding Diffusion for Text Generation. ArXiv 2022
Original
2022
Closest in time.
Dieleman, S.; Sartran, L.; Roshannai, A.; Savinov, N.; Ganin, Y.; Richemond, P. H.; Doucet, A.; Strudel, R.; Dyer, C.; Durkan, C., et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089 2022
Original
2022
Closest in time.
Chung, H. W. et al. Scaling Instruction-Finetuned Language Models. 2022
2022
Closest in time.
Gao, Z.; Guo, J.; Tan, X.; Zhu, Y.; Zhang, F.; Bian, J.; Xu, L. Difformer: Empowering Diffusion Models on the Embedding Space for Text Generation. 2023
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…