2020

Unsupervised Text Style Transfer with Padded Masked Language Models

Malmi, Eric, Severyn, Aliaksei, Rothe, Sascha

Understand

We propose Masker, an unsupervised text-editing method for style transfer.

To tackle cases when no parallel source-target pairs are available, we train masked language models (MLMs) for both the source and the target domain.
Then we find the text spans where the two models disagree the most in terms of likelihood.
This allows us to identify the source tokens to delete to transform the source text to match the style of the target domain.

Built on

A generalized framework of sequence generation with application to undirected sequence models
Original
Elman Mansimov, Alex Wang, Sean Welleck, and Kyunghyun Cho. 2019 · 1905
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910
Earlier work this paper cites.
Blank language models
Original
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020 · 2002
Earlier work this paper cites.
Felix: Flexible text editing through tagging and insertion
Original
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020 · 2003
Earlier work this paper cites.
R 3 R^{3} : Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge
Original
Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020 · 2004
Earlier work this paper cites.
Conditional rap lyrics generation with denoising autoencoders
Original
Nikola I Nikolov, Eric Malmi, Curtis G Northcutt, and Loreto Parisi. 2020 · 2004
Earlier work this paper cites.
Unsupervised lexical simplification for non-native speakers
Gustavo H Paetzold and Lucia Specia. 2016 · 2016
Earlier work this paper cites.
Style transfer from non-parallel text by cross-alignment
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017 · 2017
Earlier work this paper cites.
Delete, retrieve, generate: a simple approach to sentiment and style transfer
Juncen Li, Robin Jia, He He, and Percy Liang. 2018 · 2018
Earlier work this paper cites.

Similar

Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer
Sudha Rao and Joel Tetreault. 2018 · 2018
Cited alongside, same era.
Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018 · 2018
Cited alongside, same era.
Unsupervised text style transfer using language models as discriminators
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018 · 2018
Cited alongside, same era.
Parallel iterative edit models for local sequence transduction
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Editnts: An neural programmer-interpreter model for sentence simplification through explicit editing
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019 · 2019
Cited alongside, same era.
Discofuse: A large-scale dataset for discourse-based sentence fusion
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. 2019 · 2019
Cited alongside, same era.
Neural text style transfer via denoising and reranking
Joseph Lee, Ziang Xie, Cindy Wang, Max Drach, Dan Jurafsky, and Andrew Y Ng. 2019 · 2019
Cited alongside, same era.
A dual reinforcement learning framework for unsupervised text style transfer
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019 · 2019
Cited alongside, same era.

Then

Encode, tag, realize: High-precision text editing
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019 · 2019
Later among the works it cites.
A modular architecture for unsupervised sarcasm generation
Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019 · 2019
Later among the works it cites.
Bert has a mouth, and it must speak: Bert as a markov random field language model
Alex Wang and Kyunghyun Cho. 2019 · 2019
Later among the works it cites.
Mask and infill: applying masked language model to sentiment transfer
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019 · 2019
Later among the works it cites.
SpanBERT: Improving pre-training by representing and predicting spans
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020 · 2020
Closest in time.
Leveraging pre-trained checkpoints for sequence generation tasks
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020 · 2020
Closest in time.
Masked language model scoring
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020 · 2020
Closest in time.
Unsupervised neural text simplification
Sai Surya, Abhijit Mishra, Anirban Laha, Parag Jain, and Karthik Sankaranarayanan. 2019 · 2068
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…