2022

Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

Crothers, Evan, Japkowicz, Nathalie, Viktor, Herna

Understand

Machine generated text is increasingly difficult to distinguish from human authored text.

Powerful open-source models are freely available, and user-friendly tools that democratize access to generative models are proliferating.
ChatGPT, which was released shortly after the first edition of this survey, epitomizes these trends.
The great potential of state-of-the-art natural language generation (NLG) systems is tempered by the multitude of avenues for abuse.

Built on

A survey on Adversarial Attacks and Defenses in Text
Original
Wenqi Wang, Benxiao Tang, et al · 1902
Earlier work this paper cites.
The Curious Case of Neural Text Degeneration
Original
Ari Holtzman, Jan Buys, et al · 1904
Earlier work this paper cites.
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Original
Anton Bakhtin, Sam Gross, et al · 1906
Earlier work this paper cites.
Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment
Original
Di Jin, Zhijing Jin, et al · 1907
Earlier work this paper cites.
Human behavior and the principle of least effort
George Kingsley Zipf. 1949 · 1949
Earlier work this paper cites.
Computing Machinery and Intelligence
A. M. Turing. 1950 · 1950
Earlier work this paper cites.
The automatic creation of literature abstracts
Hans Peter Luhn. 1958 · 1958
Earlier work this paper cites.
Statistical inference for probabilistic functions of finite state Markov chains
Leonard E Baum and Ted Petrie. 1966 · 1966
Earlier work this paper cites.
ELIZA—a Computer Program for the Study of Natural Language Communication between Man and Machine
Joseph Weizenbaum. 1966 · 1966
Earlier work this paper cites.
Natural Language and Universal Grammar: Volume 1: Essays in Linguistic Theory . Vol. 1
John Lyons. 1991 · 1991
Earlier work this paper cites.
Generation that exploits corpus-based statistical knowledge. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics
Irene Langkilde and Kevin Knight. 1998 · 1998
Earlier work this paper cites.
The threats to our products
Loren Kohnfelder and Praerit Garg. 1999 · 1999
Earlier work this paper cites.
Learning from imbalanced data sets: a comparison of various strategies. In AAAI workshop on learning from imbalanced data sets , Vol. 68. AAAI Press Menlo Park, CA, 10–15
Nathalie Japkowicz et al · 2000
Earlier work this paper cites.
The Pushshift Reddit Dataset
Original
Jason Baumgartner, Savvas Zannettou, et al · 2001
Earlier work this paper cites.
Building Applied Natural Language Generation Systems
Ehud Reiter and Robert Dale. 2002 · 2002
Earlier work this paper cites.
CAPTCHA: Using hard AI problems for security. In International conference on the theory and applications of cryptographic techniques . Springer, 294–311
Luis von Ahn, Manuel Blum, et al · 2003
Earlier work this paper cites.
Statistical Acquisition of Content Selection Rules for Natural Language Generation. In Proc. 2003 Conference on Empirical Methods in Natural Language Processing . 121–128
Pablo Ariel Duboue and Kathleen R. McKeown. 2003 · 2003
Earlier work this paper cites.
SCIgen - An Automatic CS Paper Generator
Jeffrey Hargrave. 2005 · 2005
Earlier work this paper cites.
Adversarial Attacks and Defense on Texts: A Survey
Original
Aminul Huq and Mst. Tasnim Pervin. 2020 · 2005
Earlier work this paper cites.
A tutorial on energy-based learning
Yann LeCun, Sumit Chopra, et al · 2006
Earlier work this paper cites.
A comparison of machine learning techniques for phishing detection. In Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit . 60–69
Saeed Abu-Nimeh, Dario Nappa, et al · 2007
Earlier work this paper cites.
Measuring Conference Quality by Mining Program Committee Characteristics. In Proc. 7th JCDL (JCDL ’07) . ACM, New York, NY, USA, 225–234
Ziming Zhuang, Ergin Elmacioglu, et al · 2007
Earlier work this paper cites.
Learning lexical alignment policies for generating referring expressions for spoken dialogue systems. In Proc. 12th ENLG . 74–81
Srinivasan Janarthanam and Oliver Lemon. 2009 · 2009
Earlier work this paper cites.
The Radicalization Risks of GPT-3 and Advanced Neural Language Models
Original
Kris McGuffie and Alex Newhouse. 2020 · 2009
Earlier work this paper cites.
Text Generation
Li Zhang and Jian-Tao Sun. 2009 · 2009
Earlier work this paper cites.
Hierarchical Reinforcement Learning for Adaptive Text Generation. In Proc. 6th INLG . ACL
Nina Dethlefs and Heriberto Cuayáhuitl. 2010 · 2010
Earlier work this paper cites.
Algorithmic Detection of Computer Generated Text
Original
Allen Lavoie and Mukkai S. Krishnamoorthy. 2010 · 2010
Earlier work this paper cites.
Generation of formal and informal sentences. In Proc. 13th European Workshop on Natural Language Generation . 187–193
Fadi Abu Sheikha and Diana Inkpen. 2011 · 2011
Earlier work this paper cites.
One-Class versus Binary Classification: Which and When?. In 2012 11th International Conference on Machine Learning and Applications , Vol. 2. 102–106
Colin Bellinger, Shiven Sharma, et al · 2012
Earlier work this paper cites.
What makes online content viral?
Jonah Berger and Katherine L Milkman. 2012 · 2012
Earlier work this paper cites.
Extracting Training Data from Large Language Models
Original
Nicholas Carlini, Florian Tramèr, et al · 2012
Earlier work this paper cites.
Stat. lang. models based on neural networks
Tomáš Mikolov et al · 2012
Earlier work this paper cites.
YouTube2Text: Recognizing and Describing Arbitrary Activities Using Semantic Hierarchies and Zero-Shot Recognition. In 2013 IEEE ICCV . 2712–2719
Sergio Guadarrama, Niveda Krishnamoorthy, et al · 2013
Earlier work this paper cites.
Recurrent continuous translation models. In Proc. 2013 conference on empirical methods in natural language processing . 1700–1709
Nal Kalchbrenner and Phil Blunsom. 2013 · 2013
Earlier work this paper cites.
A statistical nlg framework for aggregated planning and realization. In Proc. 51st Annual Meeting of the ACL (Volume 1: Long Papers) . 1406–1415
Ravikumar Kondadadi, Blake Howald, et al · 2013
Earlier work this paper cites.
Duplicate and fake publications in the scientific literature: how many SCIgen papers in computer science?
Cyril Labbé and Dominique Labbé. 2013 · 2013
Earlier work this paper cites.
How Americans get their news
American Press Institute. 2014 · 2014
Earlier work this paper cites.
Threat modeling: Designing for security
Adam Shostack. 2014 · 2014
Earlier work this paper cites.
Bidirectional Recurrent Neural Networks as Generative Models. In NIPS
Mathias Berglund, Tapani Raiko, et al · 2015
Earlier work this paper cites.
Attention-based models for speech recognition
Jan K Chorowski, Dzmitry Bahdanau, et al · 2015
Earlier work this paper cites.
Risk Centric Threat Modeling: process for attack simulation and threat analysis
Tony UcedaVelez and Marco M Morana. 2015 · 2015
Earlier work this paper cites.
Computer-generated text detection using machine learning: A systematic review. In International Conference on Applications of Natural Language to Information Systems . Springer, 421–426
Daria Beresneva. 2016 · 2016
Earlier work this paper cites.
Semantic Cyberthreat Modelling.. In STIDS . 74–78
Siri Bromander, Audun Jøsang, et al · 2016
Earlier work this paper cites.
Social Clicks: What and Who Gets Read on Twitter?. In ACM SIGMETRICS / IFIP Performance 2016 . Antibes Juan-les-Pins, France
Maksym Gabielkov, Arthi Ramachandran, et al · 2016
Earlier work this paper cites.
Deep Reinforcement Learning for Dialogue Generation. In EMNLP
Jiwei Li, Will Monroe, et al · 2016
Earlier work this paper cites.
Scaling and Effectiveness of Email Masquerade Attacks: Exploiting Natural Language Generation. In Proc. 2017 ACM ASIA CCS ’17 . ACM, New York, NY, USA, 469–482
Shahryar Baki, Rakesh Verma, et al · 2017
Earlier work this paper cites.
Mitigating poisoning attacks on machine learning models: A data provenance based approach. In Proc. 10th ACM Workshop on Artificial Intelligence and Security . 103–110
Nathalie Baracaldo, Bryant Chen, et al · 2017
Earlier work this paper cites.
Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video. In Proc. 2017 Conf. EMNLP . ACL, Copenhagen, Denmark, 1092–1102
Haoran Li, Junnan Zhu, et al · 2017
Earlier work this paper cites.
Adversarial Ranking for Language Generation. In NIPS
Kevin Lin, Dianqi Li, et al · 2017
Earlier work this paper cites.
A unified approach to interpreting model predictions
Scott M Lundberg and Su-In Lee. 2017 · 2017
Earlier work this paper cites.
Regularizing and optim. LSTM lang. models
Original
Stephen Merity, Nitish Shirish Keskar, et al · 2017
Earlier work this paper cites.
Identifying computer-generated text using statistical analysis. In 2017 APSIPA ASC . 1504–1511
Hoang-Quoc Nguyen-Son, Ngoc-Dung T. Tieu, et al · 2017
Earlier work this paper cites.
Recent advances in natural language generation: A survey and classification of the empirical literature
Rivindu Perera and Parma Nand. 2017 · 2017
Earlier work this paper cites.
Do social bots (still) act different to humans?–Comparing metrics of social bots with those of humans. In International conference on social computing and social media . Springer, 379–395
Stefan Stieglitz, Florian Brachten, et al · 2017
Earlier work this paper cites.
Attention is All you Need
Original
Ashish Vaswani, Noam M. Shazeer, et al · 2017
Earlier work this paper cites.
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. In AAAI
Lantao Yu, Weinan Zhang, et al · 2017
Earlier work this paper cites.
Adversarial attacks and defences: A survey
Original
Anirban Chakraborty, Manaar Alam, et al · 2018
Earlier work this paper cites.
A survey of phishing attacks: Their types, vectors and technical approaches
Kang Leng Chiew, Kelvin Sheng Chek Yong, et al · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers
Original
Jacob Devlin, Ming-Wei Chang, et al · 2018
Earlier work this paper cites.
Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation
European Commission. 2018 · 2018
Earlier work this paper cites.
Topic-to-essay generation with neural networks.. In IJCAI . 4078–4084
Xiaocheng Feng, Ming Liu, et al · 2018
Earlier work this paper cites.
Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW) . IEEE, 50–56
Ji Gao, Jack Lanchantin, et al · 2018
Earlier work this paper cites.
Survey of the state of the art in natural language generation: Core tasks, applications and evaluation
Albert Gatt and Emiel Krahmer. 2018 · 2018
Earlier work this paper cites.
Universal language model fine-tuning for text classification
Original
Jeremy Howard and Sebastian Ruder. 2018 · 2018
Earlier work this paper cites.
Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning
Original
Ziming Li, Julia Kiseleva, et al · 2018
Earlier work this paper cites.
Online information laundering: The role of social media
Kirill Meleshevich and Bret Schafer. 2018 · 2018
Earlier work this paper cites.
Story Scrambler - Automatic Text Generation Using Word Level RNN-LSTM
Dipti Pawade, Avani M. Sakhapara, et al · 2018
Earlier work this paper cites.
Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms
P Jonathon Phillips, Amy N Yates, et al · 2018
Earlier work this paper cites.

Similar

Reddit Transparency Report: Suspicious Accounts
Reddit. 2018 · 2018
Cited alongside, same era.
Facebook Launches New Initiative to Help Scholars Assess Social Media’s Impact on Elections
Elliot Schrage and David Ginsberg. 2018 · 2018
Cited alongside, same era.
Toward Diverse Text Generation with Inverse Reinforcement Learning. In IJCAI
Zhan Shi, Xinchi Chen, et al · 2018
Cited alongside, same era.
Spear phishing in a barrel: Insights from a targeted phishing campaign
A. J. Burns, M. Eric Johnson, et al · 2019
Cited alongside, same era.
Towards ethical content-based detection of online influence campaigns. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) . IEEE, 1–6
Evan Crothers, Nathalie Japkowicz, et al · 2019
Cited alongside, same era.
Ethics guidelines for trustworthy AI
European Commission and Directorate-General for Communications Networks, Content and Technology. 2019 · 2019
Cited alongside, same era.
GLTR: Statistical Detection and Visualization of Generated Text. In Proc. 57th Annual Meeting of the ACL: System Demonstrations . ACL, Florence, Italy, 111–116
Sebastian Gehrmann, Hendrik Strobelt, et al · 2019
Cited alongside, same era.
Text summarization from legal documents: a survey
Ambedkar Kanapala, Sukomal Pal, et al · 2019
Cited alongside, same era.
CTRL: A Conditional Transformer Language Model for Controllable Generation
Original
Nitish Shirish Keskar, Bryan McCann, et al · 2019
Cited alongside, same era.
Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In Proc. 2019 CHI conference on human factors in computing systems . 1–12
Soomin Kim, Joonhwan Lee, et al · 2019
Cited alongside, same era.
Entangled transformer for image captioning. In Proc. IEEE/CVF . 8928–8937
Guang Li, Linchao Zhu, et al · 2019
Cited alongside, same era.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Original
Yinhan Liu, Myle Ott, et al · 2019
Cited alongside, same era.

Then

Robustness analysis of grover for machine-generated news detection. In Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association . 119–127
Rinaldo Gagiano, Maria Myung-Hee Kim, et al · 2021
Later among the works it cites.
Unsupervised and Distributional Detection of Machine-Generated Text
Matthias Gallé, Jos Rozen, et al · 2021
Later among the works it cites.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Original
Leo Gao, Stella Biderman, et al · 2021
Later among the works it cites.
Gotta CAPTCHA ’Em All: A Survey of 20 Years of the Human-or-Computer Dilemma
Meriem Guerar, Luca Verderame, et al · 2021
Later among the works it cites.
Artificial Text Detection via Examining the Topology of Attention Maps
Original
Laida Kushnareva, Daniil Cherniavskii, et al · 2021
Later among the works it cites.
Pretrained Language Model for Text Generation: A Survey. In Proc. 30th IJCAI , Zhi-Hua Zhou (Ed.). IJCAI Organization, 4492–4499
Junyi Li, Tianyi Tang, et al · 2021
Later among the works it cites.
Why do cover letters still exist?
Bryan Lufkin. 2021 · 2021
Later among the works it cites.
CodeGenX
Arya Manjaramkar. 2021 · 2021
Later among the works it cites.
Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models. In Proc. 16th Conf. Euro. Chap. ACL: Main Vol. ACL, Online, 1811–1822
Shaoor Munir, Brishna Batool, et al · 2021
Later among the works it cites.
Zero-shot text-to-image generation. In ICML . PMLR, 8821–8831
Aditya Ramesh, Mikhail Pavlov, et al · 2021
Later among the works it cites.
Generating Fake Cyber Threat Intelligence Using Transformer-Based Models. In 2021 International Joint Conference on Neural Networks (IJCNN) . 1–9
Priyanka Ranade, Aritran Piplai, et al · 2021
Later among the works it cites.
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, Andreas Blattmann, et al · 2021
Later among the works it cites.
Creating and detecting fake reviews of online products
Joni Salminen, Chandrashekhar Kandpal, et al · 2021
Later among the works it cites.
You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21) . 1559–1575
Roei Schuster, Congzheng Song, et al · 2021
Later among the works it cites.
Shostack’s 4 Question Frame for Threat Modeling
Adam Shostack. 2021 · 2021
Later among the works it cites.
Directive on automated decision-making
TB Canada Secretariat. 2021 · 2021
Later among the works it cites.
Deep Fake Recognition in Tweets Using Text Augmentation, Word Embeddings and Deep Learning. In International Conference on Computational Science and Its Applications . Springer, 523–538
Senait G Tesfagergish, Robertas Damaševičius, et al · 2021
Later among the works it cites.
Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet
Original
M. Onat Topal, Anil Bas, et al · 2021
Later among the works it cites.
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021 . Association for Computational Linguistics, Punta Cana, Dominican Republic, 2001–2016
Adaku Uchendu, Zeyu Ma, et al · 2021
Later among the works it cites.
GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model
Ben Wang and Aran Komatsuzaki. 2021 · 2021
Later among the works it cites.
A systematic review of automatic text summarization for biomedical literature and EHRs
Mengqian Wang, Manhua Wang, et al · 2021
Later among the works it cites.
PanGu- α \alpha : Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
Original
Wei Zeng, Xiaozhe Ren, et al · 2021
Later among the works it cites.
Fooling moss detection with pretrained language models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management . 2933–2943
Stella Biderman and Edward Raff. 2022 · 2022
Closest in time.
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Sid Black, Stella Biderman, et al · 2022
Closest in time.
Automatic Detection of Chinese Generated Essayss Based on Pre-trained BERT. In 2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC) , Vol. 10. IEEE, 2257–2260
Xingyuan Chen, Peng Jin, et al · 2022
Closest in time.
Adversarial Robustness of Neural-Statistical Features in Detection of Generative Transformers
Original
Evan Crothers, Nathalie Japkowicz, et al · 2022
Closest in time.
Automatic Detection of Hybrid Human-Machine Text Boundaries
Joseph Cutler, Liam Dugan, et al · 2022
Closest in time.
A Survey of Natural Language Generation
Chenhe Dong, Yinghui Li, et al · 2022
Closest in time.
Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 7250–7274
Yao Dou, Maxwell Forbes, et al · 2022
Closest in time.
Flooded with AI-generated images, some art communities ban them completely
Benj Edwards. 2022 · 2022
Closest in time.
BigScience RAIL License v1.0
Carlos Muñoz Ferrandis, Danish Contractor, et al · 2022
Closest in time.
using AI to write a youtube video
Drew Gooden. 2022 · 2022
Closest in time.
Exploiting GPT-3 prompts with malicious inputs that order the model to ignore its previous directions
Riley Goodside. 2022 · 2022
Closest in time.
The market for fake reviews
Sherry He, Brett Hollenbeck, et al · 2022
Closest in time.
The best ai writing assistant
Jasper AI. 2022 · 2022
Closest in time.
Deep Learning for Text Style Transfer: A Survey
Di Jin, Zhijing Jin, et al · 2022
Closest in time.
Trustworthy Artificial Intelligence: A Review
Davinder Kaur, Suleyman Uslu, et al · 2022
Closest in time.
Content across cultures
Simon Kemp. 2022a · 2022
Closest in time.
Digital 2022 Global Digital Overview
Simon Kemp. 2022b · 2022
Closest in time.
YaLM 100B
Michael Khrushchev, Ruslan Vasilev, et al · 2022
Closest in time.
This is the worst ai ever
Yannic Kilcher. 2022 · 2022
Closest in time.
Detecting and Understanding Textual Deepfakes in Online Reviews
Peter Kowalczyk, Marco Röder, et al · 2022
Closest in time.
Lessons from the GPT-4Chan Controversy
Andrey Kurenkov. 2022 · 2022
Closest in time.
The Time Is Now to Develop Community Norms for the Release of Foundation Models
Percy Liang, Rishi Bommasani, et al · 2022
Closest in time.
Condemning the deployment of gpt-4chan
Percy Liang and Rob Reich. 2022 · 2022
Closest in time.
DeepPhish: Understanding User Trust Towards Artificially Generated Profiles in Online Social Networks. In 31st USENIX Security Symposium (USENIX Security 22) . USENIX Association, Boston, MA, 1669–1686
Jaron Mink, Licheng Luo, et al · 2022
Closest in time.
Open Cover Letter: Generate Cover Letters with AI
Open Cover Letter. 2022 · 2022
Closest in time.
ChatGPT: Optimizing language models for dialogue
OpenAI. 2022 · 2022
Closest in time.
Natural language processing benchmarks
Papers With Code. 2022 · 2022
Closest in time.
A Generalist Agent
Scott Reed, Konrad Zolna, et al · 2022
Closest in time.
Cross-Domain Detection of GPT-2-Generated Technical Text. In Proc. 2022 Conf. Nor. Amer. Chapt. ACL: Human Language Technologies . ACL, Seattle, United States, 1213–1233
Juan Rodriguez, Todd Hay, et al · 2022
Closest in time.
SynSciPass: detecting appropriate uses of scientific text generation
Domenic Rosati. 2022 · 2022
Closest in time.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Teven Le Scao, Angela Fan, et al · 2022
Closest in time.
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
Original
Tatiana Shamardina, Vladislav Mikhailov, et al · 2022
Closest in time.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Kurt Shuster, Jing Xu, et al · 2022
Closest in time.
Artificial text detection in Russian language: a BERT-based Approach. In Dialogue
SS Skrylnikov, PA Posokhov, et al · 2022
Closest in time.
Detecting computer-generated disinformation
Harald Stiff and Fredrik Johansson. 2022 · 2022
Closest in time.
Automatic Detection of Bot-Generated Tweets. In Proc. 1st International Workshop on Multimedia AI against Disinformation (MAD ’22) . ACM, New York, NY, USA, 44–51
Julien Tourille, Babacar Sow, et al · 2022
Closest in time.
Twitter Elections Integrity Dataset
Twitter. 2019 · 2022
Closest in time.
Threat modelling
UK National Cyber Security Centre. 2022 · 2022
Closest in time.
GIT: A Generative Image-to-text Transformer for Vision and Language
Original
Jianfeng Wang, Zhengyuan Yang, et al · 2022
Closest in time.
Simon Willison. 2022
2022
Closest in time.
GPT-3 Prompt Injection Example
Juan Pablo Ossa Zapata. 2022 · 2022
Closest in time.
Opt: Open pre-trained transformer language models
Original
Susan Zhang, Stephen Roller, et al · 2022
Closest in time.
ACL 2023 policy on AI Writing Assistance
Jordan Boyd-Graber, Naoaki Okazaki, et al · 2023
Closest in time.
GPT detectors are biased against non-native English writers
Original
Weixin Liang, Mert Yuksekgonul, et al · 2023
Closest in time.
Locally Typical Sampling
Clara Meister, Tiago Pimentel, et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…