2023

Goal Driven Discovery of Distributional Differences via Language Descriptions

Zhong, Ruiqi, Zhang, Peter, Li, Steve et al.

Understand

Mining large corpora can generate useful discoveries but is time-consuming for humans.

We formulate a new task, D5, that automatically discovers differences between two large corpora in a goal-driven way.
The task input is a problem comprising a research goal "$\textit{comparing the side effects of drug A and drug B}$" and a corpus pair (two large collections of patients' self-reported reactions after taking each drug).
The output is a language description (discovery) of how these corpora differ (patients taking drug A "$\textit{mention feelings of paranoia}$" more often).

Built on

Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model
Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, and David Madigan · 1932
Earlier work this paper cites.
The structure of scientific revolutions
Dudley Shapere · 1964
Earlier work this paper cites.
Naturalistic epistemology
WVO Quine · 1969
Earlier work this paper cites.
Implicit social cognition: attitudes, self-esteem, and stereotypes
Anthony G Greenwald and Mahzarin R Banaji · 1995
Earlier work this paper cites.
Applied regression analysis , volume 326
Norman R Draper and Harry Smith · 1998
Earlier work this paper cites.
Foundations of statistical natural language processing
Christopher Manning and Hinrich Schutze · 1999
Earlier work this paper cites.
The republic of science: its political and economic theory minerva, i (1)(1962), 54-73
Michael Polanyi, John Ziman, and Steve Fuller · 2000
Earlier work this paper cites.
Latent dirichlet allocation
David M Blei, Andrew Y Ng, and Michael I Jordan · 2003
Earlier work this paper cites.
A survey of interestingness measures for knowledge discovery
Ken McGarry · 2005
Earlier work this paper cites.
A shared task involving multi-label classification of clinical free text
John P. Pestian, Chris Brew, Pawel Matykiewicz, DJ Hovermale, Neil Johnson, K. Bretonnel Cohen, and Wlodzislaw Duch · 2007
Earlier work this paper cites.
Natural language processing with Python: analyzing text with the natural language toolkit
Steven Bird, Ewan Klein, and Edward Loper · 2009
Earlier work this paper cites.
Reuter_50_50 Data Set, 2011
Zhi Liu · 2011
Earlier work this paper cites.
URL https://kaggle.com/competitions/asap-aes
The hewlett foundation: Automated essay scoring, 2012 · 2012
Earlier work this paper cites.
URL https://kaggle.com/competitions/asap-sas
The hewlett foundation: Short answer scoring, 2013 · 2013
Earlier work this paper cites.
Reasoning with neural tensor networks for knowledge base completion
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng · 2013
Earlier work this paper cites.
The snli corpus
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning · 2015
Earlier work this paper cites.
Experiments in open domain deception detection
Verónica Pérez-Rosas and Rada Mihalcea · 2015
Earlier work this paper cites.
Deception detection using real-life trial data
Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo · 2015
Earlier work this paper cites.
Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM
Ivan Habernal and Iryna Gurevych · 2016
Earlier work this paper cites.
Pointer sentinel mixture models, 2016
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2016
Earlier work this paper cites.
Learning multi-modal grounded linguistic semantics by playing" i spy"
Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J Mooney · 2016
Earlier work this paper cites.
Understanding state preferences with text as data: Introducing the un general debate corpus
Alexander Baturo, Niheer Dasandi, and Slava J Mikhaylov · 2017
Earlier work this paper cites.
Abducting economics
James J Heckman and Burton Singer · 2017
Earlier work this paper cites.
Kickstarter Projects, 2017
Mickaël Mouillé · 2017
Earlier work this paper cites.
Automatic detection of fake news
Original
Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea · 2017
Earlier work this paper cites.
U.S. Technology Jobs on Dice.com, 2017
PromptCloud · 2017
Earlier work this paper cites.
Quora Question Pairs, 2017
Quora · 2017
Earlier work this paper cites.
Daily News for Stock Market Prediction, 2017
J Sun · 2017
Earlier work this paper cites.
Armenian Online Job Postings, 2017
Udacity · 2017
Earlier work this paper cites.

Similar

A broad-coverage challenge corpus for sentence understanding through inference
Original
Adina Williams, Nikita Nangia, and Samuel R Bowman · 2017
Cited alongside, same era.
Happydb: A corpus of 100,000 crowdsourced happy moments
Original
Akari Asai, Sara Evensen, Behzad Golshan, Alon Halevy, Vivian Li, Andrei Lopatenko, Daniela Stepanov, Yoshihiko Suhara, Wang-Chiew Tan, and Yinzhan Xu · 2018
Cited alongside, same era.
Decoupling strategy and generation in negotiation dialogues, 2018
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang · 2018
Cited alongside, same era.
TMDB 5000 Movie Dataset, 2018
Kaggle · 2018
Cited alongside, same era.
A Million News Headlines, 2018
Rohit Kulkarni · 2018
Cited alongside, same era.
Multi-source social feedback of online news feeds
Nuno Moniz and Luâ€™is Torgo · 2018
Cited alongside, same era.
Know what you don’t know: Unanswerable questions for squad
Original
Pranav Rajpurkar, Robin Jia, and Percy Liang · 2018
Cited alongside, same era.
Poetry Foundation Poems, 2019
Divy Bramhecha · 2019
Cited alongside, same era.
Neural legal judgment prediction in english
Original
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras · 2019
Cited alongside, same era.
Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth · 2019
Cited alongside, same era.
Boolq: Exploring the surprising difficulty of natural yes/no questions
Original
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova · 2019
Cited alongside, same era.
Advertisement Transcripts from Various Industries, 2019
Kevin Hartman · 2019
Cited alongside, same era.

Then

Justice: A benchmark dataset for supreme court’s judgment prediction
Original
Mohammad Alali, Shaayan Syed, Mohammed Alsayed, Smit Patel, and Hemanth Bodala · 2021
Later among the works it cites.
Understanding the usage of online media for parenting from infancy to preschool at scale
Yujia Gao, Jinu Jang, and Diyi Yang · 2021
Later among the works it cites.
Goodbye world: using natural language processing to identify suicidal posts, 2021
Samuel He · 2021
Later among the works it cites.
Natural language descriptions of deep visual features
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas · 2021
Later among the works it cites.
Expertise and dynamics within crowdsourced musical knowledge curation: A case study of the genius platform
Derek Lim and Austin R Benson · 2021
Later among the works it cites.
Sculpting Data for ML: The first act of Machine Learning
Rishabh Misra and Jigyasa Grover · 2021
Later among the works it cites.
Crowdsourcing beyond annotation: Case studies in benchmark data collection
Alane Suhr, Clara Vania, Nikita Nangia, Maarten Sap, Mark Yatskar, Samuel Bowman, and Yoav Artzi · 2021
Later among the works it cites.
https://adobserver.org/, 2021
Ad observer · 2022
Later among the works it cites.
Yc company scraper
Akshay Bhalotia · 2022
Later among the works it cites.
Replication code and data for “Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration” [dataset]
Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky · 2022
Later among the works it cites.
Scaling instruction-finetuned language models
Original
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al · 2022
Later among the works it cites.
Domino: Discovering systematic errors with cross-modal embeddings
Original
Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré · 2022
Later among the works it cites.
Instruction induction: From few examples to natural language task descriptions
Original
Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy · 2022
Later among the works it cites.
India News Headlines Dataset, 2022
Rohit Kulkarni · 2022
Later among the works it cites.
Wanli: Worker and ai collaboration for natural language inference dataset creation, January 2022
Original
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi · 2022
Later among the works it cites.
Algorithmic behavioral science: Machine learning as a tool for scientific discovery
Jens Ludwig and Sendhil Mullainathan · 2022
Later among the works it cites.
Cross-task generalization via natural language crowdsourcing instructions
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Original
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022
Later among the works it cites.
Statements of Administration Policy, 2022
Demand Progress · 2022
Later among the works it cites.
Explaining patterns in data with language models via interpretable autoprompting
Original
Chandan Singh, John X Morris, Jyoti Aneja, Alexander M Rush, and Jianfeng Gao · 2022
Later among the works it cites.
Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks
Original
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al · 2022
Later among the works it cites.
Language models as inductive reasoners
Original
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei · 2022
Later among the works it cites.
Guess the instruction! making language models stronger zero-shot learners
Original
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo · 2022
Later among the works it cites.
Describing differences between text distributions with natural language
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt · 2022
Later among the works it cites.
Large language models are human-level prompt engineers
Original
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba · 2022
Later among the works it cites.
Gsclip: A framework for explaining distribution shifts in natural language
Original
Zhiying Zhu, Weixin Liang, and James Zou · 2022
Later among the works it cites.
Scaling laws for generative mixed-modal language models
Original
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer · 2023
Closest in time.
Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks
Original
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West · 2023
Closest in time.
Goal-driven explainable clustering via language descriptions
Original
Zihan Wang, Jingbo Shang, and Ruiqi Zhong · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…