Fetching the paper…

Reading the bibliography…

2020

On-The-Fly Information Retrieval Augmentation for Language Models

Wang, Hai, McAllester, David

Original
PDF
Post on X

Understand

Here we experiment with the use of information retrieval as an augmentation for pre-trained language models.

The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time.
By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training.
We also validate our IR augmentation on an event co-reference task.

On-The-Fly Information Retrieval Augmentation for Language Models · Around

Built on

Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text
Original
Haitian Sun, Tania Bedrax-Weiss, and William W Cohen. 2019 · 1904
Earlier work this paper cites.
English gigaword
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011 · 2011
Earlier work this paper cites.
Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution
Agata Cybulska and Piek Vossen. 2014 · 2014
Earlier work this paper cites.
Memory networks
Original
Jason Weston, Sumit Chopra, and Antoine Bordes. 2014 · 2014
Earlier work this paper cites.
Translating granularity of event slots into features for event coreference resolution
Agata Cybulska and Piek Vossen. 2015 · 2015
Earlier work this paper cites.
Representing text for joint embedding of text and knowledge bases
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015 · 2015
Earlier work this paper cites.
Hierarchical memory networks
Original
Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Bengio. 2016 · 2016
Earlier work this paper cites.
Contextual lstm (clstm) models for large scale nlp tasks
Original
Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck. 2016 · 2016
Earlier work this paper cites.
Improving neural language models with a continuous cache
Original
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016 · 2016
Earlier work this paper cites.
Key-value memory networks for directly reading documents
Original
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016 · 2016
Earlier work this paper cites.
The lambada dataset: Word prediction requiring a broad discourse context
Original
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016 · 2016
Earlier work this paper cites.

Similar

Scaling memory-augmented neural networks with sparse reads and writes
Original
Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. 2016 · 2016
Cited alongside, same era.
Hierarchical memory networks for answer selection on unknown words
Jing Shi, Yiqun Yao, Suncong Zheng, Bo Xu, et al. 2016 · 2016
Cited alongside, same era.
Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning
Original
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy, Alex Smola, and Andrew McCallum. 2017 · 2017
Cited alongside, same era.
Linguistic knowledge as memory for recurrent neural networks
Original
Bhuwan Dhingra, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2017 · 2017
Cited alongside, same era.
Memory augmented neural networks with wormhole connections

Then

Visual question answering with memory-augmented networks
Original
Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. 2018 · 2018
Later among the works it cites.
Extending neural generative conversational model using external knowledge sources
Original
Prasanna Parthasarathi and Joelle Pineau. 2018 · 2018
Later among the works it cites.
Learning visual knowledge memory networks for visual question answering
Original
Zhou Su, Chen Zhu, Yinpeng Dong, Dongqi Cai, Yurong Chen, and Jianguo Li. 2018 · 2018
Later among the works it cites.
Open domain question answering using early fusion of knowledge bases and text
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018 · 2018
Later among the works it cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. 2017 · 2017

Cited alongside, same era.

Topically driven neural language model

Jey Han Lau, Timothy Baldwin, and Trevor Cohn. 2017 · 2017

Cited alongside, same era.

A read-write memory network for movie story understanding

Seil Na, Sangho Lee, Jisung Kim, and Gunhee Kim. 2017 · 2017

Cited alongside, same era.

Leveraging knowledge bases in lstms for improving machine reading

Bishan Yang and Tom Mitchell. 2017 · 2017

Cited alongside, same era.

Neural natural language inference models enhanced with external knowledge

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2018 · 2018

Cited alongside, same era.

Resolving event coreference with supervised representation learning and clustering-oriented regularization

Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018 · 2018

Cited alongside, same era.

Sharp nearby, fuzzy far away: How neural language models use context

Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018 · 2018

Cited alongside, same era.

Ask me anything: Dynamic memory networks for natural language processing

Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016a

Cited in the paper.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018

Later among the works it cites.

Revisiting joint modeling of cross-document entity and event coreference resolution

Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019 · 2019

Later among the works it cites.

Large memory layers with product keys

Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2019 · 2019

Later among the works it cites.

KnowSemLM: A Knowledge Infused Semantic Language Model

Haoruo Peng, Qiang Ning, and Dan Roth. 2019 · 2019

Later among the works it cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Later among the works it cites.

Enhancing unsupervised pretraining with external knowledge for natural language inference

Xiaoyu Yang, Xiaodan Zhu, Huasha Zhao, Qiong Zhang, and Yufei Feng. 2019 · 2019

Later among the works it cites.