2022

MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning

Karpas, Ehud, Abend, Omri, Belinkov, Yonatan et al.

Understand

Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks.

Although an essential element of modern AI, LMs are also inherently limited in a number of ways.
We discuss these limitations and how they can be avoided by adopting a systems approach.
Conceptualizing the challenge as one that involves knowledge and reasoning in addition to linguistic processing, we define a flexible architecture with multiple neural models, complemented by discrete knowledge and reasoning modules.

“RoBERTa: A Robustly Optimized BERT Pretraining Approach”
Yinhan Liu et al · 1907
Earlier work this paper cites.
“Language Models as Knowledge Bases?”
Fabio Petroni et al · 1909
Earlier work this paper cites.
“How Can We Know What Language Models Know?”
Zhengbao Jiang, Frank. Xu, Jun Araki and Graham Neubig · 1911
Earlier work this paper cites.
“Break It Down: A Question Understanding Benchmark”
Tomer Wolfson et al · 2001
Earlier work this paper cites.
“The Cost of Training NLP Models: A Concise Overview”
Or Sharir, Barak Peleg and Yoav Shoham · 2004
Earlier work this paper cites.
“Language Models are Few-Shot Learners”
Tom. Brown et al · 2005
Earlier work this paper cites.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2019
Earlier work this paper cites.

“Energy and Policy Considerations for Deep Learning in NLP”
Emma Strubell, Ananya Ganesh and Andrew McCallum · 2019
Cited alongside, same era.
“Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”
Colin Raffel et al · 2020
Cited alongside, same era.
“Jurassic-1: Technical Details and Evaluation”
Opher Lieber, Or Sharir, Barak Lenz and Yoav Shoham · 2021
Cited alongside, same era.
“On the Opportunities and Risks of Foundation Models”
Rishi Bommasani et al · 2021
Cited alongside, same era.
“Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models”
Alex Tamkin, Miles Brundage, Jack Clark and Deep Ganguli · 2021
Cited alongside, same era.
“Finetuned Language Models Are Zero-Shot Learners”
Jason Wei et al · 2021
Cited alongside, same era.
“MetaICL: Learning to Learn In Context”
Sewon Min, Mike Lewis, Luke Zettlemoyer and Hannaneh Hajishirzi · 2021
Cited alongside, same era.

“Sustainable AI: Environmental Implications, Challenges and Opportunities”
Carole-Jean Wu et al · 2021
Later among the works it cites.
“The Power of Scale for Parameter-Efficient Prompt Tuning”
Brian Lester, Rami Al-Rfou and Noah Constant · 2021
Later among the works it cites.
“PaLM: Scaling Language Modeling with Pathways”
Aakanksha Chowdhery et al · 2022
Closest in time.
“Multitask Prompted Training Enables Zero-Shot Task Generalization”
Victor Sanh et al · 2022
Closest in time.
“ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning”
Vamsi Aribandi et al · 2022
Closest in time.
“Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”
Shaden Smith et al · 2022
Closest in time.
“TimeLMs: Diachronic Language Models from Twitter”
Daniel Loureiro et al · 2022
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…