Fetching the paper…

Reading the bibliography…

2019

BERTQA -- Attention on Steroids

Chadha, Ankit, Sood, Rewa

Original
PDF
Post on X

Understand

In this work, we extend the Bidirectional Encoder Representations from Transformers (BERT) with an emphasis on directed coattention to obtain an improved F1 performance on the SQUAD2.0 dataset.

The Transformer architecture on which BERT is based places hierarchical global attention on the concatenation of the context and query.
Our additions to the BERT architecture augment this attention with a more focused context to query (C2Q) and query to context (Q2C) attention via a set of modified Transformer encoder units.
In addition, we explore adding convolution-based feature extraction within the coattention architecture to add localized information to self-attention.

BERTQA -- Attention on Steroids · Around

Built on

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, https://arxiv.org/abs/1901.02860
Original
1901
Earlier work this paper cites.
Seo et al, Bidirectional Attention Flow For Machine Comprehension, ICLR 2017, https://arxiv.org/abs/1611.01603
Original
2017
Earlier work this paper cites.

Similar

Yu et al, QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension, ICLR 2018, https://arxiv.org/abs/1804.09541
Original
2018
Cited alongside, same era.
Devlin et al, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, https://arxiv.org/abs/1810.04805
Original
Cited in the paper.
Vaswani et al, Attention Is All You Need, https://arxiv.org/abs/1706.03762
Original
Cited in the paper.
The Big and Extending-Repository-of-Transformers: PyTorch pretrained models for Google’s BERT, OpenAI GPT, GPT-2, Google/CMU Transformer-XL, https://github.com/huggingface/pytorch-pretrained-BERT
Cited in the paper.
He et al, Deep Residual Learning for Image Recognition, https://arxiv.org/abs/1512.03385
Original
Cited in the paper.
SQUAD2.0 Augmented Data, https://github.com/ankit-ai/SQUAD2.Q-Augmented-Dataset
Cited in the paper.
BertQA - Attention on Steroids, https://github.com/ankit-ai/BertQA-Attention-on-Steroids
Cited in the paper.
Sun et al, U-Net: Machine Reading Comprehension with Unanswerable Questions, https://arxiv.org/pdf/1810.06638.pdf
Original
Cited in the paper.
Cui et al, Attention-over-Attention Neural Networks for Reading Comprehension, https://arxiv.org/abs/1607.04423
Original
Cited in the paper.
Tool for visualizing attention in BERT and OpenAI GPT-2, https://github.com/jessevig/bertviz
Cited in the paper.
Huang et al, FlowQA: Grasping Flow in History for Conversational Machine Comprehension, https://arxiv.org/abs/1810.06683
Original
Cited in the paper.
Neural Question Generation from Text: A Preliminary Study, https://arxiv.org/abs/1704.01792
Original
Cited in the paper.

Then

Rajpurkar et al, Know What You Don’t Know: Unanswerable Questions for SQuAD, ACL 2018, https://arxiv.org/abs/1806.03822
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…