2019

A Tensorized Transformer for Language Modeling

Ma, Xindian, Zhang, Peng, Zhang, Shuai et al.

Understand

Latest development of neural models has connected the encoder and decoder through a self-attention mechanism.

In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks.
However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting.
In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD).

Built on

Some mathematical notes on three-mode factor analysis
Ledyard R Tucker · 1966
Earlier work this paper cites.
Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition
J Douglas Carroll and Jih-Jie Chang · 1970
Earlier work this paper cites.
Model compression
Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil · 2006
Earlier work this paper cites.
Decompositions of a higher-order tensor in block terms—part ii: Definitions and uniqueness
Lieven De Lathauwer · 2008
Earlier work this paper cites.
Tensor decompositions and applications
Tamara G Kolda and Brett W Bader · 2009
Earlier work this paper cites.
Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari · 2009
Earlier work this paper cites.
Empirical evaluation and combination of advanced language modeling techniques
Tomáš Mikolov, Anoop Deoras, Stefan Kombrink, Lukáš Burget, and Jan Černockỳ · 2011
Earlier work this paper cites.
Tensor-train decomposition
Ivan V Oseledets · 2011
Earlier work this paper cites.
One billion word benchmark for measuring progress in statistical language modeling
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson · 2013
Earlier work this paper cites.
Low-rank matrix factorization for deep neural network training with high-dimensional output targets
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran · 2013
Earlier work this paper cites.
Exploiting linear structure within convolutional networks for efficient evaluation
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus · 2014
Earlier work this paper cites.
Speeding up convolutional neural networks with low rank expansions
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman · 2014
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Earlier work this paper cites.
Tensorizing neural networks
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov · 2015
Earlier work this paper cites.

Similar

Group equivariant convolutional networks
Taco Cohen and Max Welling · 2016
Cited alongside, same era.
Ultimate tensorization: compressing convolutional and fc layers alike
Original
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov · 2016
Cited alongside, same era.
Tying word vectors and word classifiers: A loss framework for language modeling
Original
Hakan Inan, Khashayar Khosravi, and Richard Socher · 2016
Cited alongside, same era.
Exploring the limits of language modeling
Original
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu · 2016
Cited alongside, same era.
Pointer sentinel mixture models
Original
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2016
Cited alongside, same era.
Linguistic input features improve neural machine translation
Original
Rico Sennrich and Barry Haddow · 2016
Cited alongside, same era.
Edinburgh neural machine translation systems for wmt 16
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2016
Cited alongside, same era.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V Le · 2016
Cited alongside, same era.
Language modeling with gated convolutional networks
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier · 2017
Cited alongside, same era.
Bt-nets: simplifying deep neural networks via block term decomposition
Original
Guangxi Li, Jinmian Ye, Haiqin Yang, Di Chen, Shuicheng Yan, and Zenglin Xu · 2017
Cited alongside, same era.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Original
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.

Then

A quantum many-body wave function inspired language modeling approach
Peng Zhang, Zhan Su, Lipeng Zhang, Benyou Wang, and Dawei Song · 2018
Later among the works it cites.
Adaptive input representations for neural language modeling
Original
Alexei Baevski and Michael Auli · 2018
Later among the works it cites.
Groupreduce: Block-wise low-rank approximation for neural language model shrinking
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh · 2018
Later among the works it cites.
Universal transformers
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Later among the works it cites.
An analysis of neural language modeling at multiple scales
Original
Stephen Merity, Nitish Shirish Keskar, and Richard Socher · 2018
Later among the works it cites.
Deep contextualized word representations
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Later among the works it cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever · 2018
Later among the works it cites.
West: Word encoded sequence transducers
Original
Ehsan Variani, Ananda Theertha Suresh, and Mitchel Weintraub · 2018
Later among the works it cites.
Learning compact recurrent neural networks with block-term tensor decomposition
Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, and Zenglin Xu · 2018
Later among the works it cites.
Generating long sequences with sparse transformer
Original
Alec Radford Rewon Child, Scott Gray and Ilya Sutskever · 2019
Closest in time.
A generalized language model in tensor space
Original
Lipeng Zhang, Peng Zhang, Xindian Ma, Shuqin Gu, Zhan Su, and Dawei Song · 2019
Closest in time.
Transformer-xl: Attentive language models beyond a fixed-length context
Original
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov · 2019
Closest in time.
Tensorized embedding layers for efficient model compression
Original
Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan Oseledets · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…