What is my math transformer doing? -- Three results on interpretability and generalization

Built on

Random Matrices
Madan Lal Mehta · 2004
Earlier work this paper cites.
Matrix Computations
Gene H. Golub and Charles F. van Loan · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Original
Ilya Loshchilov and Frank Hutter · 2016
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.

Deep learning for symbolic mathematics
Original
Guillaume Lample and François Charton · 2019
Cited alongside, same era.
Generative language modeling for automated theorem proving
Original
Stanislas Polu and Ilya Sutskever · 2020
Cited alongside, same era.
It’s not what machines can learn, it’s what we cannot teach
Gal Yehuda, Moshe Gabel, and Assaf Schuster · 2020
Cited alongside, same era.
Neural symbolic regression that scales
Original
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo · 2021
Cited alongside, same era.

Linear algebra with transformers
Original
François Charton · 2021
Later among the works it cites.
Investigating the limitations of transformers with simple arithmetic tasks
Original
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin · 2021
Later among the works it cites.
Simplifying polylogarithms with machine learning
Original
Aurélien Dersy, Matthew D. Schwartz, and Xiaoyuan Zhang · 2022
Closest in time.
Symbolic brittleness in sequence models: on systematic generalization in symbolic mathematics
Sean Welleck, Peter West, Jize Cao, and Yejin Choi · 2022
Closest in time.