Fetching the paper…
Reading the bibliography…
Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning.
An Inertial Newton Algorithm for Deep Learning
Castera C., Bolte J., Févotte C., Pauwels E. (2019) · 1905
Earlier work this paper cites.
A stochastic approximation method
Robbins H. and Monro, S. (1951) · 1951
Earlier work this paper cites.
Compiling fast partial derivatives of functions given by algorithms (No. COO-2383-0063; UILU-ENG-80-1702; UIUCDCS-R-80-1002)
Speelpenning, B. (1980) · 1980
Earlier work this paper cites.
Optimization and nonsmooth analysis
Clarke F. H. (1983) · 1983
Earlier work this paper cites.
Learning representations by back-propagating errors
Rumelhart E., Hinton E., Williams J. (1986) · 1986
Earlier work this paper cites.
ADIFOR-generating derivative codes from Fortran programs
Bischof, C., Carle, A., Corliss, G., Griewank, A., Hovland, P. (1992) · 1992
Earlier work this paper cites.
ADIFOR 2.0: Automatic differentiation of Fortran 77 programs
Bischof, C., Khademi, P., Mauer, A., Carle, A. (1996) · 1996
Earlier work this paper cites.
Geometric categories and o-minimal structures
van den Dries L. and Miller C. (1996) · 1996
Earlier work this paper cites.
Algorithm 755: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++
Griewank, A., Juedes, D., Utke, J. (1996) · 1996
Earlier work this paper cites.
Variational analysis
Rockafellar, R. T., Wets, R. J. B. (1998) · 1998
Earlier work this paper cites.
Dynamics of stochastic approximation algorithms
Benaïm, M. (1999) · 1999
Earlier work this paper cites.
RAAG notes, Institut de Recherche Mathématique de Rennes, 81 pages, November 1999
Coste M., An introduction to o-minimal geometry · 1999
Earlier work this paper cites.
A theorem of the complement and some new o-minimal structures
Wilkie, A. J. (1999) · 1999
Earlier work this paper cites.
Learning with differentiable perturbed optimizers
Berthet, Q., Blondel, M., Teboul, O., Cuturi, M., Vert, J. P., Bach, F. (2020) · 2002
Earlier work this paper cites.
Fast Differentiable Sorting and Ranking
Blondel, M., Teboul, O., Berthet, Q., Djolonga, J. (2020) · 2002
Earlier work this paper cites.
On Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions
Zhang, J., Lin, H., Sra, S., Jadbabaie, A. (2020) · 2002
Cited alongside, same era.
Stochastic approximation and recursive algorithms and applications (Vol. 35)
Kushner H. and Yin, G. G. (2003) · 2003
Cited alongside, same era.
Stochastic approximations and differential inclusions
Benaïm, M., Hofbauer, J., Sorin, S. (2005) · 2005
Cited alongside, same era.
Convergence of constant step stochastic gradient descent for non-smooth non-convex functions
Bianchi, P., Hachem, W., and Schechtman, S. (2020) · 2005
Cited alongside, same era.
On Correctness of Automatic Differentiation for Non-Differentiable Functions
Lee, W., Yu, H., Rival, X., Yang, H. (2020) · 2006
Cited alongside, same era.
Deep learning
LeCun Y., Bengio Y., Hinton, G. (2015) · 2015
Later among the works it cites.
Tensorflow: A system for large-scale machine learning
Abadi M., Barham P., Chen J., Chen Z., Davis A., Dean J., Devin M., Ghemawat S., Irving G., Isard M., Kudlur M., Levenberg J., Monga R., Moore S., Murray D., Steiner B., Tucker P., Vasudevan V., Warden P., Wicke M., Yu Y. and Zheng X. (2016) · 2016
Later among the works it cites.
On Lipschitz optimization based on gray-box piecewise linearization
Griewank A., Walther A., Fiege S. and Bosse T. (2016) · 2016
Later among the works it cites.
First-and second-order optimality conditions for piecewise smooth objective functions
Griewank, A., Walther, A. (2016) · 2016
Later among the works it cites.
Automatic differentiation in pytorch
Paszke A., Gross S., Chintala S., Chanan G., Yang E., DeVito Z., Lin Z., Desmaison A., Antiga L. and Lerer A. (2017) · 2017
Later among the works it cites.
Computationally relevant generalized derivatives: theory, evaluation and applications
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Mathematics for computer science
Lehman, E., Leighton, T., and Meyer, A. R. (2010) · 2006
Cited alongside, same era.
Clarke subgradients of stratifiable functions
Bolte, J., Daniilidis, A., Lewis, A., Shiota, M. (2007) · 2007
Cited alongside, same era.
The tradeoffs of large scale learning
Bottou L. and Bousquet O. (2008) · 2008
Cited alongside, same era.
Evaluating derivatives: principles and techniques of algorithmic differentiation (Vol. 105)
Griewank, A., Walther, A. (2008) · 2008
Cited alongside, same era.
Stochastic approximation: a dynamical systems viewpoint (Vol. 48). Springer
Borkar, V. (2009) · 2009
Cited alongside, same era.
Non-asymptotic analysis of stochastic approximation algorithms for machine learning
Moulines E. and Bach, F. (2011) · 2011
Cited alongside, same era.
Who invented the reverse mode of differentiation
Griewank, A. (2012) · 2012
Cited alongside, same era.
Barton, P. I., Khan, K. A., Stechlinski, P., Watson, H. A. (2018) · 2018
Later among the works it cites.
Automatic differentiation in machine learning: a survey
Baydin A., Pearlmutter B., Radul A. and Siskind J. (2018) · 2018
Later among the works it cites.
Optimization methods for large-scale machine learning
Bottou L., Curtis F. E. and Nocedal J. (2018) · 2018
Later among the works it cites.
Chizat, L., and Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. In Advances in neural information processing systems, 3036-3046
2018
Later among the works it cites.
Stochastic subgradient method converges on tame functions
Davis, D., Drusvyatskiy, D., Kakade, S., Lee, J. D. (2018) · 2018
Later among the works it cites.
Provably correct automatic sub-differentiation for qualified programs
Kakade, S. M. and Lee, J. D. (2018) · 2018
Later among the works it cites.
Treating artificial neural net training as a nonsmooth global optimization problem
Griewank, A., Rojas, A. (2019, September) · 2019
Later among the works it cites.
Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning
Bolte, J. and Pauwels, E. (2020) · 2020
Closest in time.
Beyond the Oracle: Opportunities of Piecewise Differentiation
Griewank, A., Walther, A. (2020) · 2020
Closest in time.