Fetching the paper…

Reading the bibliography…

2017

Backprop as Functor: A compositional perspective on supervised learning

Fong, Brendan, Spivak, David I., Tuyéras, Rémy

Original
PDF
Post on X

Understand

A supervised learning algorithm searches over a set of functions $A \to B$ parametrised by a space $P$ to find the best approximation to some ideal function $f\colon A \to B$.

It does this by taking examples $(a,f(a)) \in A\times B$, and updating the parameter according to some rule.
We define a category where these update rules may be composed, and show that gradient descent---with respect to a fixed step size and an error function satisfying a certain property---defines a monoidal functor from a category of parametrised functions to this category of update rules.
This provides a structural perspective on backpropagation, as well as a broad generalisation of neural networks.

Backprop as Functor: A compositional perspective on supervised learning · Around

Built on

From Open Learners to Open Games
Original
J. Hedges · 1902
Earlier work this paper cites.
Lenses and Learners
Original
B. Fong and M. Johnson · 1903
Earlier work this paper cites.
The geometry of tensor calculus I
A. Joyal and R. Street · 1991
Earlier work this paper cites.
Convolutional networks for images, speech, and time series
Y. Le Cun and T. Bengio · 1995
Earlier work this paper cites.
Categories for the Working Mathematician , Springer, Berlin, 1998
S. Mac Lane · 1998
Earlier work this paper cites.
Relational lenses: a language for updatable views
A. Bohannon, B. C. Pierce, and J. A. Vaughan · 2006
Earlier work this paper cites.

Similar

Spans of lenses
M. Johnson and R. D. Rosebrugh · 2014
Cited alongside, same era.
Categories in Control
J. C. Baez and J. Erbele · 2015
Cited alongside, same era.
Understanding deep image representations by inverting them
A. Mahendran and A. Vedaldi · 2015
Cited alongside, same era.
Neural Networks and Deep Learning
M. A. Nielsen · 2015
Cited alongside, same era.
A compositional framework for Markov processes
J. C. Baez, B. Fong, and B. Pollard · 2016
Cited alongside, same era.
The Algebra of Directed Acyclic Graphs
Original
M. Fiore and M. Devesas Campos
Cited in the paper.

Then

Interacting Hopf Algebras
F. Bonchi, P. Sobociński, and F. Zanasi · 2016
Later among the works it cites.
The Algebra of Open and Interconnected Systems
Original
B. Fong · 2016
Later among the works it cites.
Geometric deep learning: going beyond Euclidean data
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, P. Vandergheynst · 2017
Closest in time.
Picturing Quantum Processes A First Course in Quantum Theory and Diagrammatic Reasoning
B. Coecke and A. Kissinger · 2017
Closest in time.
Compositional Game Theory
N. Ghani, J. Hedges, V. Winschel, P. Zahn · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Backprop as Functor: A compositional perspective on supervised learning. (Extended version.)

B. Fong, D. I. Spivak, R. Tuyéras

Cited in the paper.

Algebras of open dynamical systems on the operad of wiring diagrams

D. Vagner, D. Spivak. E. Lerman

Cited in the paper.