2023

Duality of Bures and Shape Distances with Implications for Comparing Neural Representations

Harvey, Sarah E., Larsen, Brett W., Williams, Alex H.

Understand

A multitude of (dis)similarity measures between neural network representations have been proposed, resulting in a fragmented research landscape.

Most of these measures fall into one of two categories.
First, measures such as linear regression, canonical correlations analysis (CCA), and shape distances, all learn explicit mappings between neural units to quantify similarity while accounting for expected invariances.
Second, measures such as representational similarity analysis (RSA), centered kernel alignment (CKA), and normalized Bures similarity (NBS) all quantify similarity in summary statistics, such as stimulus-by-stimulus kernel matrices, which are already invariant to expected symmetries.

Built on

“A generalized solution of the orthogonal procrustes problem”
Peter Schönemann · 1966
Earlier work this paper cites.
“Foundations of multidimensional scaling.”
Richard Beals, David Krantz and Amos Tversky · 1968
Earlier work this paper cites.
“The Diffusion of Shape”
D Kendall · 1977
Earlier work this paper cites.
“Representation is representation of similarities”
Shimon Edelman · 1998
Earlier work this paper cites.
“Quantum Computation and Quantum Information”
Michael. Nielsen and Isaac. Chuang · 2000
Earlier work this paper cites.
“On Kernel-Target Alignment”
Nello Cristianini, John Shawe-Taylor, André Elisseeff and Jaz Kandola · 2001
Earlier work this paper cites.
“Learning with kernels: support vector machines, regularization, optimization, and beyond”
Bernhard Schölkopf and Alexander Smola · 2002
Earlier work this paper cites.
“Similarity of Neural Networks with Gradients”, 2020
Original
Shuai Tang, Wesley. Maddox, Charlie Dickens, Tom Diethe and Andreas Damianou · 2003
Earlier work this paper cites.
“Geometric Means in a Novel Vector Space Structure on Symmetric Positive-Definite Matrices”
Vincent Arsigny, Pierre Fillard, Xavier Pennec and Nicholas Ayache · 2007
Earlier work this paper cites.
“Duality in Mathematics and Physics” Lecture notes from the Institut de Matematica de la Universitat de Barcelona (IMUB), 2007
Michael Atiyah · 2007
Earlier work this paper cites.
“Representational similarity analysis - connecting the branches of systems neuroscience”
Nikolaus Kriegeskorte, Marieke Mur and Peter Bandettini · 2008
Earlier work this paper cites.
“Matching categorical object representations in inferior temporal cortex of man and monkey”
Nikolaus Kriegeskorte, Marieke Mur, Douglas Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka and Peter Bandettini · 2008
Earlier work this paper cites.
“Alternative fidelity measure between quantum states”
Paulo… Mendonçça, Reginaldo.. Napolitano, Marcelo. Marchiolli, Christopher. Foster and Yeong-Cherng Liang · 2008
Earlier work this paper cites.
“Non-Euclidean statistics for covariance matrices, with applications to diffusion tensor imaging”
Ian. Dryden, Alexey Koloydenko and Diwei Zhou · 2009
Earlier work this paper cites.
“Shape and shape theory”
David Kendall, Dennis Barden, Thomas Carne and Huiling Le · 2009
Earlier work this paper cites.
“Algorithms for learning kernels based on centered alignment”
Corinna Cortes, Mehryar Mohri and Afshin Rostamizadeh · 2012
Earlier work this paper cites.
“A toolbox for representational similarity analysis”
Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wilson and Nikolaus Kriegeskorte · 2014
Earlier work this paper cites.
“Distances and inference for covariance operators”
Davide Pigoli, John.. Aston, Ian. Dryden and Piercesare Secchi · 2014
Earlier work this paper cites.
“Convergent Learning: Do different neural networks learn the same representations?”
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson and John Hopcroft · 2015
Earlier work this paper cites.

Similar

“A neural network that finds a naturalistic solution for the production of muscle activity”
David Sussillo, Mark Churchland, Matthew Kaufman and Krishna Shenoy · 2015
Cited alongside, same era.
“Inferring brain-computational mechanisms with models of activity measurements”
Nikolaus Kriegeskorte and Jörn Diedrichsen · 2016
Cited alongside, same era.
“Reliability of dissimilarity measures for multi-voxel pattern analysis”
Alexander Walther, Hamed Nili, Naveed Ejaz, Arjen Alink, Nikolaus Kriegeskorte and Jörn Diedrichsen · 2016
Cited alongside, same era.
“Learning from uncertain curves: The 2-Wasserstein metric for Gaussian processes”
Anton Mallasto and Aasa Feragen · 2017
Cited alongside, same era.
“Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability”
Maithra Raghu, Justin Gilmer, Jason Yosinski and Jascha Sohl-Dickstein · 2017
Cited alongside, same era.
“On the Bures–Wasserstein distance between positive definite matrices”
Rajendra Bhatia, Tanvi Jain and Yongdo Lim · 2018
Cited alongside, same era.
“Gaussian Process Behaviour in Wide Deep Neural Networks”, 2018
Original
Alexander. de G., Mark Rowland, Jiri Hron, Richard. Turner and Zoubin Ghahramani · 2018
Cited alongside, same era.
“Neural tangent kernel: Convergence and generalization in neural networks”
Arthur Jacot, Franck Gabriel and Clément Hongler · 2018
Cited alongside, same era.
“Deep Neural Networks as Gaussian Processes”
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz and Yasaman Bahri · 2018
Cited alongside, same era.
“Wasserstein Riemannian geometry of Gaussian densities”
Luigi Malagò, Luigi Montrucchio and Giovanni Pistone · 2018
Cited alongside, same era.
“Insights on representational similarity in neural networks with canonical correlation”
Ari Morcos, Maithra Raghu and Samy Bengio · 2018
Cited alongside, same era.
“Generalizing Point Embeddings using the Wasserstein Space of Elliptical Distributions”
Boris Muzellec and Marco Cuturi · 2018
Cited alongside, same era.

Then

“Procrustes Metrics on Covariance Operators and Optimal Transportation of Gaussian Processes”
Valentina Masarotto, Victor. Panaretos and Yoav Zemel · 2019
Later among the works it cites.
“Computational Optimal Transport: With Applications to Data Science”
Gabriel Peyré and Marco Cuturi · 2019
Later among the works it cites.
“Gradient descent algorithms for Bures-Wasserstein barycenters”
Sinho Chewi, Tyler Maunu, Philippe Rigollet and Austin Stromme · 2020
Later among the works it cites.
“Long-term stability of cortical population dynamics underlying consistent behavior”
Juan Gallego, Matthew Perich, Raeed Chowdhury, Sara Solla and Lee Miller · 2020
Later among the works it cites.
“Primary visual cortex straightens natural video trajectories”
Olivier Hénaff, Yoon Bai, Julie Charlton, Ian Nauhaus, Eero Simoncelli and Robbe Goris · 2021
Later among the works it cites.
“Statistical inference for Bures–Wasserstein barycenters”
Alexey Kroshnin, Vladimir Spokoiny and Alexandra Suvorikova · 2021
Later among the works it cites.
“Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth”
Thao Nguyen, Maithra Raghu and Simon Kornblith · 2021
Later among the works it cites.
“Using distance on the Riemannian manifold to compare representations in brain and in models”
Mahdiyar Shahbazi, Ali Shirali, Hamid Aghajan and Hamed Nili · 2021
Later among the works it cites.
“Generalized Shape Metrics on Neural Representations”
Alex. Williams, Erin Kunz, Simon Kornblith and Scott. Linderman · 2021
Later among the works it cites.
“If you’ve trained one you’ve trained them all: inter-architecture similarity increases with robustness”
Haydn. Jones, Jacob. Springer, Garrett. Kenyon and Juston. Moore · 2022
Later among the works it cites.
“Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Human Behavioral Alignment and Adversarial Robustness”
Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Geary, Michael Ferguson, David Cox and James. DiCarlo · 2023
Closest in time.
“Reliability of CKA as a Similarity Measure in Deep Learning”
MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf and Eugene Belilovsky · 2023
Closest in time.
“Representational dissimilarity metric spaces for stochastic neural networks”
Lyndon. Duong, Jingyang Zhou, Josue Nassar, Jules Berman, Jeroen Olieslagers and Alex. Williams · 2023
Closest in time.
“Exploring perceptual straightness in learned visual representations”
Anne Harrington, Vasha DuTell, Ayush Tewari, Mark Hamilton, Simon Stent, Ruth Rosenholtz and William. Freeman · 2023
Closest in time.
“Similarity of Neural Network Models: A Survey of Functional and Representational Measures”, 2023
Original
Max Klabunde, Tobias Schumacher, Markus Strohmaier and Florian Lemmerich · 2023
Closest in time.
“Statistical inference on representational geometries”
Heiko Schütt, Alexander Kipnis, Jörn Diedrichsen and Nikolaus Kriegeskorte · 2023
Closest in time.
“Getting aligned on representational alignment”
Original
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley Love, Erin Grant, Jascha Achterberg and Joshua Tenenbaum · 2023
Closest in time.
“Bures–Wasserstein Minimizing Geodesics between Covariance Matrices of Different Ranks”
Yann Thanwerdas and Xavier Pennec · 2023
Closest in time.
“Diverse Deep Neural Networks All Predict Human Inferior Temporal Cortex Well, After Training and Fitting”
Katherine. Storrs, Tim. Kietzmann, Alexander Walther, Johannes Mehrer and Nikolaus Kriegeskorte · 2064
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…