Built on

On the Relationship between Self-Attention and Convolutional Layers
Original
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi · 1911
Earlier work this paper cites.
Towards Learning Convolutions from Scratch, July 2020
Original
Behnam Neyshabur · 2007
Earlier work this paper cites.
What is being transferred in transfer learning?
Original
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang · 2008
Earlier work this paper cites.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021
Original
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2010
Earlier work this paper cites.
Essentially No Barriers in Neural Network Energy Landscape
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht · 2018
Earlier work this paper cites.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson · 2018
Earlier work this paper cites.
Linear Mode Connectivity and the Lottery Ticket Hypothesis
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin · 2020
Earlier work this paper cites.

Similar

Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson · 2021
Cited alongside, same era.
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerstner, and Johanni Brea · 2021
Cited alongside, same era.
Random initialisations performing above chance and how to find them, November 2022
Original
Frederik Benzing, Simon Schug, Robert Meier, Johannes von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger · 2022
Cited alongside, same era.
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, July 2022
Original
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur · 2022
Cited alongside, same era.
What can linear interpolation of neural network loss landscapes tell us?, February 2022

Towards Democratizing Joint-Embedding Self-Supervised Learning, March 2023
Original
Florian Bordes, Randall Balestriero, and Pascal Vincent · 2023
Closest in time.
Linear Connectivity Reveals Generalization Strategies, January 2023
Original
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra · 2023
Closest in time.
FFCV: Accelerating Training by Removing Data Bottlenecks, June 2023
Original
Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry · 2023
Closest in time.
Mechanistic Lens on Mode Connectivity
Original
Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka · 2023
Closest in time.
Symmetries, Flat Minima And The Conserved Quantities Of Gradient Flow

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…

2023

Altintas, Gul Sena, Bachmann, Gregor, Noci, Lorenzo et al.

Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes.

While empirically well established, it unfortunately still lacks a proper theoretical understanding.
Even worse, although empirical data points are abound, a systematic study of when networks exhibit LMC is largely missing in the literature.
In this work we aim to close this gap.

On the Relationship between Self-Attention and Convolutional Layers
Original
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi · 1911
Earlier work this paper cites.
Towards Learning Convolutions from Scratch, July 2020
Original
Behnam Neyshabur · 2007
Earlier work this paper cites.
What is being transferred in transfer learning?
Original
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang · 2008
Earlier work this paper cites.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021
Original
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2010
Earlier work this paper cites.
Essentially No Barriers in Neural Network Energy Landscape
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht · 2018
Earlier work this paper cites.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson · 2018
Earlier work this paper cites.
Linear Mode Connectivity and the Lottery Ticket Hypothesis
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin · 2020
Earlier work this paper cites.

Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson · 2021
Cited alongside, same era.
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerstner, and Johanni Brea · 2021
Cited alongside, same era.
Random initialisations performing above chance and how to find them, November 2022
Original
Frederik Benzing, Simon Schug, Robert Meier, Johannes von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger · 2022
Cited alongside, same era.
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, July 2022
Original
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur · 2022
Cited alongside, same era.
What can linear interpolation of neural network loss landscapes tell us?, February 2022