Fetching the paper…
Reading the bibliography…
Recent work has highlighted several advantages of enforcing orthogonality in the weight layers of deep networks, such as maintaining the stability of activations, preserving gradient norms, and enhancing adversarial robustness by enforcing low Lipschitz constants.
An iterative algorithm for computing the best estimate of an orthogonal matrix
Åke Björck and Clazett Bowie · 1971
Earlier work this paper cites.
Fundamentals of digital image processing
Anil K Jain · 1989
Earlier work this paper cites.
Fixup initialization: Residual learning without normalization
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma · 1989
Earlier work this paper cites.
Remarks on the cayley representation of orthogonal matrices and on perturbing the diagonal of a matrix to make it invertible
Jean Gallier · 2006
Earlier work this paper cites.
The Schur complement and its applications , volume 4
Fuzhen Zhang · 2006
Earlier work this paper cites.
Optimization algorithms on matrix manifolds
P-A Absil, Robert Mahony, and Rodolphe Sepulchre · 2009
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli · 2013
Earlier work this paper cites.
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Unitary evolution recurrent neural networks, 2016
Martin Arjovsky, Amar Shah, and Yoshua Bengio · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Tim Salimans and Durk P Kingma · 2016
Earlier work this paper cites.
Sergey Zagoruyko and Nikos Komodakis · 2016
Earlier work this paper cites.
Spectrally-normalized margin bounds for neural networks
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky · 2017
Earlier work this paper cites.
Towards evaluating the robustness of neural networks
Nicholas Carlini and David Wagner · 2017
Earlier work this paper cites.
Maximum resilience of artificial neural networks
Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess · 2017
Earlier work this paper cites.
Parseval networks: Improving robustness to adversarial examples
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier · 2017
Earlier work this paper cites.
Dawnbench: An end-to-end deep learning benchmark and competition
Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia · 2017
Earlier work this paper cites.
Formal verification of piece-wise linear feed-forward neural networks
Ruediger Ehlers · 2017
Cited alongside, same era.
Formal guarantees on the robustness of a classifier against adversarial manipulation
Matthias Hein and Maksym Andriushchenko · 2017
Cited alongside, same era.
Safety verification of deep neural networks
Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu · 2017
Cited alongside, same era.
Provable defenses against adversarial examples via the convex outer adversarial polytope
J Zico Kolter and Eric Wong · 2017
Cited alongside, same era.
An approach to reachability analysis for feed-forward relu neural networks
Alessio Lomuscio and Lalit Maganti · 2017
Cited alongside, same era.
Scaling provable adversarial defenses
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter · 2018
Later among the works it cites.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington · 2018
Later among the works it cites.
Sorting out lipschitz function approximation
Cem Anil, James Lucas, and Roger Grosse · 2019
Later among the works it cites.
Certified adversarial robustness via randomized smoothing
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter · 2019
Later among the works it cites.
Efficient and accurate estimation of lipschitz constants for deep neural networks
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas · 2019
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli · 2017
Cited alongside, same era.
Regularizing cnns with locally constrained decorrelations, 2017
Pau Rodríguez, Jordi Gonzàlez, Guillem Cucurull, Josep M. Gonfaus, and Xavier Roca · 2017
Cited alongside, same era.
Verifying neural networks with mixed integer programming
Vincent Tjeng and Russ Tedrake · 2017
Cited alongside, same era.
On orthogonality and learning recurrent networks with long term dependencies
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal · 2017
Cited alongside, same era.
Can we gain more from orthogonality regularizations in training deep networks?
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang · 2018
Cited alongside, same era.
Regularisation of neural networks by enforcing lipschitz continuity
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree · 2018
Cited alongside, same era.
Orthogonal recurrent neural networks with scaled cayley transform
Kyle Helfrich, Devin Willmott, and Qiang Ye · 2018
Cited alongside, same era.
Trivializations for gradient-based optimization on manifolds
Mario Lezcano-Casado · 2019
Later among the works it cites.
Mario Lezcano-Casado and David Martínez-Rubio · 2019
Later among the works it cites.
Preventing gradient attenuation in lipschitz constrained convolutional networks
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn-Henrik Jacobsen · 2019
Later among the works it cites.
Complex unitary recurrent neural networks using scaled cayley transform
Kehelwala DG Maduranga, Kyle E Helfrich, and Qiang Ye · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala · 2019
Later among the works it cites.
Adversarial robustness on in- and out-distribution improves explainability, 2020
Maximilian Augustin, Alexander Meinke, and Matthias Hein · 2020
Later among the works it cites.
Semialgebraic optimization for lipschitz constants of relu networks
Tong Chen, Jean B Lasserre, Victor Magron, and Edouard Pauwels · 2020
Later among the works it cites.
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
Francesco Croce and Matthias Hein · 2020
Later among the works it cites.
Provable benefit of orthogonal initialization in optimizing deep linear networks
Wei Hu, Lechao Xiao, and Jeffrey Pennington · 2020
Later among the works it cites.
Lipschitz constant estimation of neural networks via sparse polynomial optimization
Fabian Latorre, Paul Rolland, and Volkan Cevher · 2020
Later among the works it cites.
Efficient riemannian optimization on the stiefel manifold via the cayley transform
Jun Li, Li Fuxin, and Sinisa Todorovic · 2020
Later among the works it cites.
Deep isometric learning for visual recognition
Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik · 2020
Later among the works it cites.
A closer look at accuracy vs. robustness, 2020
Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri · 2020
Later among the works it cites.