What if Neural Networks had SVDs?

The WY Representation for Products of Householder Matrices

Christian Bischof and Charles Van Loan · 1987

Earlier work this paper cites.

Constructive Ways for Generating (Generalized) Real Orthogonal Matrices as Products of (Generalized) Symmetries

Frank Uhlig · 2001

Earlier work this paper cites.

Linear Algebra and its Applications

Gilbert Strang · 2006

Earlier work this paper cites.

Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition

Jian Xue, Jinyu Li, and Yifan Gong · 2013

Earlier work this paper cites.

NICE: Non-Linear Independent Components Estimation

Laurent Dinh, David Krueger, and Yoshua Bengio · 2015

Earlier work this paper cites.

Improving Variational Auto-Encoders using Householder Flow

Jakub M Tomczak and Max Welling · 2016

Earlier work this paper cites.

The Reversible Residual Network: Backpropagation Without Storing Activations

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse · 2017

Cited alongside, same era.

Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections

Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey · 2017

Cited alongside, same era.

Glow: Generative Flow with Invertible 1x1 Convolutions

Diederik P Kingma and Prafulla Dhariwal · 2018

Cited alongside, same era.

Spectral Normalization for Generative Adversarial Networks

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida · 2018

Cited alongside, same era.

Sylvester Normalizing Flows for Variational Inference

Rianne van den Berg, Leonard Hasenclever, Jakub Tomczak, and Max Welling · 2018

Cited alongside, same era.

Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization

Trivializations for Gradient-Based Optimization on Manifolds

Mario Lezcano Casado · 2019

Later among the works it cites.

Improving Normalizing Flows via Better Orthogonal Parameterizations

Adam Golinski, Mario Lezcano-Casado, and Tom Rainforth · 2019

Later among the works it cites.

Emerging Convolutions for Generative Normalizing Flows

Emiel Hoogeboom, Rianne van den Berg, and Max Welling · 2019

Later among the works it cites.

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

Mario Lezcano-Casado and David Martínez-Rubio · 2019

Later among the works it cites.

Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform

Jun Li, Fuxin Li, and Sinisa Todorovic · 2020

Closest in time.