2018

Hyperspherical Variational Auto-Encoders

Davidson, Tim R., Falorsi, Luca, De Cao, Nicola et al.

Understand

The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models.

But although the default choice of a Gaussian distribution for both the prior and posterior represents a mathematically convenient distribution often leading to competitive results, we show that this parameterization fails to model data with a latent hyperspherical structure.
To address this issue we propose using a von Mises-Fisher (vMF) distribution instead, leading to a hyperspherical latent space.
Through a series of experiments we show how such a hyperspherical VAE, or $\mathcal{S}$-VAE, is more suitable for capturing data with a hyperspherical latent structure, while outperforming a normal, $\mathcal{N}$-VAE, in low dimensions on other data types.

Built on

Statistics of directional data
Mardia, K. V. (1975) · 1975
Earlier work this paper cites.
Computer generation of distributions on the m-sphere
Ulrich, G. (1984) · 1984
Earlier work this paper cites.
Statistical analysis of spherical data
Fisher, N. I., Lewis, T., and Embleton, B. J. (1987) · 1987
Earlier work this paper cites.
Latent dirichlet allocation
Blei, D. M., Ng, A. Y., and Jordan, M. I. (2001) · 2001
Earlier work this paper cites.
Generative model-based clustering of directional data
Banerjee, A., Dhillon, I., Ghosh, J., and Sra, S. (2003) · 2003
Earlier work this paper cites.
Clustering on the unit hypersphere using von mises-fisher distributions
Banerjee, A., Dhillon, I. S., Ghosh, J., and Sra, S. (2005) · 2005
Earlier work this paper cites.
Topic models over text streams: A study of batch and online unsupervised learning
Banerjee, A. and Basu, S. (2007) · 2007
Earlier work this paper cites.
On the quantitative analysis of deep belief networks
Salakhutdinov, R. and Murray, I. (2008) · 2008
Earlier work this paper cites.
Collective classification in network data
Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., and Eliassi-Rad, T. (2008) · 2008
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y. (2010) · 2010
Earlier work this paper cites.
Spherical topic models
Reisinger, J., Waters, A., Silverthorn, B., and Mooney, R. J. (2010) · 2010
Earlier work this paper cites.
Von mises-fisher clustering models
Gopal, S. and Yang, Y. (2014) · 2014
Earlier work this paper cites.

Similar

Semi-supervised learning with deep generative models
Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M. (2014) · 2014
Cited alongside, same era.
Auto-encoding variational bayes
Kingma, D. P. and Welling, M. (2014a) · 2014
Cited alongside, same era.
Efficient gradient-based inference through transformations between bayes nets and neural nets
Kingma, D. P. and Welling, M. (2014b) · 2014
Cited alongside, same era.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014) · 2014
Cited alongside, same era.
Spherical and hyperbolic embeddings of data
Wilson, R. C., Hancock, E. R., Pekalska, E., and Duin, R. P. (2014) · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J. (2015) · 2015
Cited alongside, same era.
Variational inference with normalizing flows
Rezende, D. J. and Mohamed, S. (2015) · 2015
Cited alongside, same era.
Generating sentences from a continuous space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. (2016) · 2016
Cited alongside, same era.
Importance weighted autoencoders
Burda, Y., Grosse, R. B., and Salakhutdinov, R. (2016) · 2016
Cited alongside, same era.
Deep unsupervised clustering with gaussian mixture variational autoencoders
Original
Dilokthanakul, N., Mediano, P. A. M., Garnelo, M., Lee, M. C. H., Salimbeni, H., Arulkumaran, K., and Shanahan, M. (2016) · 2016
Cited alongside, same era.
Variational Graph Auto-Encoders
Kipf, T. N. and Welling, M. (2016) · 2016
Cited alongside, same era.

Then

Structure-from-motion in spherical video using the von mises-fisher distribution
Guan, H. and Smith, W. A. (2017) · 2017
Later among the works it cites.
von mises-fisher mixture model-based deep learning: Application to face verification
Original
Hasnat, M., Bohné, J., Milgram, J., Gentric, S., Chen, L., et al. (2017) · 2017
Later among the works it cites.
Deep hyperspherical learning
Liu, W., Zhang, Y., Li, X., Liu, Z., Dai, B., Zhao, T., and Song, L. (2017) · 2017
Later among the works it cites.
Reparameterization gradients through acceptance-rejection sampling algorithms
Naesseth, C. A., Ruiz, F. J. R., Linderman, S. W., and Blei, D. M. (2017) · 2017
Later among the works it cites.
Stick-breaking variational autoencoders
Nalisnick, E. T. and Smyth, P. (2017) · 2017
Later among the works it cites.
Poincaré embeddings for learning hierarchical representations
Nickel, M. and Kiela, D. (2017) · 2017
Later among the works it cites.
Clustering and unsupervised anomaly detection with l2 normalized deep auto-encoder representations
Original
Aytekin, C., Ni, X., Cricri, F., and Aksu, E. (2018) · 2018
Closest in time.
Spherical cnns
Cohen, T. S., Geiger, M., Köhler, J., and Welling, M. (2018) · 2018
Closest in time.
Generating sentences by editing prototypes
Guu, K., Hashimoto, T. B., Oren, Y., and Liang, P. (2018) · 2018
Closest in time.
Riemannian stein variational gradient descent for bayesian inference
Liu, C. and Zhu, J. (2018) · 2018
Closest in time.
VAE with a vampprior
Tomczak, J. M. and Welling, M. (2018) · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…