2019

On the Limitations of Representing Functions on Sets

Wagstaff, Edward, Fuchs, Fabian B., Engelcke, Martin et al.

Understand

Recent work on the representation of functions on sets has considered the use of summation in a latent space to enforce permutation invariance.

In particular, it has been conjectured that the dimension of this latent space may remain fixed as the cardinality of the sets under consideration increases.
However, we demonstrate that the analysis leading to this conjecture requires mappings which are highly discontinuous and argue that this is only of limited practical use.
Motivated by this observation, we prove that an implementation of this model via continuous mappings (as provided by e.g.

Built on

Approximation by superpositions of a sigmoidal function
Cybenko, G · 1989
Earlier work this paper cites.
Gaussian Processes for Machine Learning
Rasmussen, C. E. and Williams, C. K. I · 2006
Earlier work this paper cites.
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., and Le, Q · 2014
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
Order matters: Sequence to sequence for sets
Original
Vinyals, O., Bengio, S., and Kudlur, M · 2015
Earlier work this paper cites.
Density estimation using Real NVP
Original
Dinh, L., Sohl-Dickstein, J., and Bengio, S · 2016
Earlier work this paper cites.
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
Eslami, S. M. A., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Kavukcuoglu, K., and Hinton, G. E · 2016
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Deep Learning with Sets and Point Clouds
Original
Ravanbakhsh, S., Schneider, J., and Poczos, B · 2016
Earlier work this paper cites.

Similar

Value-Decomposition Networks For Cooperative Multi-Agent Learning
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al · 2017
Cited alongside, same era.
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmer, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., and Polosukhin, I · 2017
Cited alongside, same era.
Deep Sets
Zaheer, M., Kottur, S., Ravanbhakhsh, S., Póczos, B., Salakhutdinov, R., and Smola, A · 2017
Cited alongside, same era.
Learning Representations and Generative Models for 3D Point Clouds
Achlioptas, P., Diamanti, O., Mitliagkas, I., and Guibas, L · 2018
Cited alongside, same era.
Neural scene representation and rendering
Eslami, S. M. A., Rezende, D. J., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., Reichert, D. P., Buesing, L., Weber, T., Vinyals, O., Rosenbaum, D., Rabinowitz, N. C., King, H., Hillier, C., Botvinick, M. M., Wierstra, D., Kavukcuoglu, K., and Hassabis, D · 2018
Cited alongside, same era.
Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
Original
Herzig, R., Raboh, M., Chechik, G., Berant, J., and Globerson, A · 2018
Cited alongside, same era.
BRUNO: A Deep Recurrent Model for Exchangeable Data
Korshunova, I., Degrave, J., Huszár, F., Gal, Y., Gretton, A., and Dambre, J · 2018
Cited alongside, same era.
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
Kosiorek, A. R., Kim, H., Posner, I., and Teh, Y. W · 2018
Cited alongside, same era.
Conditional Neural Processes
Garnelo, M., Rosenbaum, D., Maddison, C. J., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D. J., and Eslami, S. M. A
Cited in the paper.
Neural Processes
Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S. M. A., and Teh, Y. W
Cited in the paper.
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Qi, C. R., Su, H., Mo, K., and Guibas, L
Cited in the paper.
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
Qi, C. R., Yi, L., Su, H., and Guibas, L. J
Cited in the paper.

Then

Consistent Generative Query Networks
Original
Kumar, A., Eslami, S., Rezende, D. J., Garnelo, M., Viola, F., Lockhart, E., and Shanahan, M · 2018
Later among the works it cites.
Set Transformer
Original
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., and Teh, Y. W · 2018
Later among the works it cites.
Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
Original
Murphy, R. L., Srinivasan, B., Rao, V., and Ribeiro, B · 2018
Later among the works it cites.
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., and Guibas, L. J · 2018
Later among the works it cites.
Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
Original
Yang, B., Wang, S., Markham, A., and Trigoni, N · 2018
Later among the works it cites.
GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud
Original
Yi, L., Zhao, W., Wang, H., Sung, M., and Guibas, L · 2018
Later among the works it cites.
Probabilistic Symmetry and Invariant Neural Networks
Original
Bloem-Reddy, B. and Teh, Y. W · 2019
Closest in time.
Attentive Neural Processes
Original
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…