2018

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

Lee, Juho, Lee, Yoonho, Kim, Jungtaek et al.

Understand

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances.

Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant.
We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set.
The model consists of an encoder and a decoder, both of which rely on attention mechanisms.

Built on

Evolutionary Principles in Self-Referential Learning
Schmidhuber, J · 1987
Earlier work this paper cites.
Solving the multiple instance problem with axis-parallel rectangles
Dietterich, T. G., Lathrop Richard, H., and Lozano-Pérez, T · 1997
Earlier work this paper cites.
A framework for multiple-instance learning
Maron, O. and Lozano-Pérez, T · 1998
Earlier work this paper cites.
Learning to Learn
Thrun, S. and Pratt, L · 1998
Earlier work this paper cites.
Spectral grouping using the Nyström method
Fowlkes, C., Belongie, S., Chung, F., and Malik, J · 2004
Earlier work this paper cites.
Sparse Gaussian processes using pseudo-inputs
Snelson, E. and Ghahramani, Z · 2005
Earlier work this paper cites.
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Learning from distributions via support measure machines
Muandet, K., Fukumizu, K., Dinuzzo, F., and Schölkopf, B · 2012
Earlier work this paper cites.
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-r., and Hinton, G. E · 2013
Earlier work this paper cites.
Distribution to distribution regression
Oliva, J., Póczos, B., and Schneider, J · 2013
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Earlier work this paper cites.
ShapeNet: An information-rich 3D model repository
Original
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., and Yu, F · 2015
Earlier work this paper cites.

Similar

Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B · 2015
Cited alongside, same era.
DeepPano: deep panoramic representation for 3-D shape recognition
Shi, B., Bai, S., Zhou, Z., and Bai, X · 2015
Cited alongside, same era.
Multi-view convolutional neural networks for 3D shape recognition
Su, H., Maji, S., Kalogerakis, E., and Learned-Miller, E · 2015
Cited alongside, same era.
3D ShapeNets: a deep representation for volumetric shapes
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J · 2015
Cited alongside, same era.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Cited alongside, same era.
Order matters: sequence to sequence for sets
Vinyals, O., Bengio, S., and Kudlur, M · 2016
Cited alongside, same era.
PointNet: Deep learning on point sets for 3D classification and segmentation
Charles, R. Q., Su, H., Kaichun, M., and Guibas, L. J · 2017
Cited alongside, same era.
Towards a neural statistician
Edwards, H. and Storkey, A · 2017
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Finn, C., Abbeel, P., and Levine, S · 2017
Cited alongside, same era.
Discovering causal signals in images
Lopez-Paz, D., Nishihara, R., Chintala, S., Schölkopf, B., and Bottou, L · 2017
Cited alongside, same era.
Neural episodic control
Pritzel, A., Uria, B., Srinivasan, S., Puigdomenech, A., Vinyals, O., Hassabis, D., Wierstra, D., and Blundell, C · 2017
Cited alongside, same era.
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barret, D. G. T., Malinowski, M., Pascanu, R., and Battaglia, P · 2017
Cited alongside, same era.

Then

Prototypical networks for few-shot learning
Snell, J., Swersky, K., and Zemel, R · 2017
Later among the works it cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Later among the works it cites.
Deep sets
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J · 2017
Later among the works it cites.
Conditional neural processes
Garnelo, M., Rosenbaum, D., Maddison, C. J., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D. J., and Eslami, S. M. A · 2018
Closest in time.
Attention-based deep multiple instance learning
Ilse, M., Tomczak, J. M., and Welling, M · 2018
Closest in time.
Gradient-based meta-learning with learned layerwise metric and subspace
Lee, Y. and Choi, S · 2018
Closest in time.
Attend and interact: higher-order object interactions for video understanding
Ma, C.-Y., Kadav, A., Melvin, I., Kira, Z., AlRegib, G., and Peter Graf, H · 2018
Closest in time.
A simple neural attentive meta-learner
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P · 2018
Closest in time.
Attentional aggregation of deep feature sets for multi-view 3D reconstruction
Original
Yang, B., Wang, S., Markham, A., and Trigoni, N · 2018
Closest in time.
Probabilistic symmetry and invariant neural networks
Original
Bloem-Reddy, B. and Teh, Y.-W · 2019
Closest in time.
Attentive neural processes
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W · 2019
Closest in time.
On the limitations of representing functions on sets
Original
Wagstaff, E., Fuchs, F. B., Engelcke, M., Posner, I., and Osborne, M · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…