2020

SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

Fuchs, Fabian B., Worrall, Daniel E., Fischer, Volker et al.

Understand

We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations.

Equivariance is important to ensure stable and predictable performance in the presence of nuisance transformations of the data input.
A positive corollary of equivariance is increased weight-tying within the model.
The SE(3)-Transformer leverages the benefits of self-attention to operate on large point clouds and graphs with varying number of points, while guaranteeing SE(3)-equivariance for robustness.

Built on

Engineering applications of noncommutative harmonic analysis: with emphasis on rotation and motion groups
Gregory S Chirikjian, Alexander B Kyatkin, and AC Buckingham · 2001
Earlier work this paper cites.
Quantum chemistry structures and properties of 134 kilo molecules
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole von Lilienfeld · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
3d shapenets: A deep representation for volumetric shapes
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao · 2015
Earlier work this paper cites.
Layer normalization
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton · 2016
Earlier work this paper cites.
Steerable cnns
Taco S. Cohen and Max Welling · 2017
Earlier work this paper cites.
Wavelet scattering regression of quantum chemical energies
Matthew J. Hirn, Stéphane Mallat, and Nicolas Poilvert · 2017
Earlier work this paper cites.
Vain: Attentional multi-agent predictive modeling
Yedid Hoshen · 2017
Earlier work this paper cites.
A structured self-attentive sentence embedding
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio · 2017
Earlier work this paper cites.
Schnet: A continuous-filter convolutional neural network for modeling quantum interactions
K. T. Schütt, P.-J. Kindermans, H. E. Sauceda, S. Chmiela1, A. Tkatchenko, and K.-R. Müller · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2017
Earlier work this paper cites.
Harmonic networks: Deep translation and rotation equivariance
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow · 2017
Earlier work this paper cites.
Deep Sets
Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander Smola · 2017
Earlier work this paper cites.
3dmfv: Three-dimensional point cloud classification in realtime using convolutional neural networks
Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer · 2018
Earlier work this paper cites.

Similar

Spherical cnns
Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling · 2018
Cited alongside, same era.
Neural relational inference for interacting systems
Thomas N. Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard S. Zemel · 2018
Cited alongside, same era.
N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials
Risi Kondor · 2018
Cited alongside, same era.
Self-attention with relative position representations
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani · 2018
Cited alongside, same era.
Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley · 2018
Cited alongside, same era.
Relational neural expectation maximization: Unsupervised discovery of objects and their interactions
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber · 2018
Cited alongside, same era.
Graph attention networks
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio · 2018
Cited alongside, same era.
3d steerable cnns: Learning rotationally equivariant features in volumetric data
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen · 2018
Cited alongside, same era.
3d g-cnns for pulmonary nodule detection
Marysia Winkels and Taco S. Cohen · 2018
Cited alongside, same era.
Attentional shapecontextnet for point cloud recognition
Saining Xie, Sainan Liu, and Zeyu Chen Zhuowen Tu · 2018
Cited alongside, same era.
Spidercnn: Deep learning on point sets with parameterized convolutional filters
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao · 2018
Cited alongside, same era.
Cormorant: Covariant molecular neural networks
Brandon Anderson, Truong Son Hy, and Risi Kondor · 2019
Cited alongside, same era.

Then

Spherical fractal convolutional neural networks for point cloud recognition
Yongming Rao, Jiwen Lu, and Jie Zhou · 2019
Later among the works it cites.
Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung · 2019
Later among the works it cites.
On the limitations of representing functions on sets
Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael A. Osborne · 2019
Later among the works it cites.
Dynamic graph cnn for learning on point clouds
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon · 2019
Later among the works it cites.
General E(2)-Equivariant Steerable CNNs
Maurice Weiler and Gabriele Cesa · 2019
Later among the works it cites.
Deep scale-spaces: Equivariance over scale
Daniel E. Worrall and Max Welling · 2019
Later among the works it cites.
Modeling point clouds with self-attention and gumbel subset sampling
Jiancheng Yang, Qiang Zhang, and Bingbing Ni · 2019
Later among the works it cites.
Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution
Yang You, Yujing Lou, Qi Liu, Yu-Wing Tai, Lizhuang Ma, Cewu Lu, and Weiming Wang · 2019
Later among the works it cites.
Rotation invariant convolutions for 3d point clouds deep learning
Zhiyuan Zhang, Binh-Son Hua, David W. Rosen, and Sai-Kit Yeung · 2019
Later among the works it cites.
Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Wilson · 2020
Closest in time.
End-to-end recurrent multi-object tracking and prediction with relational reasoning
Fabian B. Fuchs, Adam R. Kosiorek, Li Sun, Oiwi Parker Jones, and Ingmar Posner · 2020
Closest in time.
Neural message passing for quantum chemistry
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, and Oriol Vinyals George E. Dahl · 2020
Closest in time.
Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds
Yongming Rao, Jiwen Lu, and Jie Zhou · 2020
Closest in time.
Attentive group equivariant convolutional networks
David W. Romero, Erik J. Bekkers, Jakub M. Tomczak, and Mark Hoogendoorn · 2020
Closest in time.
Scale-equivariant steerable networks
Ivan Sosnovik, Michał Szmaja, and Arnold Smeulders · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…