2017

Non-local Neural Networks

Wang, Xiaolong, Girshick, Ross, Gupta, Abhinav et al.

Understand

Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time.

In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies.
Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions.
This building block can be plugged into many computer vision architectures.

Built on

Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition
K. Fukushima and S. Miyake · 1982
Earlier work this paper cites.
Learning representations by back-propagating errors
D. E. Rumelhart, G. E. Hinton, and R. J. Williams · 1986
Earlier work this paper cites.
Backpropagation applied to handwritten zip code recognition
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel · 1989
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Bilateral filtering for gray and color images
C. Tomasi and R. Manduchi · 1998
Earlier work this paper cites.
Texture synthesis by non-parametric sampling
A. A. Efros and T. K. Leung · 1999
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
J. Lafferty, A. McCallum, and F. C. Pereira · 2001
Earlier work this paper cites.
A non-local algorithm for image denoising
A. Buades, B. Coll, and J.-M. Morel · 2005
Earlier work this paper cites.
Image denoising by sparse 3-d transform-domain collaborative filtering
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian · 2007
Earlier work this paper cites.
Patchmatch: A randomized correspondence algorithm for structural image editing
C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman · 2009
Earlier work this paper cites.
Super-resolution from a single image
D. Glasner, S. Bagon, and M. Irani · 2009
Earlier work this paper cites.
The graph neural network model
F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini · 2009
Earlier work this paper cites.
3d convolutional neural networks for human action recognition
S. Ji, W. Xu, M. Yang, and K. Yu · 2010
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
V. Nair and G. E. Hinton · 2010
Earlier work this paper cites.
Efficient inference in fully connected crfs with gaussian edge potentials
P. Krähenbühl and V. Koltun · 2011
Earlier work this paper cites.
Image denoising: Can plain neural networks compete with BM3D?
H. C. Burger, C. J. Schuler, and S. Harmeling · 2012
Earlier work this paper cites.
Image denoising with multi-layer perceptrons, part 2: training trade-offs and analysis of their mechanisms
Original
H. C. Burger, C. J. Schuler, and S. Harmeling · 2012
Earlier work this paper cites.
Improving neural networks by preventing co-adaptation of feature detectors
Original
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov · 2012
Earlier work this paper cites.
Action recognition with improved trajectories
H. Wang and C. Schmid · 2013
Earlier work this paper cites.

Similar

Semantic image segmentation with deep convolutional nets and fully connected CRFs
Original
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2014
Cited alongside, same era.
Microsoft COCO: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
Two-stream convolutional networks for action recognition in videos
K. Simonyan and A. Zisserman · 2014
Cited alongside, same era.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Cited alongside, same era.
ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Cited alongside, same era.
Fully connected deep structured networks
Original
A. G. Schwing and R. Urtasun · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Cited alongside, same era.
Learning spatiotemporal features with 3d convolutional networks
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri · 2015
Cited alongside, same era.
Action recognition with trajectory-pooled deep-convolutional descriptors
L. Wang, Y. Qiao, and X. Tang · 2015
Cited alongside, same era.
Beyond short snippets: Deep networks for video classification
J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici · 2015
Cited alongside, same era.

Then

Quo vadis, action recognition? a new model and the kinetics dataset
J. Carreira and A. Zisserman · 2017
Closest in time.
Dense and low-rank Gaussian CRFs using deep embeddings
S. Chandra, N. Usunier, and I. Kokkinos · 2017
Closest in time.
Convolutional sequence to sequence learning
J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin · 2017
Closest in time.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Original
P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He · 2017
Closest in time.
Segmentation-aware convolutional networks using local attention masks
A. Harley, K. Derpanis, and I. Kokkinos · 2017
Closest in time.
Mask R-CNN
K. He, G. Gkioxari, P. Dollár, and R. Girshick · 2017
Closest in time.
Multi-agent predictive modeling with attentional commnets
Y. Hoshen · 2017
Closest in time.
The kinetics human action video dataset
Original
W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al · 2017
Closest in time.
Non-local color image denoising with convolutional neural networks
S. Lefkimmiatis · 2017
Closest in time.
Feature pyramid networks for object detection
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie · 2017
Closest in time.
Learning affinity via spatial propagation networks
S. Liu, S. De Mello, J. Gu, G. Zhong, M.-H. Yang, and J. Kautz · 2017
Closest in time.
Faster R-CNN: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2017
Closest in time.
A simple neural network module for relational reasoning
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap · 2017
Closest in time.
Asynchronous temporal fields for action recognition
G. A. Sigurdsson, S. Divvala, A. Farhadi, and A. Gupta · 2017
Closest in time.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin · 2017
Closest in time.
Visual interaction networks
N. Watters, A. Tacchetti, T. Weber, R. Pascanu, P. Battaglia, and D. Zoran · 2017
Closest in time.
Aggregated residual transformations for deep neural networks
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2017
Closest in time.
The Microsoft 2016 Conversational Speech Recognition System
W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…