2020

Unsupervised Object Keypoint Learning using Local Spatial Predictability

Gopalakrishnan, Anand, van Steenkiste, Sjoerd, Schmidhuber, Jürgen

Understand

We propose PermaKey, a novel approach to representation learning based on object keypoints.

It leverages the predictability of local image regions from spatial neighborhoods to identify salient regions that correspond to object parts, which are then converted to keypoints.
Unlike prior approaches, it utilizes predictability to discover object keypoints, an intrinsic property of objects.
This ensures that it does not overly bias keypoints to focus on characteristics that are not unique to objects, such as movement, shape, colour etc.

Built on

Unsupervised learning
Horace B Barlow · 1989
Earlier work this paper cites.
Recursive distributed representations
Jordan B. Pollack · 1990
Earlier work this paper cites.
Saliency maps and attention selection in scale and spatial coordinates: An information theoretic approach
Martin Jagersand · 1995
Earlier work this paper cites.
Inductive learning in symbolic domains using structure-driven recurrent neural networks
Andreas Küchler and Christoph Goller · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
A model of saliency-based visual attention for rapid scene analysis
Laurent Itti, Christof Koch, and Ernst Niebur · 1998
Earlier work this paper cites.
Learning to forget: continual prediction with lstm
F.A. Gers · 1999
Earlier work this paper cites.
Saliency, scale and image description
Timor Kadir and Michael Brady · 2001
Earlier work this paper cites.
Saliency based on information maximization
Neil D. B. Bruce and John K. Tsotsos · 2005
Earlier work this paper cites.
Visual salience
L. Itti · 2007
Earlier work this paper cites.
The graph neural network model
F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini · 2009
Earlier work this paper cites.
What is an object?
B. Alexe, Thomas Deselaers, and V. Ferrari · 2010
Earlier work this paper cites.
Exploiting local and global patch rarities for saliency detection
Ali Borji and Laurent Itti · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Earlier work this paper cites.
Representation learning: A review and new perspectives
Yoshua Bengio, Aaron Courville, and Pascal Vincent · 2013
Earlier work this paper cites.
Crisp boundary detection using pointwise mutual information
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
Diederik P. Kingma and Max Welling · 2014
Earlier work this paper cites.
Deep recurrent q-learning for partially observable mdps
Matthew J. Hausknecht and P. Stone · 2015
Earlier work this paper cites.
Learning visual groups from co-occurrences in space and time
Original
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson · 2015
Earlier work this paper cites.

Similar

Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Cited alongside, same era.
Interaction networks for learning about objects, relations and physics
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al · 2016
Cited alongside, same era.
Attend, infer, repeat: Fast scene understanding with generative models
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al · 2016
Cited alongside, same era.
Deep reinforcement learning with double q-learning
H. V. Hasselt, A. Guez, and D. Silver · 2016
Cited alongside, same era.
A compositional object-based approach to learning physical dynamics
Michael Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum · 2017
Cited alongside, same era.
Neural expectation maximization
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber · 2017
Cited alongside, same era.
Building machines that learn and think like people
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Cited alongside, same era.
Relational inductive biases, deep learning, and graph networks
Original
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al · 2018
Cited alongside, same era.
Rainbow: Combining improvements in deep reinforcement learning
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver · 2018
Cited alongside, same era.
Unsupervised learning of object landmarks through conditional image generation
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi · 2018
Cited alongside, same era.

Then

Centernet: Keypoint triplets for object detection
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian · 2019
Later among the works it cites.
Multi-object representation learning with iterative variational inference
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner · 2019
Later among the works it cites.
Reasoning about physical interactions with object-centric models
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu · 2019
Later among the works it cites.
Unsupervised learning of object keypoints for perception and control
Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih · 2019
Later among the works it cites.
Unsupervised learning of object structure and dynamics from videos
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee · 2019
Later among the works it cites.
Towards interpretable reinforcement learning using attention augmented agents
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende · 2019
Later among the works it cites.
An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents
Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Ludwig Schubert, Marc G. Bellemare, Jeff Clune, and Joel Lehman · 2019
Later among the works it cites.
A perspective on objects and systematic generalization in model-based rl
Sjoerd van Steenkiste, Klaus Greff, and Jürgen Schmidhuber · 2019
Later among the works it cites.
Entity abstraction in visual model-based reinforcement learning
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B Tenenbaum, and Sergey Levine · 2019
Later among the works it cites.
Deep reinforcement learning with relational inductive biases
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia · 2019
Later among the works it cites.
Information-bottleneck approach to salient region discovery
Andrey Zhmoginov, Ian Fischer, and Mark Sandler · 2019
Later among the works it cites.
Objects as points
Original
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl · 2019
Later among the works it cites.
On the binding problem in artificial neural networks
Original
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber · 2020
Closest in time.
Model based reinforcement learning for atari
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski · 2020
Closest in time.
Contrastive learning of structured world models
Thomas Kipf, Elise van der Pol, and Max Welling · 2020
Closest in time.
SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn · 2020
Closest in time.
Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio · 2020
Closest in time.
Hierarchical relational inference
Aleksandar Stanić, Sjoerd van Steenkiste, and Jürgen Schmidhuber · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…