2019

Recurrent Attentive Neural Process for Sequential Data

Qin, Shenghao, Zhu, Jiacheng, Qin, Jimmy et al.

Understand

Neural processes (NPs) learn stochastic processes and predict the distribution of target output adaptively conditioned on a context set of observed input-output pairs.

Furthermore, Attentive Neural Process (ANP) improved the prediction accuracy of NPs by incorporating attention mechanism among contexts and targets.
In a number of real-world applications such as robotics, finance, speech, and biology, it is critical to learn the temporal order and recurrent structure from sequential data.
However, the capability of NPs capturing these properties is limited due to its permutation invariance instinct.

Built on

Deep kernel learning
Original
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing · 1910
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Gaussian process dynamical models
Jack Wang, Aaron Hertzmann, and David J Fleet · 2006
Earlier work this paper cites.
Vehicle Dynamics and Control
R. Rajamani · 2011
Earlier work this paper cites.
Gaussian process regression networks
Original
Andrew Gordon Wilson, David A Knowles, and Zoubin Ghahramani · 2011
Earlier work this paper cites.
Subspace identification for linear systems: Theory—Implementation—Applications
Peter Van Overschee and BL De Moor · 2012
Earlier work this paper cites.
Speech recognition with deep recurrent neural networks
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Earlier work this paper cites.
A recurrent latent variable model for sequential data
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio · 2015
Earlier work this paper cites.
Recurrent gaussian processes
Original
César Lincoln C Mattos, Zhenwen Dai, Andreas Damianou, Jeremy Forth, Guilherme A Barreto, and Neil D Lawrence · 2015
Earlier work this paper cites.

Similar

Thoughts on massively scalable gaussian processes
Original
Andrew Gordon Wilson, Christoph Dann, and Hannes Nickisch · 2015
Cited alongside, same era.
Learning scalable deep kernels with recurrent structure
Original
Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, and Eric P Xing · 2016
Cited alongside, same era.
Deep variational bayes filters: Unsupervised learning of state space models from raw data
Original
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt · 2016
Cited alongside, same era.
Deep kalman filters
Rahul G Krishnan, Uri Shalit, and David Sontag · 2016
Cited alongside, same era.
Identification of gaussian process state space models
Stefanos Eleftheriadis, Tom Nicholson, Marc Deisenroth, and James Hensman · 2017
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.
A disentangled recognition and nonlinear dynamics model for unsupervised learning
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther · 2017
Cited alongside, same era.
Generative temporal models with memory
Original
Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo J Rezende, David Amos, and Timothy Lillicrap · 2017
Cited alongside, same era.
Combining lstm and latent topic modeling for mortality prediction
Original
Yohan Jo, Lisa Lee, and Shruti Palaskar · 2017
Cited alongside, same era.
Structured inference networks for nonlinear state space models
Rahul G Krishnan, Uri Shalit, and David Sontag · 2017
Cited alongside, same era.
Conditional neural processes
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami
Cited in the paper.
Neural processes
Original
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh
Cited in the paper.

Then

The statistical recurrent unit
Junier B Oliva, Barnabás Póczos, and Jeff Schneider · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Later among the works it cites.
Structural recurrent neural network (srnn) for group activity analysis
Sovan Biswas and Juergen Gall · 2018
Later among the works it cites.
Probabilistic recurrent state-space models
Original
Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, Stefan Schaal, Marc Toussaint, and Sebastian Trimpe · 2018
Later among the works it cites.
Neural scene representation and rendering
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al · 2018
Later among the works it cites.
Generative temporal models with spatial memory for partially observed environments
Original
Marco Fraccaro, Danilo Jimenez Rezende, Yori Zwols, Alexander Pritzel, SM Eslami, and Fabio Viola · 2018
Later among the works it cites.
Consistent generative query networks
Original
Ananya Kumar, SM Eslami, Danilo J Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, and Murray Shanahan · 2018
Later among the works it cites.
Attentive neural processes
Original
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh · 2019
Closest in time.
The functional neural process, 2019
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling · 2019
Closest in time.
Sequential neural processes
Original
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…