2017

Discrete Event, Continuous Time RNNs

Mozer, Michael C., Kazakov, Denis, Lindsey, Robert V.

Understand

We investigate recurrent neural network architectures for event-sequence processing.

Event sequences, characterized by discrete observations stamped with continuous-valued times of occurrence, are challenging due to the potentially wide dynamic range of relevant time scales as well as interactions between time scales.
We describe four forms of inductive bias that should benefit architectures for event sequences: temporal locality, position and scale homogeneity, and scale interdependence.
We extend the popular gated recurrent unit (GRU) architecture to incorporate these biases via intrinsic temporal dynamics, obtaining a continuous-time GRU.

Built on

Signal detection theory and psychophysics
D. M. Green and J. A. Swets · 1966
Earlier work this paper cites.
Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
Kunihiko Fukushima · 1980
Earlier work this paper cites.
Early parallel processing in reading: A connectionist approach
Michael C. Mozer · 1987
Earlier work this paper cites.
Human memory: An adaptive perspective
J. R. Anderson and R. Milson · 1989
Earlier work this paper cites.
Phoneme recognition using time-delay neural networks
Alexander Waibel, Toshiyuki Hanazawa, Geofrey Hinton, Kiyohiro Shikano, and Kevin J. Lang · 1990
Earlier work this paper cites.
Induction of multiscale temporal structure
Michael C Mozer · 1992
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Y. Bengio, P. Simard, and P. Frasconi · 1994
Earlier work this paper cites.
Network intrusion detection
Biswanath Mukherjee, L Todd Heberlein, and Karl N Levitt · 1994
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
The vanishing gradient problem during learning recurrent neural nets and problem solutions
Sepp Hochreiter · 1998
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Learning to forget: Continual prediction with LSTM
Felix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins · 2000
Earlier work this paper cites.
Gradient flow in recurrent nets: the difficulty of learning long-term dependencies
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber · 2001
Earlier work this paper cites.
Forgetting to remember: The functional relationship of decay and interference
E. M. Altmann and W. D. Gray · 2002
Earlier work this paper cites.
Temporal convolution machines for sequence learning
A. J. Lockett and R. Miikkulainen · 2009
Earlier work this paper cites.
Predicting the optimal spacing of study: A multiscale context model of memory
Michael C Mozer, Harold Pashler, Nicholas Cepeda, Robert V Lindsey, and Ed Vul · 2009
Earlier work this paper cites.

Similar

Collaborative filtering with temporal dynamics
Yehuda Koren · 2010
Cited alongside, same era.
Convolutional learning of spatio-temporal features
Graham W. Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler · 2010
Cited alongside, same era.
Multiscale convolutional neural networks for vision–based classification of cells
Pierre Buyssens, Abderrahim Elmoataz, and Olivier Lézoray · 2013
Cited alongside, same era.
Exact simulation of Hawkes process with exponentially decaying intensity
Angelos Dassios and Hongbiao Zhao · 2013
Cited alongside, same era.
UCI machine learning repository, 2013
M. Lichman · 2013
Cited alongside, same era.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio · 2014
Cited alongside, same era.
A convolutional neural network for modelling sentences
Original
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom · 2014
Cited alongside, same era.
A hazard based approach to user return time prediction
Komal Kapoor, Mingxuan Sun, Jaideep Srivastava, and Tao Ye · 2014
Cited alongside, same era.
Time-decaying bandits for non-stationary systems
Junpei Komiyama and Tao Qin · 2014
Cited alongside, same era.
A clockwork RNN
Jan Koutník, Klaus Greff, Faustino Gomez, and Jürgen Schmidhuber · 2014
Cited alongside, same era.
Improving students’ long-term knowledge retention through personalized review
Robert V. Lindsey, Jeffery D. Shroyer, Harold Pashler, and Michael C. Mozer · 2014
Cited alongside, same era.
Multiscale time series prediction for intrusion detection
G Palanivel and K Duraiswamy · 2014
Cited alongside, same era.

Then

Session-based recommendations with recurrent neural networks
Original
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk · 2015
Later among the works it cites.
An empirical exploration of recurrent network architectures
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever · 2015
Later among the works it cites.
"I like to explore sometimes": Adapting to dynamic user novelty preferences
Komal Kapoor, Vikas Kumar, Loren Terveen, Joseph A. Konstan, and Paul Schrater · 2015
Later among the works it cites.
Doctor ai: Predicting clinical events via recurrent neural networks
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun · 2016
Later among the works it cites.
Hierarchical multiscale recurrent neural networks
Original
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio · 2016
Later among the works it cites.
Multi-scale convolutional neural networks for time series classification
Original
Zhicheng Cui, Wenlin Chen, and Yixin Chen · 2016
Later among the works it cites.
Recurrent coevolutionary latent feature processes for continuous-time recommendation
Hanjun Dai, Yichen Wang, Rakshit Trivedi, and Le Song · 2016
Later among the works it cites.
Recurrent marked temporal point processes: Embedding event history to vector
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song · 2016
Later among the works it cites.
Dna sequence classification by convolutional neural network
Ngoc Giang Nguyen, Vu Anh Tran, Duc Luu Ngo, Dau Phan, Favorisen Rosyking Lumbanraja, Mohammad Reza Faisal, Bahriddin Abapihi, Mamoru Kubo, Kenji Satou, et al · 2016
Later among the works it cites.
Multi-rate deep learning for temporal recommendation
Yang Song, Ali Mamdouh Elkahky, and Xiaodong He · 2016
Later among the works it cites.
Theano: A Python framework for fast computation of mathematical expressions
Original
Theano Development Team · 2016
Later among the works it cites.
Personal recommendation using deep recurrent neural networks in netease
Sai Wu, Weichao Ren, Chengchao Yu, Gang Chen, Dongxiang Zhang, and Jingbo Zhu · 2016
Later among the works it cites.
Convolutional neural network architectures for predicting dna–protein binding
H. Zeng, M. D. Edwards, G. Liu, and D. K. Gifford · 2016
Later among the works it cites.
A multiscale survival process for modeling human activity patterns
Tianyang Zhang, Peng Cui, Chaoming Song, Wenwu Zhu, and Shiqiang Yang · 2016
Later among the works it cites.
Recurrent recommender networks
Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J. Smola, and How Jing · 2017
Closest in time.
Multi-scale convolutional neural networks for crowd counting
Original
Lingke Zeng, Xiangmin Xu, Bolun Cai, Suo Qiu, and Tong Zhang · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…