2021

ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models

Yan, Tijin, Zhang, Hongwei, Zhou, Tong et al.

Understand

Multivariate time series prediction has attracted a lot of attention because of its wide applications such as intelligence transportation, AIOps.

Generative models have achieved impressive results in time series modeling because they can model data distribution and take noise into consideration.
However, many existing works can not be widely used because of the constraints of functional form of generative models or the sensitivity to hyperparameters.
In this paper, we propose ScoreGrad, a multivariate probabilistic time series forecasting framework based on continuous energy-based generative models.

Built on

G. E. Box and G. M. Jenkins, “Some recent advances in forecasting and control,” Journal of the Royal Statistical Society: Series C (Applied Statistics) , vol. 17, no. 2, pp. 91–109, 1968
1968
Earlier work this paper cites.
J. E. Matheson and R. L. Winkler, “Scoring rules for continuous probability distributions,” Management science , vol. 22, no. 10, pp. 1087–1096, 1976
1976
Earlier work this paper cites.
B. D. Anderson, “Reverse-time diffusion equation models,” Stochastic Processes and their Applications , vol. 12, no. 3, pp. 313–326, 1982
1982
Earlier work this paper cites.
S. Duane, A. D. Kennedy, B. J. Pendleton, and D. Roweth, “Hybrid monte carlo,” Physics letters B , vol. 195, no. 2, pp. 216–222, 1987
1987
Earlier work this paper cites.
L. Younes, “On the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates,” Stochastics: An International Journal of Probability and Stochastic Processes , vol. 65, no. 3-4, pp. 177–228, 1999
1999
Earlier work this paper cites.
R. Van der Weide, “Go-garch: a multivariate generalized orthogonal garch model,” Journal of Applied Econometrics , vol. 17, no. 5, pp. 549–564, 2002
2002
Earlier work this paper cites.
A. Hyvärinen and P. Dayan, “Estimation of non-normalized statistical models by score matching.” Journal of Machine Learning Research , vol. 6, no. 4, 2005
2005
Earlier work this paper cites.
H. Lütkepohl, New introduction to multiple time series analysis . Springer Science & Business Media, 2005
2005
Earlier work this paper cites.
R. Hyndman, A. B. Koehler, J. K. Ord, and R. D. Snyder, Forecasting with exponential smoothing: the state space approach . Springer Science & Business Media, 2008
2008
Earlier work this paper cites.
M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics . JMLR Workshop and Conference Proceedings, 2010, pp. 297–304
2010
Earlier work this paper cites.
P. Vincent, “A connection between score matching and denoising autoencoders,” Neural computation , vol. 23, no. 7, pp. 1661–1674, 2011
2011
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” arXiv preprint arXiv:1310.4546 , 2013
Original
2013
Earlier work this paper cites.
L. Dinh, D. Krueger, and Y. Bengio, “Nice: Non-linear independent components estimation,” arXiv preprint arXiv:1410.8516 , 2014
Original
2014
Earlier work this paper cites.
D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in International Conference on Machine Learning . PMLR, 2015, pp. 1530–1538
2015
Earlier work this paper cites.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International Conference on Machine Learning . PMLR, 2015, pp. 2256–2265
2015
Earlier work this paper cites.

Similar

A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499 , 2016
Original
2016
Cited alongside, same era.
R. Wen, K. Torkkola, B. Narayanaswamy, and D. Madeka, “A multi-horizon quantile recurrent forecaster,” arXiv preprint arXiv:1711.11053 , 2017
Original
2017
Cited alongside, same era.
S. Chang, Y. Zhang, W. Han, M. Yu, X. Guo, W. Tan, X. Cui, M. Witbrock, M. Hasegawa-Johnson, and T. S. Huang, “Dilated recurrent neural networks,” arXiv preprint arXiv:1710.02224 , 2017
Original
2017
Cited alongside, same era.
J. Kim, M. El-Khamy, and J. Lee, “Residual lstm: Design of a deep recurrent architecture for distant speech recognition,” arXiv preprint arXiv:1701.03360 , 2017
Original
2017
Cited alongside, same era.
Y. Qin, D. Song, H. Chen, W. Cheng, G. Jiang, and G. Cottrell, “A dual-stage attention-based recurrent neural network for time series prediction,” arXiv preprint arXiv:1704.02971 , 2017
Original
2017
Cited alongside, same era.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762 , 2017
Original
2017
Cited alongside, same era.
A. Jordan, F. Krüger, and S. Lerch, “Evaluating probabilistic forecasts with scoringrules,” arXiv preprint arXiv:1709.04743 , 2017
Original
2017
Cited alongside, same era.
M. Fraccaro, S. Kamronn, U. Paquet, and O. Winther, “A disentangled recognition and nonlinear dynamics model for unsupervised learning,” arXiv preprint arXiv:1710.05741 , 2017
Original
2017
Cited alongside, same era.
G. Papamakarios, T. Pavlakou, and I. Murray, “Masked autoregressive flow for density estimation,” arXiv preprint arXiv:1705.07057 , 2017
Original
2017
Cited alongside, same era.
I. Lana, J. Del Ser, M. Velez, and E. I. Vlahogianni, “Road traffic forecasting: Recent advances and new challenges,” IEEE Intelligent Transportation Systems Magazine , vol. 10, no. 2, pp. 93–109, 2018
2018
Cited alongside, same era.
G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long-and short-term temporal patterns with deep neural networks,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , 2018, pp. 95–104
2018
Cited alongside, same era.
A. Alexandrov, K. Benidis, M. Bohlke-Schneider, V. Flunkert, J. Gasthaus, T. Januschowski, D. C. Maddix, S. Rangapuram, D. Salinas, J. Schulz et al. , “Gluonts: Probabilistic time series models in python,” arXiv preprint arXiv:1906.05264 , 2019
Original
2019
Cited alongside, same era.

Then

L. Wenliang, D. Sutherland, H. Strathmann, and A. Gretton, “Learning deep kernels for exponential family densities,” in International Conference on Machine Learning . PMLR, 2019, pp. 6737–6746
2019
Later among the works it cites.
Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” arXiv preprint arXiv:1907.05600 , 2019
Original
2019
Later among the works it cites.
D. Salinas, M. Bohlke-Schneider, L. Callot, R. Medico, and J. Gasthaus, “High-dimensional multivariate forecasting with low-rank gaussian copula processes,” arXiv preprint arXiv:1910.03002 , 2019
Original
2019
Later among the works it cites.
R. Shen and Y. T. Lee, “The randomized midpoint method for log-concave sampling,” arXiv preprint arXiv:1909.05503 , 2019
Original
2019
Later among the works it cites.
D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski, “Deepar: Probabilistic forecasting with autoregressive recurrent networks,” International Journal of Forecasting , vol. 36, no. 3, pp. 1181–1191, 2020
2020
Later among the works it cites.
K. Rasul, A.-S. Sheikh, I. Schuster, U. Bergmann, and R. Vollgraf, “Multi-variate probabilistic time series forecasting via conditioned normalizing flows,” arXiv preprint arXiv:2002.06103 , 2020
Original
2020
Later among the works it cites.
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456 , 2020
Original
2020
Later among the works it cites.
E. de Bézenac, S. S. Rangapuram, K. Benidis, M. Bohlke-Schneider, R. Kurle, L. Stella, H. Hasson, P. Gallinari, and T. Januschowski, “Normalizing kalman filters for multivariate time series analysis,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Later among the works it cites.
Y. Deng, A. Bakhtin, M. Ott, A. Szlam, and M. Ranzato, “Residual energy-based models for text generation,” arXiv preprint arXiv:2004.11714 , 2020
Original
2020
Later among the works it cites.
Y. Song, S. Garg, J. Shi, and S. Ermon, “Sliced score matching: A scalable approach to density and score estimation,” in Uncertainty in Artificial Intelligence . PMLR, 2020, pp. 574–584
2020
Later among the works it cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” arXiv preprint arXiv:2006.11239 , 2020
Original
2020
Later among the works it cites.
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” arXiv preprint arXiv:2009.09761 , 2020
Original
2020
Later among the works it cites.
M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” arXiv preprint arXiv:2006.10739 , 2020
Original
2020
Later among the works it cites.
Y. Song and S. Ermon, “Improved techniques for training score-based generative models,” arXiv preprint arXiv:2006.09011 , 2020
Original
2020
Later among the works it cites.
J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 , 2020
Original
2020
Later among the works it cites.
K. Rasul, C. Seward, I. Schuster, and R. Vollgraf, “Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting,” arXiv preprint arXiv:2101.12072 , 2021
Original
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…