Foundation Models for Time Series Analysis: A Tutorial and Survey

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901

Earlier work this paper cites.

Attention is all you need. In Advances in Neural Information Processing Systems 30 . 5998–6008

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT . 4171–4186

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Pre-Trained Bidirectional Temporal Representation for Crowd Flows Prediction in Regular Region

Wenying Duan, Liu Jiang, Ning Wang, and Hong Rao. 2019 · 2019

Earlier work this paper cites.

Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019

Earlier work this paper cites.

A Simple Framework for Contrastive Learning of Visual Representations. In ICML , Vol. 119. 1597–1607

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020 · 2020

Earlier work this paper cites.

Trembr: Exploring Road Networks for Trajectory Representation Learning

Tao-Yang Fu and Wang-Chien Lee. 2020 · 2020

Earlier work this paper cites.

On the opportunities and risks of foundation models

Original

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021

Earlier work this paper cites.

Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM international conference on information & knowledge management . 402–411

Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. 2021 · 2021

Earlier work this paper cites.

Recurrent neural networks for time series forecasting: Current status and future directions

Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. 2021 · 2021

Earlier work this paper cites.

Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. 2021 · 2021

Earlier work this paper cites.

Learning transferable visual models from natural language supervision. In International conference on machine learning . PMLR, 8748–8763

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021

Earlier work this paper cites.

Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning . PMLR, 8857–8868

Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. 2021 · 2021

Earlier work this paper cites.

Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. In International Conference on Learning Representations

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2021 · 2021

Earlier work this paper cites.

ScoreGrad: Multivariate probabilistic time series forecasting with continuous energy-based generative models

Original

Tijin Yan, Hongwei Zhang, Tong Zhou, Yufeng Zhan, and Yuanqing Xia. 2021 · 2021

Earlier work this paper cites.

Voice2series: Reprogramming acoustic models for time series classification. In International conference on machine learning . PMLR, 11808–11819

Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen. 2021 · 2021

Earlier work this paper cites.

CloudRCA: A root cause analysis framework for cloud computing platforms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 4373–4382

Yingying Zhang, Zhengxiong Guan, Huajie Qian, Leili Xu, Hengbo Liu, Qingsong Wen, Liang Sun, Junwei Jiang, Lunting Fan, and Min Ke. 2021 · 2021

Earlier work this paper cites.

Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , Vol. 35. 11106–11115

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021 · 2021

Earlier work this paper cites.

Modeling temporal data as continuous functions with process diffusion

Original

Marin Biloš, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Günnemann. 2022 · 2022

Earlier work this paper cites.

Self-supervised Trajectory Representation Learning with Temporal Regularities and Travel Semantics

Original

Jiawei Jiang, Dayan Pan, Houxing Ren, Xiaohan Jiang, Chao Li, and Jingyuan Wang. 2022 · 2022

Earlier work this paper cites.

Generative time series forecasting with diffusion, denoise, and disentanglement

Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. 2022a · 2022

Earlier work this paper cites.

When do contrastive learning signals help spatio-temporal graph forecasting?. In Proceedings of the 30th International Conference on Advances in Geographic Information Systems . 1–12

Xu Liu, Yuxuan Liang, Chao Huang, Yu Zheng, Bryan Hooi, and Roger Zimmermann. 2022 · 2022

Earlier work this paper cites.

A time series is worth 64 words: Long-term forecasting with transformers

Original

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022 · 2022

Earlier work this paper cites.

Contrastive learning for unsupervised domain adaptation of time series

Original

Yilmazcan Ozyurt, Stefan Feuerriegel, and Ce Zhang. 2022 · 2022

Earlier work this paper cites.

Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators

Original

Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al · 2022

Earlier work this paper cites.

High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022 · 2022

Earlier work this paper cites.

Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 1567–1577

Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. 2022 · 2022

Earlier work this paper cites.

MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting. In 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI) . IEEE, 982–989

Peiwang Tang and Xianchao Zhang. 2022 · 2022

Earlier work this paper cites.

Robust time series analysis and applications: An industrial perspective. In 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 4836–4837

Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. 2022 · 2022

Earlier work this paper cites.

Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2022 · 2022

Earlier work this paper cites.

PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting

Original

Hao Xue and Flora D Salim. 2022 · 2022

Earlier work this paper cites.

Leveraging language foundation models for human mobility forecasting. In the 30th International Conference on Advances in Geographic Information Systems . 1–9

Hao Xue, Bhanu Prakash Voutharoja, and Flora D Salim. 2022 · 2022

Earlier work this paper cites.

A large language model for electronic health records

Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al · 2022

Earlier work this paper cites.

Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 36. 8980–8987

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022 · 2022

Earlier work this paper cites.

Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning . 27268–27286

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022 · 2022

Earlier work this paper cites.

Foundational Models Defining a New Era in Vision: A Survey and Outlook

Original

Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023 · 2023

Earlier work this paper cites.

Accurate medium-range global weather forecasting with 3D neural networks

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023a · 2023

Earlier work this paper cites.

Accurate medium-range global weather forecasting with 3D neural networks

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023b · 2023

Cited alongside, same era.

DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting

Original

Salva Rühling Cachay, Bo Zhao, Hailey James, and Rose Yu. 2023 · 2023

Cited alongside, same era.

TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

Original

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. 2023 · 2023

Cited alongside, same era.

LLM4TS: Two-Stage Fine-Tuning for Time-Series Forecasting with Pre-Trained LLMs

Original

Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. 2023a · 2023

Cited alongside, same era.

Contrastive Trajectory Similarity Learning with Dual-Feature Attention. In 2023 IEEE 39th International Conference on Data Engineering (ICDE) . IEEE, 2933–2945

DiffLoad: uncertainty quantification in load forecasting with diffusion model

Original

Zhixian Wang, Qingsong Wen, Chaoli Zhang, Liang Sun, and Yi Wang. 2023d · 2023

Later among the works it cites.

Leveraging vision-language models for granular market change prediction

Original

Christopher Wimmer and Navid Rekabsaz. 2023 · 2023

Later among the works it cites.

The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges

Original

Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. 2023 · 2023

Later among the works it cites.

Temporal Data Meets LLM–Explainable Financial Time Series Forecasting

Original

Xinli Yu, Zheng Chen, Yuan Ling, Shujing Dong, Zongyi Liu, and Yanbin Lu. 2023 · 2023

Later among the works it cites.