2022

Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

Zhang, Wenrui, Yang, Ling, Geng, Shijia et al.

Understand

Unsupervised/self-supervised representation learning in time series is critical since labeled samples are usually scarce in real-world scenarios.

Existing approaches mainly leverage the contrastive learning framework, which automatically learns to understand the similar and dissimilar data pairs.
Nevertheless, they are restricted to the prior knowledge of constructing pairs, cumbersome sampling policy, and unstable performances when encountering sampling bias.
Also, few works have focused on effectively modeling across temporal-spectral relations to extend the capacity of representations.

Built on

A. V. Oppenheim, A. S. Willsky, and S. H. Nawab, Signals &Amp; Systems (2Nd Ed.) . Prentice-Hall, Inc., 1996
1996
Earlier work this paper cites.
A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley, “Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals,” circulation , vol. 101, no. 23, pp. e215–e220, 2000
2000
Earlier work this paper cites.
B. Kemp, A. H. Zwinderman, B. Tuk, H. A. Kamphuisen, and J. J. Oberye, “Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg,” IEEE Transactions on Biomedical Engineering , vol. 47, no. 9, pp. 1185–1194, 2000
2000
Earlier work this paper cites.
L. Ye and E. Keogh, “Time series shapelets: a new primitive for data mining,” in Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , 2009, pp. 947–956
2009
Earlier work this paper cites.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proceedings of the 26th annual international conference on machine learning , 2009, pp. 41–48
2009
Earlier work this paper cites.
M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Research, Y. W. Teh and M. Titterington, Eds., vol. 9, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010, pp. 297–304
2010
Earlier work this paper cites.
D. Anguita, A. Ghio, L. Oneto, X. Parra Perez, and J. L. Reyes Ortiz, “A public domain dataset for human activity recognition using smartphones,” in Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning , 2013, pp. 437–442
2013
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2794–2802
2015
Earlier work this paper cites.
A. Hyvarinen and H. Morioka, “Unsupervised feature extraction by time-contrastive learning and nonlinear ica,” Advances in Neural Information Processing Systems , vol. 29, 2016
2016
Earlier work this paper cites.
A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio.” SSW , vol. 125, p. 2, 2016
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
E. L. Denton et al. , “Unsupervised learning of disentangled representations from video,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018
Original
2018
Earlier work this paper cites.
R. Nawaz, H. Nisar, and Y. V. Voon, “The effect of music on human brain; frequency domain and time series analysis using electroencephalogram,” IEEE Access , vol. 6, pp. 45 191–45 205, 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 132–149
2018
Earlier work this paper cites.
J.-Y. Franceschi, A. Dieuleveut, and M. Jaggi, “Unsupervised scalable representation learning for multivariate time series,” Advances in neural information processing systems , vol. 32, 2019
2019
Earlier work this paper cites.

Similar

C. Zhang, D. Song, Y. Chen, X. Feng, C. Lumezanu, W. Cheng, J. Ni, B. Zong, H. Chen, and N. V. Chawla, “A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data,” in Proceedings of the AAAI conference on artificial intelligence , vol. 33, no. 01, 2019, pp. 1409–1416
2019
Cited alongside, same era.
M. Rhif, A. Ben Abbes, I. R. Farah, B. Martínez, and Y. Sang, “Wavelet transform application for/in non-stationary time-series analysis: A review,” Applied Sciences , vol. 9, no. 7, 2019
2019
Cited alongside, same era.
Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, “Ernie: Enhanced language representation with informative entities,” arXiv preprint arXiv:1905.07129 , 2019
Original
2019
Cited alongside, same era.
Q. Chen, A. Zhang, T. Huang, Q. He, and Y. Song, “Imbalanced dataset-based echo state networks for anomaly detection,” Neural Computing and Applications , vol. 32, no. 8, pp. 3685–3694, 2020
2020
Cited alongside, same era.
N. Zhang, “Learning adversarial transformer for symbolic music generation,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–10, 2020
2020
Cited alongside, same era.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “work for contrastive learning of visual representations,” in Proceedings of the 37th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 119, 13–18 Jul 2020, pp. 1597–1607
2020
Cited alongside, same era.
M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “Spanbert: Improving pre-training by representing and predicting spans,” Transactions of the Association for Computational Linguistics , vol. 8, pp. 64–77, 2020
2020
Cited alongside, same era.
X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297 , 2020
Original
2020
Cited alongside, same era.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in Neural Information Processing Systems , vol. 33, pp. 9912–9924, 2020
2020
Cited alongside, same era.
P. Wagner, N. Strodthoff, R.-D. Bousseljot, D. Kreiseler, F. I. Lunze, W. Samek, and T. Schaeffter, “Ptb-xl, a large publicly available electrocardiography dataset,” Scientific data , vol. 7, no. 1, pp. 1–15, 2020
2020
Cited alongside, same era.
Q. Tan, M. Ye, A. J. Ma, B. Yang, T. C.-F. Yip, G. L.-H. Wong, and P. C. Yuen, “Explainable uncertainty-aware convolutional recurrent neural network for irregular medical time series,” IEEE Transactions on Neural Networks and Learning Systems , vol. 32, no. 10, pp. 4665–4679, 2021
2021
Cited alongside, same era.
F. M. Bianchi, S. Scardapane, S. Løkse, and R. Jenssen, “Reservoir computing approaches for representation and classification of multivariate time series,” IEEE Transactions on Neural Networks and Learning Systems , vol. 32, no. 5, pp. 2169–2179, 2021
2021
Cited alongside, same era.

Then

H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in Proceedings of AAAI , 2021
2021
Later among the works it cites.
G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff, “A transformer-based framework for multivariate time series representation learning,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , 2021, pp. 2114–2124
2021
Later among the works it cites.
H. Zhang, J. Y. Koh, J. Baldridge, H. Lee, and Y. Yang, “Cross-modal contrastive learning for text-to-image generation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 833–842
2021
Later among the works it cites.
Z. Yue, Y. Wang, J. Duan, T. Yang, C. Huang, Y. Tong, and B. Xu, “Ts2vec: Towards universal representation of time series,” arXiv preprint arXiv:2106.10466 , 2021
Original
2021
Later among the works it cites.
E. Eldele, Z. Chen, C. Liu, M. Wu, C.-K. Kwoh, X. Li, and C. Guan, “An attention-based deep learning approach for sleep stage classification with single-channel eeg,” IEEE Transactions on Neural Systems and Rehabilitation Engineering , vol. 29, pp. 809–818, 2021
2021
Later among the works it cites.
Y. Huang, G. G. Yen, and V. S. Tseng, “Snippet policy network v2: Knee-guided neuroevolution for multi-lead ecg early classification,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–15, 2022
2022
Closest in time.
W. Zheng and J. Hu, “Multivariate time series prediction based on temporal change information learning method,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–15, 2022
2022
Closest in time.
A. Garg, W. Zhang, J. Samaran, R. Savitha, and C.-S. Foo, “An evaluation of anomaly detection and diagnosis in multivariate time series,” IEEE Transactions on Neural Networks and Learning Systems , vol. 33, no. 6, pp. 2508–2517, 2022
2022
Closest in time.
S.-E. Benkabou, K. Benabdeslem, V. Kraus, K. Bourhis, and B. Canitia, “Local anomaly detection for multivariate time series by temporal dependency based on poisson model,” IEEE Transactions on Neural Networks and Learning Systems , vol. 33, no. 11, pp. 6701–6711, 2022
2022
Closest in time.
L. Yang and S. Hong, “Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion,” in International Conference on Machine Learning . PMLR, 2022, pp. 25 038–25 054
2022
Closest in time.
Q. Song, B. Sun, and S. Li, “Multimodal sparse transformer network for audio-visual speech recognition,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–11, 2022
2022
Closest in time.
Z. Yang, H. Yu, Y. He, W. Sun, Z.-H. Mao, and A. Mian, “Fully convolutional network-based self-supervised learning for semantic segmentation,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–11, 2022
2022
Closest in time.
C. Liu, Y. Yao, D. Luo, Y. Zhou, and Q. Ye, “Self-supervised motion perception for spatiotemporal representation learning,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–15, 2022
2022
Closest in time.
L. Yang and S. Hong, “Omni-granular ego-semantic propagation for self-supervised graph representation learning,” in International Conference on Machine Learning . PMLR, 2022, pp. 25 022–25 037
2022
Closest in time.
Y. Zheng, M. Jin, S. Pan, Y.-F. Li, H. Peng, M. Li, and Z. Li, “Toward graph self-supervised learning with contrastive adjusted zooming,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–15, 2022
2022
Closest in time.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 000–16 009
2022
Closest in time.
J. Jiang, J. Chen, and Y. Guo, “A dual-masked auto-encoder for robust motion capture with spatial-temporal skeletal token completion,” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 5123–5131
2022
Closest in time.
L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, Y. Shao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” arXiv preprint arXiv:2209.00796 , 2022
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…