Fetching the paper…
Reading the bibliography…
Normalization techniques are crucial for enhancing Transformer models' performance and stability in time series analysis tasks, yet traditional methods like batch and layer normalization often lead to issues such as token shift, attention shift, and sparse attention.
A Survey on Vision Transformer
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao · 1939
Earlier work this paper cites.
Batch normalization: accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Tail bounds via generic chaining
Sjoerd Dirksen · 2015
Earlier work this paper cites.
ElectricityLoadDiagrams20112014, 2015
Artur Trindade · 2015
Earlier work this paper cites.
Layer Normalization, July 2016
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton · 2016
Earlier work this paper cites.
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, June 2016
Tim Salimans and Diederik P. Kingma · 2016
Earlier work this paper cites.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Mean Field Residual Networks: On the Edge of Chaos, December 2017
Greg Yang and Samuel S. Schoenholz · 2017
Earlier work this paper cites.
Word vectors, reuse, and replicability: Towards a community repository of large-text resources
Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal · 2017
Earlier work this paper cites.
High-Dimensional Probability: An Introduction with Applications in Data Science
Roman Vershynin · 2018
Earlier work this paper cites.
Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu · 2018
Earlier work this paper cites.
The UEA multivariate time series classification archive, 2018, October 2018
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh · 2018
Earlier work this paper cites.
Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom · 2018
Earlier work this paper cites.
Learning Deep Transformer Models for Machine Translation
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao · 2019
Cited alongside, same era.
Root Mean Square Layer Normalization
Biao Zhang and Rico Sennrich · 2019
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Cited alongside, same era.
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush · 2020
Cited alongside, same era.
PowerNorm: rethinking batch normalization in transformers
Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer · 2020
Cited alongside, same era.
Scratching Visual Transformer’s Back with Uniform Attention, October 2022
Nam Hyeon-Woo, Kim Yu-Ji, Byeongho Heo, Dongyoon Han, Seong Joon Oh, and Tae-Hyun Oh · 2022
Later among the works it cites.
Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting
Yunhao Zhang and Junchi Yan · 2022
Later among the works it cites.
FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin · 2022
Later among the works it cites.
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam · 2022
Later among the works it cites.
TransDBC: Transformer for Multivariate Time-Series based Driver Behavior Classification
Jayant Vyas, Nishit Bhardwaj, Bhumika, and Debasis Das · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
On layer normalization in the transformer architecture
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu · 2020
Cited alongside, same era.
Language Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Cited alongside, same era.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2020
Cited alongside, same era.
Beyond batchnorm: towards a unified understanding of normalization in deep learning
Ekdeep S Lubana, Robert Dick, and Hidenori Tanaka · 2021
Cited alongside, same era.
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui · 2021
Cited alongside, same era.
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang · 2021
Cited alongside, same era.
Understanding the Failure of Batch Normalization for Transformers in NLP, October 2022
Jiaxi Wang, Ji Wu, and Lei Huang · 2022
Cited alongside, same era.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun · 2023
Later among the works it cites.
Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind · 2023
Later among the works it cites.
On the Expressivity Role of LayerNorm in Transformers’ Attention
Shaked Brody, Uri Alon, and Eran Yahav · 2023
Later among the works it cites.
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long · 2023
Later among the works it cites.
Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution
Yan Li, Xinjiang Lu, Haoyi Xiong, Jian Tang, Jiantao Su, Bo Jin, and Dejing Dou · 2023
Later among the works it cites.
TransNAS-TSAD: Harnessing Transformers for Multi-Objective Neural Architecture Search in Time Series Anomaly Detection, December 2023
Ijaz Ul Haq and Byung Suk Lee · 2023
Later among the works it cites.
DDMT: Denoising Diffusion Mask Transformer Models for Multivariate Time Series Anomaly Detection, October 2023
Chaocheng Yang, Tingyin Wang, and Xuanhui Yan · 2023
Later among the works it cites.