Fetching the paper…
Reading the bibliography…
Non-overlapping patch-wise convolution is the default image tokenizer for all state-of-the-art vision Transformer (ViT) models.
Numerical Solution of Partial Differential Equations: Finite Difference Methods (Oxford Applied Mathematics and Computing Science Series) 3rd Edition
G. D. Smith · 1986
Earlier work this paper cites.
Ten Lectures on Wavelets
Ingrid Daubechies · 1992
Earlier work this paper cites.
Wavelets and Filter Banks
Gilbert Strang and Truong Nguyen · 1996
Earlier work this paper cites.
JPEG2000 Standard for Image Compression: Concepts, Algorithms and VLSI Architectures
Tinku Acharya and Ping-Sing Tsai · 2004
Earlier work this paper cites.
The pascal visual object classes challenge
M. Everingham, L. Van Gool, C.K. Williams, J. Winn, and A. Zisserman · 2010
Earlier work this paper cites.
Are we ready for autonomous driving? the kitti vision benchmark suite
A. Geiger, P. Lenz, and R. Urtasun · 2012
Earlier work this paper cites.
Explaining and Harnessing Adversarial Examples
I. J. Goodfellow, J. Shlens, and C. Szegedy · 2014
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2014
Earlier work this paper cites.
Numerical Methods for Partial Differential Equations: Finite Difference and Finite Volume Methods 1st Edition
Sandip Mazumder · 2015
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2016
Earlier work this paper cites.
Semantic understanding of scenes through the ade20k dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba · 2016
Earlier work this paper cites.
Neural discrete representation learning
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
mixup: Beyond empirical risk minimization
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz · 2017
Earlier work this paper cites.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson · 2018
Earlier work this paper cites.
Gabor convolutional networks
Shangzhen Luan, Chen Chen, Baochang Zhang, J. Han, and Jianzhuang Liu · 2018
Earlier work this paper cites.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran · 2018
Earlier work this paper cites.
An iterative wavelet threshold for signal denoising
Fábio Mariano Bayer, Alice J. Kozakevicius, and Renato J. Cintra · 2019
Earlier work this paper cites.
Randaugment: Practical automated data augmentation with a reduced search space
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le · 2019
Earlier work this paper cites.
Digital Image Processing 4th edition
Rafael C. Gonzalez and Richard E. Woods · 2019
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2019
Earlier work this paper cites.
Etc: Encoding long and structured inputs in transformers
Joshua Ainslie, S. Ontañón, C. Alberti, V. Cvicek, Zachary Kenneth Fisher, Philip Pham, Anirudh Ravula, S. Sanghai, Qifan Wang, and L. Yang · 2020
Earlier work this paper cites.
Longformer: The long-document transformer
Iz Beltagy, Matthew E. Peters, and Arman Cohan · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom B. Brown, Benjamin Pickman Mann, Nick Ryder, Melanie Subbiah, Jean Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric J Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Cited alongside, same era.
Generative pretraining from pixels
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever · 2020
Cited alongside, same era.
Masked language modeling for proteins via linearly scalable long-context transformers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamás Sarlós, David Belanger, Lucy J. Colwell, and Adrian Weller · 2020
Cited alongside, same era.
Transformer feed-forward layers are key-value memories
Mor Geva, R. Schuster, Jonathan Berant, and Omer Levy · 2020
Cited alongside, same era.
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel · 2021
Later among the works it cites.
H-transformer-1d: Fast one-dimensional hierarchical attention for sequences
Zhenhai Zhu and Radu Soricut · 2021
Later among the works it cites.
PaLI: A jointly-scaled multilingual language-image model
Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut · 2022
Later among the works it cites.
Flashattention: Fast and memory-efficient exact attention with io-awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R’e · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Reformer: The efficient transformer
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya · 2020
Cited alongside, same era.
Efficient content-based sparse attention with routing transformers
Aurko Roy, M. Saffar, Ashish Vaswani, and David Grangier · 2020
Cited alongside, same era.
Fast WordPiece tokenization
Xinying Song, Alexandru Salcianu, Yang Song, Dave Dopson, and Denny Zhou · 2020
Cited alongside, same era.
Efficient transformers: A survey
Yi Tay, M. Dehghani, Dara Bahri, and Donald Metzler · 2020
Cited alongside, same era.
Linformer: Self-attention with linear complexity
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma · 2020
Cited alongside, same era.
Big bird: Transformers for longer sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed · 2020
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2021
Cited alongside, same era.
Taming transformers for high-resolution image synthesis
Patrick Esser, Robin Rombach, and Björn Ommer · 2021
Cited alongside, same era.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al · 2022
Later among the works it cites.
Pix2Struct: Screenshot parsing as pretraining for visual language understanding
Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova · 2022
Later among the works it cites.
Swin Transformer V2: Scaling Up Capacity and Resolution
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al · 2022
Later among the works it cites.
UL2: Unifying language learning paradigms
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier García, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler · 2022
Later among the works it cites.
Multi-stage image denoising with the wavelet transform
Chunwei Tian, Menghua Zheng, Wangmeng Zuo, Bob Zhang, Yanning Zhang, and David Zhang · 2022
Later among the works it cites.
DeiT III: Revenge of the ViT
Hugo Touvron, Matthieu Cord, and Hervé Jégou · 2022
Later among the works it cites.
Wave-vit: Unifying wavelet and transformers for visual representation learning
Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, and Tao Mei · 2022
Later among the works it cites.
A-vit: Adaptive tokens for efficient vision transformer
Hongxu Yin, Arash Vahdat, José Manuel Álvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov · 2022
Later among the works it cites.
Vector-quantized image modeling with improved vqgan
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu · 2022
Later among the works it cites.
Coca: Contrastive captioners are image-text foundation models
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu · 2022
Later among the works it cites.
Scaling vision transformers
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer · 2022
Later among the works it cites.
Flexivit: One model for all patch sizes
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic · 2023
Later among the works it cites.
Flashattention-2: Faster attention with better parallelism and work partitioning
Tri Dao · 2023
Later among the works it cites.
Patch n’ Pack: NaViT, a vision transformer for any aspect ratio and resolution
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M. Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey A. Gritsenko, Mario Luvci’c, and Neil Houlsby · 2023
Later among the works it cites.
Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings
Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson · 2023
Later among the works it cites.
Vision transformers with mixed-resolution tokenization
Tomer Ronen, Omer Levy, and Avram Golbert · 2023
Later among the works it cites.
Adversarial attacks on image classification models: Fgsm and patch attacks and their impact
Jaydip Sen and Subhasish Dasgupta · 2023
Later among the works it cites.
Learnable gabor kernels in convolutional neural networks for seismic interpretation tasks
Fu Wang and Tariq Alkhalifah · 2023
Later among the works it cites.