Fetching the paper…
Reading the bibliography…
Tensor Attention, a multi-view attention that is able to capture high-order correlations among multiple modalities, can overcome the representational limitations of classical matrix attention.
On the complexity of k-sat
Russell Impagliazzo and Ramamohan Paturi · 2001
Earlier work this paper cites.
Tensortextures: Multilinear image-based rendering
M. Alex O. Vasilescu and Demetri Terzopoulos · 2004
Earlier work this paper cites.
Modeling and multiway analysis of chatroom tensors
Evrim Acar, Seyit A. Çamtepe, Mukkai S. Krishnamoorthy, and Bülent Yener · 2005
Earlier work this paper cites.
Out-of-core tensor approximation of multi-dimensional matrices of visual data
Hongcheng Wang, Qing Wu, Lin Shi, Yizhou Yu, and Narendra Ahuja · 2005
Earlier work this paper cites.
Collective sampling and analysis of high order tensors for chatroom communications
Evrim Acar, Seyit A. Camtepe, and Bülent Yener · 2006
Earlier work this paper cites.
The tophits model for higher-order web link analysis
Tamara Kolda and Brett Bader · 2006
Earlier work this paper cites.
Scalable tensor decompositions for multi-aspect data mining
Tamara G Kolda and Jimeng Sun · 2008
Earlier work this paper cites.
Tensors in image processing and computer vision
Santiago Aja-Fernández, Rodrigo de Luis Garcia, Dacheng Tao, and Xuelong Li · 2009
Earlier work this paper cites.
A multilinear (tensor) algebraic framework for computer graphics, computer vision, and machine learning
M. Alex O. Vasilescu · 2009
Earlier work this paper cites.
Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering
Alexandros Karatzoglou, Xavier Amatriain, Linas Baltrunas, and Nuria Oliver · 2010
Earlier work this paper cites.
Robust tensor analysis with l1-norm
Yanwei Pang, Xuelong Li, and Yuan Yuan · 2010
Earlier work this paper cites.
Pairwise interaction tensor factorization for personalized tag recommendation
Steffen Rendle and Lars Schmidt-Thieme · 2010
Earlier work this paper cites.
Applications of tensor (multiway array) factorizations and decompositions in data mining
Morten Mørup · 2011
Earlier work this paper cites.
Sequential transfer in multi-armed bandit with finite set of models
Mohammad Gheshlaghi Azar, Alessandro Lazaric, and Emma Brunskill · 2013
Earlier work this paper cites.
Algebraic complexity theory
Peter Bürgisser, Michael Clausen, and Mohammad A Shokrollahi · 2013
Earlier work this paper cites.
Fast matrix multiplication
Markus Bläser · 2013
Earlier work this paper cites.
Learning mixtures of spherical gaussians: moment methods and spectral decompositions
Daniel Hsu and Sham M Kakade · 2013
Earlier work this paper cites.
Typed tensor decomposition of knowledge bases for relation extraction
Kai-Wei Chang, Scott Wen tau Yih, Bishan Yang, and Chris Meek · 2014
Earlier work this paper cites.
Learning mixtures of discrete product distributions using spectral decompositions
Prateek Jain and Sewoong Oh · 2014
Earlier work this paper cites.
Low-rank tensors for scoring dependency structures
Tao Lei, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola · 2014
Earlier work this paper cites.
Spectral methods meet em: A provably optimal algorithm for crowdsourcing
Yuchen Zhang, Xi Chen, Denny Zhou, and Michael I Jordan · 2014
Earlier work this paper cites.
Matrix and tensor factorization methods for natural language processing
Guillaume Bouchard, Jason Naradowsky, Sebastian Riedel, Tim Rocktäschel, and Andreas Vlachos · 2015
Earlier work this paper cites.
Tensor decompositions for signal processing applications: From two-way to multiway component analysis
Andrzej Cichocki, Danilo Mandic, Lieven De Lathauwer, Guoxu Zhou, Qibin Zhao, Cesar Caiafa, and Huy Anh Phan · 2015
Earlier work this paper cites.
Multimodal data fusion: an overview of methods, challenges, and prospects
Dana Lahat, Tülay Adali, and Christian Jutten · 2015
Earlier work this paper cites.
High-order low-rank tensors for semantic role labeling
Tao Lei, Yuan Zhang, Alessandro Moschitti, and Regina Barzilay · 2015
Earlier work this paper cites.
Rethinking lda: moment matching for discrete ica
Anastasia Podosinnikova, Francis Bach, and Simon Lacoste-Julien · 2015
Earlier work this paper cites.
Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization
Canyi Lu, Jiashi Feng, Yudong Chen, Wei Liu, Zhouchen Lin, and Shuicheng Yan · 2016
Earlier work this paper cites.
The search problem in mixture models
Avik Ray, Joe Neeman, Sujay Sanghavi, and Sanjay Shakkottai · 2016
Earlier work this paper cites.
Newtonian program analysis via tensor product
Thomas Reps, Emma Turetsky, and Prathmesh Prabhu · 2016
Earlier work this paper cites.
Xinyang Yi, Constantine Caramanis, and Sujay Sanghavi · 2016
Earlier work this paper cites.
Iterative block tensor singular value thresholding for extraction of low rank component of image data
Longxi Chen, Yipeng Liu, and Ce Zhu · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Recovery guarantees for one-hidden-layer neural networks
Kai Zhong, Zhao Song, Prateek Jain, Peter L. Bartlett, and Inderjit S. Dhillon · 2017
Earlier work this paper cites.
On some fine-grained questions in algorithms and complexity
Virginia Vassilevska Williams · 2018
Earlier work this paper cites.
Camembert: a tasty french language model
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoit Sagot · 2019
Earlier work this paper cites.
A tensorized transformer for language modeling
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song · 2019
Earlier work this paper cites.
Transformer dissection: a unified understanding of transformer’s attention via the lens of kernel
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov · 2019
Earlier work this paper cites.
Sf-net: Structured feature network for continuous sign language recognition
Zhaoyang Yang, Zhenmei Shi, Xiaoyong Shen, and Yu-Wing Tai · 2019
Earlier work this paper cites.
Algorithms and hardness for linear algebra on geometric graphs
Josh Alman, Timothy Chu, Aaron Schild, and Zhao Song · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2020
Earlier work this paper cites.
Transformers are rnns: Fast autoregressive transformers with linear attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret · 2020
Earlier work this paper cites.
Attention-based sentiment analysis using convolutional and recurrent neural network
Mohd Usama, Belal Ahmad, Enmin Song, M Shamim Hossain, Mubarak Alrashoud, and Ghulam Muhammad · 2020
Earlier work this paper cites.
Attention mechanism for neural machine translation: a survey
Weihua He, Yongyun Wu, and Xiaohua Li · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.
Linear transformers are secretly fast weight programmers
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber · 2021
Cited alongside, same era.
Optimal-degree polynomial approximations for exponentials and gaussian kernel density estimation
Amol Aggarwal and Josh Alman · 2022
Cited alongside, same era.
Flamingo: a visual language model for few-shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al · 2022
Cited alongside, same era.
Introducing meta llama 3: The most capable openly available llm to date, 2024
Meta AI · 2024
Closest in time.
The claude 3 model family: Opus, sonnet, haiku, 2024
Anthropic · 2024
Closest in time.
The fine-grained complexity of gradient computation for training large language models
Josh Alman and Zhao Song · 2024
Closest in time.
How to capture higher-order correlations? generalizing matrix softmax attention to kronecker computation
Josh Alman and Zhao Song · 2024
Closest in time.
Unlimiformer: Long-range transformers with unlimited length input
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley · 2024
Closest in time.
Bypassing the exponential dependency: Looped transformers efficiently learn in-context by multi-step gradient descent, 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Attentive walk-aggregating graph neural networks
Mehmet F Demirel, Shengchao Liu, Siddhant Garg, Zhenmei Shi, and Yingyu Liang · 2022
Cited alongside, same era.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi · 2022
Cited alongside, same era.
Rethinking the role of demonstrations: What makes in-context learning work?
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer · 2022
Cited alongside, same era.
In-context learning and induction heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al · 2022
Cited alongside, same era.
Deep online fused video stabilization
Zhenmei Shi, Fuhao Shi, Wei-Sheng Lai, Chia-Kai Liang, and Yingyu Liang · 2022
Cited alongside, same era.
Linear complexity randomized self-attention mechanism
Lin Zheng, Chong Wang, and Lingpeng Kong · 2022
Cited alongside, same era.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Cited alongside, same era.
Bo Chen, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, and Zhao Song · 2024
Closest in time.
Hsr-enhanced sparse attention acceleration, 2024
Bo Chen, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, and Zhao Song · 2024
Closest in time.
A survey on evaluation of large language models
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al · 2024
Closest in time.
Faith and fate: Limits of transformers on compositionality
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al · 2024
Closest in time.
Longrope: Extending llm context window beyond 2 million tokens
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang · 2024
Closest in time.
Gemini 1.5 pro updates, 1.5 flash debut and 2 new gemma models
Google Gemini · 2024
Closest in time.
Gemini breaks new ground with a faster model, longer context, ai agents and more
Google · 2024
Closest in time.
Low rank matrix completion via robust alternating minimization in nearly linear time
Yuzhou Gu, Zhao Song, Junze Yin, and Lichen Zhang · 2024
Closest in time.
Outlier-efficient hopfield layers for large transformer-based models
Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Haozheng Luo, Hong-Yu Chen, Weijian Li, Wei-Po Wang, and Han Liu · 2024
Closest in time.
Nonparametric modern hopfield models
Jerry Yao-Chieh Hu, Bo-Yu Chen, Dennis Wu, Feng Ruan, and Han Liu · 2024
Closest in time.
Hyperattention: Long-context attention in near-linear time
Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David Woodruff, and Amir Zandieh · 2024
Closest in time.
Large language models for software engineering: A systematic literature review, 2024
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang · 2024
Closest in time.
Llm maybe longlm: Self-extend llm context window without tuning
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu · 2024
Closest in time.
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al · 2024
Closest in time.
Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Tianyi Zhou · 2024
Closest in time.
Fine-grained attention i/o complexity: Comprehensive analysis for backward passes, 2024
Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Yufa Zhou · 2024
Closest in time.
Yingyu Liang, Heshan Liu, Zhenmei Shi, Zhao Song, and Junze Yin · 2024
Closest in time.
Beyond linear approximations: A novel pruning approach for attention matrix, 2024
Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, and Yufa Zhou · 2024
Closest in time.
Chenyang Li, Yingyu Liang, Zhenmei Shi, and Zhao Song · 2024
Closest in time.
A tighter complexity analysis of sparsegpt
Xiaoyu Li, Yingyu Liang, Zhenmei Shi, and Zhao Song · 2024
Closest in time.
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee · 2024
Closest in time.
Multi-layer transformers gradient can be approximated in almost linear time
Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Yufa Zhou · 2024
Closest in time.
Toward infinite-long prefix in transformer
Yingyu Liang, Zhenmei Shi, Zhao Song, and Chiwun Yang · 2024
Closest in time.
Mm1: Methods, analysis & insights from multimodal llm pre-training
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al · 2024
Closest in time.
Megalodon: Efficient llm pretraining and inference with unlimited context length
Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou · 2024
Closest in time.
Hello gpt-4o
OpenAI · 2024
Closest in time.
Yarn: Efficient context window extension of large language models
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole · 2024
Closest in time.
Roformer: Enhanced transformer with rotary position embedding
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu · 2024
Closest in time.
Massive activations in large language models
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu · 2024
Closest in time.
You only cache once: Decoder-decoder architectures for language models
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei · 2024
Closest in time.
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty · 2024
Closest in time.
Fast dynamic sampling for determinantal point processes
Zhao Song, Junze Yin, Lichen Zhang, and Ruizhe Zhang · 2024
Closest in time.
Gemma: Open models based on gemini research and technology
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al · 2024
Closest in time.
Uniform memory retrieval with larger capacity for modern hopfield models
Dennis Wu, Jerry Yao-Chieh Hu, Teng-Yun Hsiao, and Han Liu · 2024
Closest in time.
Is a picture worth a thousand words? delving into spatial reasoning for vision language models
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, and Neel Joshi · 2024
Closest in time.
Do large language models have compositional ability? an investigation into limitations and scalability
Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang · 2024
Closest in time.
Towards few-shot adaptation of foundation models via multitask finetuning
Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang · 2024
Closest in time.
The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry
Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re · 2024
Closest in time.