Fetching the paper…
Reading the bibliography…
Building a human-like system that continuously interacts with complex environments -- whether simulated digital worlds or human society -- presents several key challenges.
The joy of lex: How to have fun with 860,341,500 words
Gyles Daubeney Brandreth · 1980
Earlier work this paper cites.
Realm: Retrieval-augmented language model pre-training, 2020
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang · 2002
Earlier work this paper cites.
Mikhail S. Burtsev and Grigory V. Sapunov · 2006
Earlier work this paper cites.
Are women really more talkative than men?
Matthias R Mehl, Simine Vazire, Nairán Ramírez-Esparza, Richard B Slatcher, and James W Pennebaker · 2007
Earlier work this paper cites.
The probabilistic relevance framework: Bm25 and beyond
Stephen Robertson, Hugo Zaragoza, et al · 2009
Earlier work this paper cites.
Jason Weston, Sumit Chopra, and Antoine Bordes · 2014
Earlier work this paper cites.
End-to-end memory networks
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al · 2015
Earlier work this paper cites.
Using fast weights to attend to the recent past
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu · 2016
Earlier work this paper cites.
Memory augmented neural networks with wormhole connections
Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Transformer-xl: Attentive language models beyond a fixed-length context
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov · 2019
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova · 2019
Earlier work this paper cites.
Generalization through memorization: Nearest neighbor language models
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis · 2019
Earlier work this paper cites.
Armin: Towards a more efficient and light-weight recurrent memory network
Zhangheng Li, Jia-Xing Zhong, Jingjia Huang, Tao Zhang, Thomas Li, and Ge Li · 2019
Earlier work this paper cites.
Ordered memory
Yikang Shen, Shawn Tan, Arian Hosseini, Zhouhan Lin, Alessandro Sordoni, and Aaron C Courville · 2019
Earlier work this paper cites.
Dark experience for general continual learning: a strong, simple baseline
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara · 2020
Earlier work this paper cites.
Rethinking positional encoding in language pre-training
Guolin Ke, Di He, and Tie-Yan Liu · 2020
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu · 2020
Earlier work this paper cites.
ERNIE 2.0: A continual pre-training framework for language understanding
Yu Sun, Shuohuan Wang, Yu-Kun Li, et al · 2020
Earlier work this paper cites.
Memory transformer with hierarchical attention for long document processing
Arij Al Adel and Mikhail S Burtsev · 2021
Earlier work this paper cites.
Skyformer: Remodel self-attention with gaussian kernel and nyström method
Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang · 2021
Earlier work this paper cites.
Language model as an annotator: Exploring dialogpt for dialogue summarization
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu · 2021
Earlier work this paper cites.
Unsupervised dense information retrieval with contrastive learning
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave · 2021
Earlier work this paper cites.
Cape: Encoding relative positions with continuous augmented positional embeddings
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov · 2021
Earlier work this paper cites.
Train short, test long: Attention with linear biases enables input length extrapolation
Ofir Press, Noah Smith, and Mike Lewis · 2021
Earlier work this paper cites.
Learning to retrieve passages without supervision
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson · 2021
Earlier work this paper cites.
Roformer: Enhanced transformer with rotary position embedding
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu · 2021
Earlier work this paper cites.
GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model
Ben Wang and Aran Komatsuzaki · 2021
Earlier work this paper cites.
Dual memory structure for memory augmented neural networks for question-answering tasks
Amir Bidokhti and Shahrokh Ghaemmaghami · 2022
Earlier work this paper cites.
Recurrent memory transformer
Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev · 2022
Earlier work this paper cites.
Sketching as a tool for understanding and accelerating self-attention for long sequences
Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang · 2022
Earlier work this paper cites.
Continual pre-training mitigates forgetting in language and vision
Andrea Cossu, Tinne Tuytelaars, Antonio Carta, et al · 2022
Earlier work this paper cites.
Promptagator: Few-shot dense retrieval from 8 examples
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang · 2022
Earlier work this paper cites.
Calibrating factual knowledge in pretrained language models
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li · 2022
Earlier work this paper cites.
A survey on dialogue summarization: Recent advances and new frontiers
Xiachong Feng, Xiaocheng Feng, and Bing Qin · 2022
Earlier work this paper cites.
Aging with grace: Lifelong model editing with discrete key-value adaptors
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi · 2022
Earlier work this paper cites.
Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia · 2022
Earlier work this paper cites.
Locating and editing factual associations in gpt
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov · 2022
Earlier work this paper cites.
Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara · 2022
Cited alongside, same era.
A length-extrapolatable transformer
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei · 2022
Cited alongside, same era.
Memorizing transformers
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy · 2022
Cited alongside, same era.
Generate rather than retrieve: Large language models are strong context generators
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang · 2022
Cited alongside, same era.
Editing large language models: Problems, methods, and opportunities
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang · 2023
Later among the works it cites.
Self information update for large language models through mitigating exposure bias
Pengfei Yu and Heng Ji · 2023
Later among the works it cites.
Memorybank: Enhancing large language models with long-term memory
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang · 2023
Later among the works it cites.
Recurrentgpt: Interactive generation of (arbitrarily) long text
Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan · 2023
Later among the works it cites.
Pose: Efficient context window extension of llms via positional skip-wise training
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou · 2022
Cited alongside, same era.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Cited alongside, same era.
Llemma: An open language model for mathematics
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, et al · 2023
Cited alongside, same era.
Scaling transformer to 1m tokens and beyond with rmt
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev · 2023
Cited alongside, same era.
Quantifying memorization across neural language models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang · 2023
Cited alongside, same era.
Uprise: Universal prompt retrieval for improving zero-shot evaluation
Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang · 2023
Cited alongside, same era.
Dissecting transformer length extrapolation via the lens of receptive field analysis
Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge · 2023
Cited alongside, same era.
Longnet: Scaling transformers to 1,000,000,000 tokens
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei · 2023
Cited alongside, same era.
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li · 2023
Later among the works it cites.
Arigraph: Learning knowledge graph world models with episodic memory for llm agents, 2024
Petr Anokhin, Nikita Semenov, Artyom Sorokin, Dmitry Evseev, Mikhail Burtsev, and Evgeny Burnaev · 2024
Closest in time.
xlstm: Extended long short-term memory
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter · 2024
Closest in time.
Toward subgraph-guided knowledge graph question generation with graph neural networks
Yu Chen, Lingfei Wu, and Mohammed J. Zaki · 2024
Closest in time.
Lift yourself up: Retrieval-augmented text generation with self-memory
Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan · 2024
Closest in time.
Larimar: Large language models with episodic memory control
Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarathkrishna Swaminathan, Sihui Dai, Aurélie C. Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jirí Navrátil, Soham Dan, and Pin-Yu Chen · 2024
Closest in time.
Longrope: Extending llm context window beyond 2 million tokens
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang · 2024
Closest in time.
Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen · 2024
Closest in time.
From local to global: A graph rag approach to query-focused summarization
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson · 2024
Closest in time.
Model editing by pure fine-tuning
Govind Gangadhar and Karl Stratos · 2024
Closest in time.
Tiezheng Guo, Chen Wang, Yanyi Liu, Jiawei Tang, Pan Li, Sai Xu, Qingwen Yang, Xianlin Gao, Zhi Li, and Yingyou Wen · 2024
Closest in time.
Hipporag: Neurobiologically inspired long-term memory for large language models
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su · 2024
Closest in time.
Lm-infinite: Zero-shot extreme length generalization for large language models
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang · 2024
Closest in time.
GRAG: graph retrieval-augmented generation
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao · 2024
Closest in time.
Longrag: Enhancing retrieval-augmented generation with long-context llms
Ziyan Jiang, Xueguang Ma, and Wenhu Chen · 2024
Closest in time.
LLM maybe longlm: Self-extend LLM context window without tuning
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu · 2024
Closest in time.
Seo Hyun Kim, Kai Tzu-iunn Ong, Taeyoon Kwon, Namyoung Kim, Keummin Ka, SeongHyeon Bae, Yohan Jo, Seung-won Hwang, Dongha Lee, and Jinyoung Yeo · 2024
Closest in time.
A human-inspired reading agent with gist memory of very long contexts
Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer · 2024
Closest in time.
Lost in the middle: How language models use long contexts
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang · 2024
Closest in time.
Kun Luo, Zheng Liu, Shitao Xiao, and Kang Liu · 2024
Closest in time.
Leave no context behind: Efficient infinite context transformers with infini-attention
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal · 2024
Closest in time.
Transformers are multi-state rnns
Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz · 2024
Closest in time.
Memoria: Resolving fateful forgetting problem through human-inspired memory architecture, 2024
Sangjun Park and JinYeong Bak · 2024
Closest in time.
On the efficacy of eviction policy for key-value constrained generative language model inference
Siyu Ren and Kenny Q Zhu · 2024
Closest in time.
Raptor: Recursive abstractive processing for tree-organized retrieval
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning · 2024
Closest in time.
Ai-native memory: A pathway from llms towards agi
Jingbo Shang, Zai Zheng, Xiang Ying, Felix Tao, and Mindverse Team · 2024
Closest in time.
Flexibly scaling large language models contexts through extensible tokenization
Ninglu Shao, Shitao Xiao, Zheng Liu, and Peitian Zhang · 2024
Closest in time.
Learning to (learn at test time): Rnns with expressive hidden states
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al · 2024
Closest in time.
A survey on self-evolution of large language models
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou · 2024
Closest in time.
Continual learning for large language models: A survey
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari · 2024
Closest in time.
Efficient streaming language models with attention sinks
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis · 2024
Closest in time.
Memory 3 {}^{\mbox{3}} : Language modeling with explicit memory
Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, Chenyang Xi, Yu Yu, Kai Chen, Feiyu Xiong, Linpeng Tang, and Weinan E · 2024
Closest in time.
Zi Yang and Nan Hua · 2024
Closest in time.
Symbolic learning enables self-evolving agents
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al · 2024
Closest in time.