Built on

A diversity-promoting objective function for neural conversation models
Original
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan · 2015
Earlier work this paper cites.
Hierarchical neural story generation
Original
Angela Fan, Mike Lewis, and Yann Dauphin · 2018
Earlier work this paper cites.
Strategies for structuring story generation
Original
Angela Fan, Mike Lewis, and Yann Dauphin · 2019
Earlier work this paper cites.
Plan, write, and revise: an interactive system for open-domain story generation
Original
Seraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng · 2019
Earlier work this paper cites.
Plan-and-write: Towards better automatic storytelling
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan · 2019
Earlier work this paper cites.
Measuring massive multitask language understanding
Original
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt · 2020
Earlier work this paper cites.
Zero: Memory optimizations toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He · 2020
Earlier work this paper cites.
Summarize, outline, and elaborate: Long-text generation via hierarchical supervision from extractive summaries
Original
Xiaofei Sun, Zijun Sun, Yuxian Meng, Jiwei Li, and Chun Fan · 2020
Earlier work this paper cites.
Progressive generation of long text with pretrained language models
Original
Bowen Tan, Zichao Yang, Maruan AI-Shedivat, Eric P Xing, and Zhiting Hu · 2020
Earlier work this paper cites.
Wordcraft: A human-ai collaborative editor for story writing
Original
Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan · 2021
Earlier work this paper cites.
Talebrush: Sketching stories with generative pretrained language models
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang · 2022
Earlier work this paper cites.
Self-instruct: Aligning language models with self-generated instructions
Original
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Earlier work this paper cites.

Similar

Longbench: A bilingual, multitask benchmark for long context understanding
Original
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al · 2023
Cited alongside, same era.
Lm-infinite: Simple on-the-fly length generalization for large language models
Original
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang · 2023
Cited alongside, same era.
Mistral 7b
Original
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al · 2023
Cited alongside, same era.
Needle in a haystack - pressure testing llms, 2023
Gregory Kamradt · 2023
Cited alongside, same era.
Efficient memory management for large language model serving with pagedattention

Then

Self-play with execution feedback: Improving instruction-following capabilities of large language models
Original
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou · 2024
Closest in time.
The llama 3 herd of models, 2024
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al · 2024
Closest in time.
Towards a unified view of preference learning for large language models: A survey
Original
Bofei Gao, Feifan Song, Yibo Miao, Zefan Cai, Zhe Yang, Liang Chen, Helan Hu, Runxin Xu, Qingxiu Dong, Ce Zheng, et al · 2024
Closest in time.
Rephrasing the web: A recipe for compute and data-efficient language modeling
Original
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly · 2024
Closest in time.
The what, why, and how of context length extension techniques in large language models–a detailed survey

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Language Models can Self-Lengthen to Generate Long Texts

Quan, Shanghaoran, Tang, Tianyi, Yu, Bowen et al.

Understand

Recent advancements in Large Language Models (LLMs) have significantly enhanced their ability to process long contexts, yet a notable gap remains in generating long, aligned outputs.

This limitation stems from a training gap where pre-training lacks effective instructions for long-text generation, and post-training data primarily consists of short query-response pairs.
Current approaches, such as instruction backtranslation and behavior imitation, face challenges including data quality, copyright issues, and constraints on proprietary model usage.
In this paper, we introduce an innovative iterative training framework called Self-Lengthen that leverages only the intrinsic knowledge and skills of LLMs without the need for auxiliary data or proprietary models.

Built on

A diversity-promoting objective function for neural conversation models
Original
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan · 2015
Earlier work this paper cites.
Hierarchical neural story generation
Original
Angela Fan, Mike Lewis, and Yann Dauphin · 2018
Earlier work this paper cites.
Strategies for structuring story generation
Original
Angela Fan, Mike Lewis, and Yann Dauphin · 2019
Earlier work this paper cites.
Plan, write, and revise: an interactive system for open-domain story generation
Original
Seraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng · 2019
Earlier work this paper cites.
Plan-and-write: Towards better automatic storytelling
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan · 2019
Earlier work this paper cites.
Measuring massive multitask language understanding
Original
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt · 2020
Earlier work this paper cites.
Zero: Memory optimizations toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He · 2020
Earlier work this paper cites.
Summarize, outline, and elaborate: Long-text generation via hierarchical supervision from extractive summaries
Original
Xiaofei Sun, Zijun Sun, Yuxian Meng, Jiwei Li, and Chun Fan · 2020
Earlier work this paper cites.
Progressive generation of long text with pretrained language models
Original
Bowen Tan, Zichao Yang, Maruan AI-Shedivat, Eric P Xing, and Zhiting Hu · 2020
Earlier work this paper cites.
Wordcraft: A human-ai collaborative editor for story writing
Original
Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan · 2021
Earlier work this paper cites.
Talebrush: Sketching stories with generative pretrained language models
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang · 2022
Earlier work this paper cites.
Self-instruct: Aligning language models with self-generated instructions
Original
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Earlier work this paper cites.

Similar

Longbench: A bilingual, multitask benchmark for long context understanding
Original
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al · 2023
Cited alongside, same era.
Lm-infinite: Simple on-the-fly length generalization for large language models
Original
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang · 2023
Cited alongside, same era.
Mistral 7b
Original
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al · 2023
Cited alongside, same era.
Needle in a haystack - pressure testing llms, 2023
Gregory Kamradt · 2023
Cited alongside, same era.
Efficient memory management for large language model serving with pagedattention

Then

Self-play with execution feedback: Improving instruction-following capabilities of large language models
Original
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou · 2024
Closest in time.
The llama 3 herd of models, 2024
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al · 2024
Closest in time.
Towards a unified view of preference learning for large language models: A survey
Original
Bofei Gao, Feifan Song, Yibo Miao, Zefan Cai, Zhe Yang, Liang Chen, Helan Hu, Runxin Xu, Qingxiu Dong, Ce Zheng, et al · 2024
Closest in time.
Rephrasing the web: A recipe for compute and data-efficient language modeling
Original
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly · 2024
Closest in time.
The what, why, and how of context length extension techniques in large language models–a detailed survey

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica · 2023

Cited alongside, same era.

Self-alignment with instruction backtranslation

Original

Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis · 2023

Cited alongside, same era.

Alignbench: Benchmarking chinese alignment of large language models

Original

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al · 2023

Cited alongside, same era.

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al · 2023

Cited alongside, same era.

Recurrentgpt: Interactive generation of (arbitrarily) long text

Original

Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan · 2023

Cited alongside, same era.

Anthropic: Introducing claude 3.5 sonnet, 2024

Anthropic · 2024

Cited alongside, same era.

Longwriter: Unleashing 10,000+ word generation from long context llms

Original

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li · 2024

Cited alongside, same era.

Towards scalable automated alignment of llms: A survey

Original

Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He, Xianpei Han, et al · 2024

Cited alongside, same era.

Original

Saurav Pawar, SM Tonmoy, SM Zaman, Vinija Jain, Aman Chadha, and Amitava Das · 2024

Closest in time.

Suri: Multi-constraint instruction following for long-form text generation

Original

Chau Minh Pham, Simeng Sun, and Mohit Iyyer · 2024

Closest in time.

Hellobench: Evaluating long text generation capabilities of large language models

Original

Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al · 2024

Closest in time.

Assisting in writing wikipedia-like articles from scratch with large language models

Original

Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam · 2024

Closest in time.

Weaver: Foundation models for creative writing

Original

Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, et al · 2024

Closest in time.

A survey on knowledge distillation of large language models

Original

Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou · 2024

Closest in time.

Qwen2 technical report

Original

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al · 2024

Closest in time.

Llamafactory: Unified efficient fine-tuning of 100+ language models

Original

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo · 2024

Closest in time.