Fetching the paper…
Reading the bibliography…
In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music.
Byte pair encoding: A text compression scheme that accelerates pattern matching
Yusuke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, and Takeshi Shinohara · 1999
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Music transcription modelling and composition using deep learning
Bob L. Sturm, João Felipe Santos, Oded Ben-Tal, and Iryna Korshunova · 2016
Earlier work this paper cites.
webmushra—a comprehensive framework for web-based listening tests
Michael Schoeffler, Sarah Bartoschek, Fabian-Robert Stöter, Marlene Roess, Susanne Westphal, Bernd Edler, and Jürgen Herre · 2018
Earlier work this paper cites.
Music transformer
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck · 2019
Earlier work this paper cites.
Deep music analogy via latent representation disentanglement
Ruihan Yang, Dingsu Wang, Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia · 2019
Earlier work this paper cites.
Root mean square layer normalization, 2019
Biao Zhang and Rico Sennrich · 2019
Earlier work this paper cites.
wav2vec 2.0: A framework for self-supervised learning of speech representations
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli · 2020
Earlier work this paper cites.
Scaling laws for autoregressive generative modeling, 2020
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish · 2020
Earlier work this paper cites.
Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions
Yu-Siang Huang and Yi-Hsuan Yang · 2020
Earlier work this paper cites.
Scaling laws for neural language models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Earlier work this paper cites.
Glu variants improve transformer
Noam Shazeer · 2020
Earlier work this paper cites.
Learning interpretable representation for controllable polyphonic music generation
Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia · 2020
Earlier work this paper cites.
Scaling laws for neural machine translation, 2021
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry · 2021
Earlier work this paper cites.
Scaling laws for transfer, 2021
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish · 2021
Earlier work this paper cites.
Hubert: Self-supervised speech representation learning by masked prediction of hidden units
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed · 2021
Cited alongside, same era.
Musicbert: A self-supervised learning of music representation
Hongyuan Zhu, Ye Niu, Di Fu, and Hao Wang · 2021
Cited alongside, same era.
Mae-ast: Masked autoencoding audio spectrogram transformer
Alan Baade, Puyuan Peng, and David Harwath · 2022
Cited alongside, same era.
Data2vec: A general framework for self-supervised learning in speech, vision and language
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli · 2022
Cited alongside, same era.
Scaling laws for reward model overoptimization, 2022
Leo Gao, John Schulman, and Jacob Hilton · 2022
Cited alongside, same era.
Musecoco: Generating symbolic music from text
Peiling Lu, Xin Xu, Chenfei Kang, Botao Yu, Chengyi Xing, Xu Tan, and Jiang Bian · 2023
Later among the works it cites.
Roformer: Enhanced transformer with rotary position embedding, 2023
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu · 2023
Later among the works it cites.
Anticipatory music transformer
John Thickstun, David Hall, Chris Donahue, and Percy Liang · 2023
Later among the works it cites.
Attention is all you need, 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2023
Later among the works it cites.
Exploring the efficacy of pre-trained checkpoints in text-to-music generation task
Shangda Wu and Maosong Sun · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al · 2022
Cited alongside, same era.
Masked autoencoders that listen
Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer · 2022
Cited alongside, same era.
Generating folk-like music in abc-notation with masked language models
Luca Casini, Nicolas Jonason, and Bob L. T. Sturm · 2023
Cited alongside, same era.
Beats: Audio pre-training with acoustic tokenizers
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei · 2023
Cited alongside, same era.
Singsong: Generating musical accompaniments from singing
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al · 2023
Cited alongside, same era.
Multitrack music transformer
Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatrick · 2023
Cited alongside, same era.
Mert: Acoustic music understanding model with large-scale self-supervised training
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al · 2023
Cited alongside, same era.
Tunesformer: Forming irish tunes with control codes by bar patching
Shangda Wu, Xiaobing Li, Feng Yu, and Maosong Sun · 2023
Later among the works it cites.
Clamp: Contrastive language-music pre-training for cross-modal symbolic music information retrieval
Shangda Wu, Dingyao Yu, Xu Tan, and Maosong Sun · 2023
Later among the works it cites.
Fast-hubert: an efficient training framework for self-supervised speech representation learning
Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, and Xie Chen · 2023
Later among the works it cites.
Eat: Self-supervised pre-training with efficient audio transformer
Wenxi Chen, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, and Xie Chen · 2024
Closest in time.
Simple and controllable music generation
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez · 2024
Closest in time.
Scaling data-constrained language models
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel · 2024
Closest in time.
Whole-song hierarchical generation of symbolic music using cascaded diffusion models
Ziyu Wang, Lejun Min, and Gus Xia · 2024
Closest in time.
Youtokentome: Unsupervised text tokenization library, 2021
YouTokenToMe · 2024
Closest in time.
Chatmusician: Understanding and generating music intrinsically with llm
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al · 2024
Closest in time.