Fetching the paper…
Reading the bibliography…
In the rapidly advancing arena of large language models (LLMs), a key challenge is to enhance their capabilities amid a looming shortage of high-quality training data.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J · 2009
Earlier work this paper cites.
Active learning with optimal instance subset selection
Fu, Y., Zhu, X., and Elmagarmid, A. K · 2013
Earlier work this paper cites.
Online batch selection for faster training of neural networks
Loshchilov, I. and Hutter, F · 2015
Earlier work this paper cites.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S · 2015
Earlier work this paper cites.
Decoupled weight decay regularization
Loshchilov, I. and Hutter, F · 2017
Earlier work this paper cites.
Active learning for convolutional neural networks: A core-set approach
Sener, O. and Savarese, S · 2017
Earlier work this paper cites.
Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions
Wang, M., Fang, E. X., and Liu, H · 2017
Earlier work this paper cites.
Think you have solved question answering? try arc, the ai2 reasoning challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Earlier work this paper cites.
Not all samples are created equal: Deep learning with importance sampling
Katharopoulos, A. and Fleuret, F · 2018
Earlier work this paper cites.
Learning from synthetic data: Addressing domain shift for semantic segmentation
Sankaranarayanan, S., Balaji, Y., Jain, A., Lim, S. N., and Chellappa, R · 2018
Earlier work this paper cites.
A simple method for commonsense reasoning
Trinh, T. H. and Le, Q. V · 2018
Earlier work this paper cites.
Boolq: Exploring the surprising difficulty of natural yes/no questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K · 2019
Earlier work this paper cites.
Accelerating deep learning by focusing on the biggest losers
Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al · 2019
Earlier work this paper cites.
Robust optimization over multiple domains
Qian, Q., Zhu, S., Tang, J., Jin, R., Sun, B., and Li, H · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al · 2019
Earlier work this paper cites.
Hellaswag: Can a machine really finish your sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y · 2019
Earlier work this paper cites.
Piqa: Reasoning about physical commonsense in natural language
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al · 2020
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
The pile: An 800gb dataset of diverse text for language modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al · 2020
Cited alongside, same era.
Measuring massive multitask language understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2020
Cited alongside, same era.
Coresets for data-efficient training of machine learning models
Mirzasoleiman, B., Bilmes, J., and Leskovec, J · 2020
Cited alongside, same era.
Auditing and generating synthetic data with controllable trust trade-offs
Belgodere, B., Dognin, P., Ivankay, A., Melnyk, I., Mroueh, Y., Mojsilovic, A., Navartil, J., Nitsure, A., Padhi, I., Rigotti, M., et al · 2023
Later among the works it cites.
Skill-it! a data-driven skills framework for understanding and training language models
Chen, M. F., Roberts, N., Bhatia, K., Wang, J., Zhang, C., Sala, F., and Ré, C · 2023
Later among the works it cites.
Irreducible curriculum for language model pretraining
Fan, S. and Jaggi, M · 2023
Later among the works it cites.
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Exploring the limits of transfer learning with a unified text-to-text transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J · 2020
Cited alongside, same era.
An online method for a class of distributionally robust optimization with non-convex objectives
Qi, Q., Guo, Z., Xu, Y., Jin, R., and Yang, T · 2021
Cited alongside, same era.
Winogrande: An adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y · 2021
Cited alongside, same era.
Adversarial attack generation empowered by min-max optimization
Wang, J., Zhang, T., Liu, S., Chen, P.-Y., Xu, J., Fardad, M., and Li, B · 2021
Cited alongside, same era.
Is synthetic data from generative models ready for image recognition?
He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and Qi, X · 2022
Cited alongside, same era.
Stochastic constrained dro with a complexity independent of sample size
Qi, Q., Lyu, J., Bai, E. W., Yang, T., et al · 2022
Cited alongside, same era.
Will we run out of data? an analysis of the limits of scaling datasets in machine learning
Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A · 2022
Cited alongside, same era.
Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al · 2023
Later among the works it cites.
Platypus: Quick, cheap, and powerful refinement of llms
Lee, A. N., Hunter, C. J., and Ruiz, N · 2023
Later among the works it cites.
Textbooks are all you need ii: phi-1.5 technical report
Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T · 2023
Later among the works it cites.
When less is more: Investigating data pruning for pretraining llms at scale
Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S · 2023
Later among the works it cites.
Orca 2: Teaching small language models how to reason
Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., Chen, X., Razdaibiedina, A., Jones, E., Aggarwal, K., et al · 2023
Later among the works it cites.
A comprehensive overview of large language models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Barnes, N., and Mian, A · 2023
Later among the works it cites.
Attentional-biased stochastic gradient descent
Qi, Q., Xu, Y., Yin, W., Jin, R., and Yang, T · 2023
Later among the works it cites.
Qiu, Z.-H., Hu, Q., Yuan, Z., Zhou, D., Zhang, L., and Yang, T · 2023
Later among the works it cites.
Stanford alpaca: An instruction-following llama model
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al · 2023
Later among the works it cites.
Sheared llama: Accelerating language model pre-training via structured pruning
Xia, M., Gao, T., Zeng, Z., and Chen, D · 2023
Later among the works it cites.
Doremi: Optimizing data mixtures speeds up language model pretraining
Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W · 2023
Later among the works it cites.