2022

TabLLM: Few-shot Classification of Tabular Data with Large Language Models

Hegselmann, Stefan, Buendia, Alejandro, Lang, Hunter et al.

Understand

We study the application of large language models to zero-shot and few-shot classification of tabular data.

We prompt the large language model with a serialization of the tabular data to a natural-language string, together with a short description of the classification problem.
In the few-shot setting, we fine-tune the large language model using some labeled examples.
We evaluate several serialization methods including templates, table-to-text models, and large language models.

Built on

Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020) · 1901
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020) · 1901
Earlier work this paper cites.
Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus
Smith, J. W., Everhart, J., Dickson, W., Knowler, W., and Johannes, R. (1988) · 1988
Earlier work this paper cites.
International application of a new probability algorithm for the diagnosis of coronary artery disease
Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmid, J.-J., Sandhu, S., Guppy, K. H., Lee, S., and Froelicher, V. (1989) · 1989
Earlier work this paper cites.
Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid
Kohavi, R. et al. (1996) · 1996
Earlier work this paper cites.
Sparse spatial autoregressions
Pace, R. K. and Barry, R. (1997) · 1997
Earlier work this paper cites.
Knowledge discovery on rfm model using bernoulli sequence
Yeh, I.-C., Yang, K.-J., and Ting, T.-M. (2009) · 2009
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011) · 2011
Earlier work this paper cites.
TabTransformer: Tabular Data Modeling Using Contextual Embeddings
Original
Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. (2020) · 2012
Earlier work this paper cites.
A data-driven approach to predict the success of bank telemarketing
Moro, S., Cortez, P., and Rita, P. (2014) · 2014
Earlier work this paper cites.
Endgame analysis of dou shou qi
van Rijn, J. N. and Vis, J. K. (2014) · 2014
Earlier work this paper cites.
Deep Neural Decision Forests
Kontschieder, P., Fiterau, M., Criminisi, A., and Bulo, S. R. (2015) · 2015
Earlier work this paper cites.
Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers
Hripcsak, G., Duke, J. D., Shah, N. H., Reich, C. G., Huser, V., Schuemie, M. J., Suchard, M. A., Park, R. W., Wong, I. C. K., Rijnbeek, P. R., van der Lei, J., Pratt, N., Nor&#233, N, G. N., Li, Y.-C., Stang, P. E., Madigan, D., and Ryan, P. B. (2015) · 2015
Earlier work this paper cites.
XGBoost: A Scalable Tree Boosting System
Chen, T. and Guestrin, C. (2016) · 2016
Earlier work this paper cites.
Randomized trial of communication facilitators to reduce family distress and intensity of end-of-life care
Curtis, J. R., Treece, P. D., Nielsen, E. L., Gold, J., Ciechanowski, P. S., Shannon, S. E., Khandelwal, N., Young, J. P., and Engelberg, R. A. (2016) · 2016
Earlier work this paper cites.
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. (2017) · 2017
Earlier work this paper cites.
UCI machine learning repository
Dua, D. and Graff, C. (2017) · 2017
Earlier work this paper cites.
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S., Downing, L., Ng, A., and Shah, N. H. (2018) · 2018
Earlier work this paper cites.
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S., Downing, L., Ng, A., and Shah, N. H. (2018) · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019) · 2019
Earlier work this paper cites.
How many rare diseases are there?
Haendel, M., Vasilevsky, N., Unni, D., Bologa, C., Harris, N., Rehm, H., Hamosh, A., Baynam, G., Groza, T., McMurry, J., et al. (2020) · 2020
Earlier work this paper cites.
Deep entity matching with pre-trained language models
Li, Y., Li, J., Suhara, Y., Doan, A., and Tan, W.-C. (2020) · 2020
Earlier work this paper cites.

Similar

Neural oblivious decision ensembles for deep learning on tabular data
Popov, S., Morozov, S., and Babenko, A. (2020) · 2020
Cited alongside, same era.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-t., and Riedel, S. (2020) · 2020
Cited alongside, same era.
VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain
Yoon, J., Zhang, Y., Jordon, J., and van der Schaar, M. (2020) · 2020
Cited alongside, same era.
Diabetes Prediction Using Ensembling of Different Machine Learning Classifiers
Hasan, M. K., Alam, M. A., Das, D., Hossain, E., and Hasan, M. (2020) · 2020
Cited alongside, same era.
Early and accurate detection and diagnosis of heart disease using intelligent computational model
Muhammad, Y., Tahir, M., Hayat, M., and Chong, K. T. (2020) · 2020
Cited alongside, same era.
Tabnet: Attentive interpretable tabular learning
Arik, S. Ö. and Pfister, T. (2021) · 2021
Cited alongside, same era.
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. (2021) · 2021
Cited alongside, same era.
Well-tuned simple nets excel on tabular datasets
Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. (2021) · 2021
Cited alongside, same era.
Deep contextual clinical prediction with reverse distillation
Kodialam, R., Boiarsky, R., Lim, J., Sai, A., Dixit, N., and Sontag, D. (2021) · 2021
Cited alongside, same era.
GPT Understands, Too
Original
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. (2021) · 2021
Cited alongside, same era.
True few-shot learning with language models
Perez, E., Kiela, D., and Cho, K. (2021) · 2021
Cited alongside, same era.
Prompt programming for large language models: Beyond the few-shot paradigm
Reynolds, L. and McDonell, K. (2021) · 2021
Cited alongside, same era.

Then

LIFT: Language-interfaced fine-tuning for non-language machine learning tasks
Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., yong Sohn, J., Papailiopoulos, D., and Lee, K. (2022) · 2022
Closest in time.
On embeddings for numerical features in tabular deep learning
Original
Gorishniy, Y., Rubachev, I., and Babenko, A. (2022) · 2022
Closest in time.
Why do tree-based models still outperform deep learning on typical tabular data?
Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022) · 2022
Closest in time.
Few-shot tabular data enrichment using fine-tuned transformer architectures
Harari, A. and Katz, G. (2022) · 2022
Closest in time.
Tabpfn: A transformer that solves small tabular classification problems in a second
Original
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. (2022) · 2022
Closest in time.
Machine Learning Core
Horng, S. (2022) · 2022
Closest in time.
A good prompt is worth millions of parameters? low-resource prompt-based learning for vision-language models
Jin, W., Cheng, Y., Shen, Y., Chen, W., and Ren, X. (2022) · 2022
Closest in time.
Co-training improves prompt-based learning for large language models
Lang, H., Agrawal, M. N., Kim, Y., and Sontag, D. (2022) · 2022
Closest in time.
Transfer Learning with Deep Tabular Models
Original
Levin, R., Cherepanova, V., Schwarzschild, A., Bansal, A., Bruss, C. B., Goldstein, T., Wilson, A. G., and Goldblum, M. (2022) · 2022
Closest in time.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Original
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. (2022) · 2022
Closest in time.
Rethinking the role of demonstrations: What makes in-context learning work?
Original
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022) · 2022
Closest in time.
Can Foundation Models Wrangle Your Data?
Original
Narayan, A., Chami, I., Orr, L., and Ré, C. (2022) · 2022
Closest in time.
Training language models to follow instructions with human feedback
Original
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022) · 2022
Closest in time.
Multitask prompted training enables zero-shot task generalization
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L., Biderman, S., Gao, L., Wolf, T., and Rush, A. M. (2022) · 2022
Closest in time.
Tabular data: Deep learning is not all you need
Shwartz-Ziv, R. and Armon, A. (2022) · 2022
Closest in time.
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A. and Pavlick, E. (2022) · 2022
Closest in time.
Chain of Thought Prompting Elicits Reasoning in Large Language Models
Original
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022) · 2022
Closest in time.
Deep Neural Networks and Tabular Data: A Survey
Original
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., and Kasneci, G. (2022) · 2022
Closest in time.
Why do tree-based models still outperform deep learning on typical tabular data?
Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022) · 2022
Closest in time.
Training language models to follow instructions with human feedback
Original
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022) · 2022
Closest in time.
Multitask prompted training enables zero-shot task generalization
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L., Biderman, S., Gao, L., Wolf, T., and Rush, A. M. (2022) · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…