Fetching the paper…
Reading the bibliography…
The long-standing dominance of gradient-boosted decision trees on tabular data is currently challenged by tabular foundation models using In-Context Learning (ICL): setting the training data as context for the test data and predicting in a single forward pass without parameter updates.
Balance Scale
Siegler, R · 1976
Earlier work this paper cites.
Strictly proper scoring rules, prediction, and estimation
Gneiting, T. and Raftery, A. E · 2007
Earlier work this paper cites.
Random features for large-scale kernel machines
Rahimi, A. and Recht, B · 2007
Earlier work this paper cites.
A survey of hierarchical classification across different application domains
Silla, C. N. and Freitas, A. A · 2011
Earlier work this paper cites.
Efficient Estimation of Word Representations in Vector Space, September 2013
Mikolov, T., Chen, K., Corrado, G., and Dean, J · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
XGBoost: A Scalable Tree Boosting System
Chen, T. and Guestrin, C · 2016
Earlier work this paper cites.
Gaussian error linear units (gelus)
Hendrycks, D. and Gimpel, K · 2016
Earlier work this paper cites.
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G · 2016
Earlier work this paper cites.
Self-normalizing neural networks
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S · 2017
Earlier work this paper cites.
CatBoost: Gradient boosting with categorical features support, October 2018
Dorogush, A. V., Ershov, V., and Gulin, A · 2018
Earlier work this paper cites.
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks, May 2019
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., and Teh, Y. W · 2019
Earlier work this paper cites.
Language Models are Few-Shot Learners, July 2020
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020
Earlier work this paper cites.
Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbalanced Data, January 2021
Cartella, F., Anunciacao, O., Funabiki, Y., Yamaguchi, D., Akishita, T., and Elshocht, O · 2021
Earlier work this paper cites.
Self-attention between datapoints: Going beyond individual input-output pairs in deep learning
Kossen, J., Band, N., Lyle, C., Gomez, A. N., Rainforth, T., and Gal, Y · 2021
Earlier work this paper cites.
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning, April 2021
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y · 2021
Earlier work this paper cites.
Representing Numbers in NLP: A Survey and a Vision, March 2021
Thawani, A., Pujara, J., Szekely, P. A., and Ilievski, F · 2021
Earlier work this paper cites.
On embeddings for numerical features in tabular deep learning
Gorishniy, Y., Rubachev, I., and Babenko, A · 2022
Cited alongside, same era.
Why do tree-based models still outperform deep learning on tabular data?, July 2022
Grinsztajn, L., Oyallon, E., and Varoquaux, G · 2022
Cited alongside, same era.
Tabpfn: A transformer that solves small tabular classification problems in a second
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F · 2022
Cited alongside, same era.
An Explanation of In-context Learning as Implicit Bayesian Inference, July 2022
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T · 2022
Cited alongside, same era.
Sample path regularity of gaussian processes from the covariance kernel
Da Costa, N., Pförtner, M., Da Costa, L., and Hennig, P · 2023
Cited alongside, same era.
Large language models(llms) on tabular data: Prediction, generation, and understanding – a survey, 2024
Fang, X., Xu, W., Tan, F. A., Zhang, J., Hu, Z., Qi, Y., Nickleach, S., Socolinsky, D., Sengamedu, S., and Faloutsos, C · 2024
Later among the works it cites.
TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Feuer, B., Schirrmeister, R. T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C · 2024
Later among the works it cites.
Large Scale Transfer Learning for Tabular Data via Language Modeling, June 2024
Gardner, J., Perdomo, J. C., and Schmidt, L · 2024
Later among the works it cites.
TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling, November 2024
Gorishniy, Y., Kotelnikov, A., and Babenko, A · 2024
Later among the works it cites.
Réconcilier l’apprentissage profond avec les données tabulaires
Grinsztajn, L · 2024
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, August 2023
Garg, S., Tsipras, D., Liang, P., and Valiant, G · 2023
Cited alongside, same era.
Tabllm: Few-shot classification of tabular data with large language models
Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D · 2023
Cited alongside, same era.
MotherNet: A Foundational Hypernetwork for Tabular Classification, December 2023
Müller, A., Curino, C., and Ramakrishnan, R · 2023
Cited alongside, same era.
RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y · 2023
Cited alongside, same era.
Transformers learn in-context by gradient descent, May 2023
von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M · 2023
Cited alongside, same era.
Effective Long-Context Scaling of Foundation Models, November 2023
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H · 2023
Cited alongside, same era.
XTab: Cross-table Pretraining for Tabular Transformers, May 2023
Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M · 2023
Cited alongside, same era.
CARTE: Pretraining and Transfer for Tabular Learning, May 2024
Kim, M. J., Grinsztajn, L., and Varoquaux, G · 2024
Later among the works it cites.
Towards Localization via Data Embedding for TabPFN
Koshil, M., Nagler, T., Feurer, M., and Eggensperger, K · 2024
Later among the works it cites.
Transformers Can Do Bayesian Inference, August 2024
Müller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, F · 2024
Later among the works it cites.
Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks
Rubachev, I., Kartashev, N., Gorishniy, Y., and Babenko, A · 2024
Later among the works it cites.
Retrieval & Fine-Tuning for In-Context Tabular Models, June 2024
Thomas, V., Ma, J., Hosseinzadeh, R., Golestan, K., Yu, G., Volkovs, M., and Caterini, A · 2024
Later among the works it cites.
Why Tabular Foundation Models Should Be a Research Priority, June 2024
van Breugel, B. and van der Schaar, M · 2024
Later among the works it cites.
Mixture of In-Context Prompters for Tabular PFNs, May 2024
Xu, D., Cirit, O., Asadi, R., Sun, Y., and Wang, W · 2024
Later among the works it cites.
Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later, July 2024
Ye, H.-J., Yin, H.-H., and Zhan, D.-C · 2024
Later among the works it cites.
Tabflex: Scaling tabular learning to millions with linear attention
Zeng, Y., Kang, W., and Mueller, A. C · 2024
Later among the works it cites.
A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., Peng, H., Li, J., Wu, J., Liu, Z., Xie, P., Xiong, C., Pei, J., Yu, P. S., and Sun, L · 2024
Later among the works it cites.
Accurate predictions on small data with a tabular foundation model
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F · 2025
Closest in time.
A Closer Look at Deep Learning Methods on Tabular Datasets, January 2025
Ye, H.-J., Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., and Zhan, D.-C · 2025
Closest in time.