2022

Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization

Ramé, Alexandre, Ahuja, Kartik, Zhang, Jianyu et al.

Understand

Foundation models are redefining how AI systems are built.

Practitioners now follow a standard procedure to build their machine learning solutions: from a pre-trained foundation model, they fine-tune the weights on the target task of interest.
So, the Internet is swarmed by a handful of foundation models fine-tuned on many diverse tasks: these individual fine-tunings exist in isolation without benefiting from each other.
In our opinion, this is a missed opportunity, as these specialized models contain rich and diverse features.

Built on

Principles of risk minimization for learning theory
Vapnik, V · 1992
Earlier work this paper cites.
Comparison of classifier selection methods for improving committee performance
Aksela, M · 2003
Earlier work this paper cites.
Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy
Kuncheva, L. I. and Whitaker, C. J · 2003
Earlier work this paper cites.
Torchvision the machine-vision package of Torch
Marcel, S. and Rodriguez, Y · 2010
Earlier work this paper cites.
Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias
Fang, C., Xu, Y., and Rockmore, D. N · 2013
Earlier work this paper cites.
Learning and transferring mid-level image representations using convolutional neural networks
Oquab, M., Bottou, L., Laptev, I., and Sivic, J · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
ImageNet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Return of frustratingly easy domain adaptation
Sun, B., Feng, J., and Saenko, K · 2016
Earlier work this paper cites.
Rethinking the inception architecture for computer vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z · 2016
Earlier work this paper cites.
Alignment for advanced machine learning systems
Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A · 2016
Earlier work this paper cites.
Software 2.0
Karpathy, A · 2017
Earlier work this paper cites.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, B., Pritzel, A., and Blundell, C · 2017
Earlier work this paper cites.
Deeper, broader and artier domain generalization
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M · 2017
Earlier work this paper cites.
iCaRL: Incremental classifier and representation learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H · 2017
Earlier work this paper cites.
Deep hashing network for unsupervised domain adaptation
Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S · 2017
Earlier work this paper cites.
Recognition in Terra Incognita
Beery, S., Van Horn, G., and Perona, P · 2018
Earlier work this paper cites.
Essentially no barriers in neural network energy landscape
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F · 2018
Earlier work this paper cites.
Averaging weights leads to wider optima and better generalization
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A · 2018
Earlier work this paper cites.
Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks
Phang, J., Févry, T., and Bowman, S. R · 2018
Earlier work this paper cites.
Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K · 2018
Earlier work this paper cites.
Invariant risk minimization
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D · 2019
Earlier work this paper cites.
Benchmarking neural network robustness to common corruptions and perturbations
Hendrycks, D. and Dietterich, T · 2019
Earlier work this paper cites.
Federated learning systems: Vision, hype and reality for data privacy and protection
Li, Q., Wen, Z., and He, B · 2019
Earlier work this paper cites.
Uniform convergence may be unable to explain generalization in deep learning
Nagarajan, V. and Kolter, J. Z · 2019
Earlier work this paper cites.
Moment matching for multi-source domain adaptation
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B · 2019
Earlier work this paper cites.
RxRx1: An image set for cellular morphological variation across many experimental batches
Taylor, J., Earnshaw, B., Mabey, B., Victors, M., and Yosinski, J · 2019
Earlier work this paper cites.
PyTorch Image Models
Wightman, R · 2019
Earlier work this paper cites.
Linear mode connectivity and the lottery ticket hypothesis
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M · 2020
Earlier work this paper cites.
A survey of deep learning applications to autonomous vehicle control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., and Fallah, S · 2020
Earlier work this paper cites.

Similar

The effect of natural distribution shift on question answering models
Miller, J., Krauth, K., Recht, B., and Schmidt, L · 2020
Cited alongside, same era.
The right way to make ratatouille
Monaco, E · 2020
Cited alongside, same era.
What is being transferred in transfer learning?
Neyshabur, B., Sedghi, H., and Zhang, C · 2020
Cited alongside, same era.
Intermediate-task transfer learning with pretrained language models: When and why does it work?
Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., and Bowman, S · 2020
Cited alongside, same era.
Towards crowdsourced training of large neural networks using decentralized mixture-of-experts
Ryabinin, M. and Gusev, A · 2020
Cited alongside, same era.
The pitfalls of simplicity bias in neural networks
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P · 2020
Cited alongside, same era.
Measuring robustness to natural distribution shifts in image classification
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L · 2020
Cited alongside, same era.
Transformers: State-of-the-art natural language processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A · 2020
Cited alongside, same era.
Ensemble of averages: Improving model selection and boosting performance in domain generalization
Arpit, D., Wang, H., Zhou, Y., and Xiong, C · 2021
Cited alongside, same era.
The iwildcam 2021 competition dataset
Beery, S., Agarwal, A., Cole, E., and Birodkar, V · 2021
Cited alongside, same era.
Reflections on foundation models
Bommasani, R. and Liang, P · 2021
Cited alongside, same era.
On the opportunities and risks of foundation models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al · 2021
Cited alongside, same era.

Then

Data determines distributional robustness in contrastive language image pre-training (CLIP)
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L · 2022
Closest in time.
No one representation to rule them all: Overlapping features of training methods
Gontijo-Lopes, R., Dauphin, Y., and Cubuk, E. D · 2022
Closest in time.
Patching open-vocabulary models by interpolating weights
Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L · 2022
Closest in time.
Combining diverse feature priors
Jain, S., Tsipras, D., and Madry, A · 2022
Closest in time.
Stop wasting my time! saving days of imagenet and BERT training with latest weight averaging
Kaddour, J · 2022
Closest in time.
Plex: Towards reliability using pretrained large model extensions
Kirsch, A. C., Lakshminarayanan, B., Hu, C. H., Sculley, D., Phan, D., Tran, D., Snoek, J. R., Liu, J., Ren, J. J., van Amersfoort, J., Han, K., Buchanan, K., Murphy, K. P., Collier, M. P., Dusenberry, M. W., Band, N., Thain, N., Jenatton, R., Rudner, T. G. J., Gal, Y., Nado, Z., Mariet, Z., Wang, Z., and Ghahramani, Z · 2022
Closest in time.
Fine-tuning can distort pretrained features and underperform out-of-distribution
Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P · 2022
Closest in time.
BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models
Langnickel, L., Schulz, A., Hammer, B., and Fluck, J · 2022
Closest in time.
Measuring and signing fairness as performance under multiple stakeholder distributions
Lopez-Paz, D., Bouchacourt, D., Sagun, L., and Usunier, N · 2022
Closest in time.
Model soups improve performance of dermoscopic skin cancer classifiers
Maron, R. C., Hekler, A., Haggenmüller, S., von Kalle, C., Utikal, J. S., Müller, V., Gaiser, M., Meier, F., Hobelsberger, S., Gellrich, F. F., et al · 2022
Closest in time.
Merging models with Fisher-weighted averaging
Matena, M. and Raffel, C · 2022
Closest in time.
Diverse ImageNet models transfer better
Nayman, N., Golbert, A., Noy, A., Ping, T., and Zelnik-Manor, L · 2022
Closest in time.
Quality not quantity: On the interaction between dataset design and robustness of CLIP
Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L · 2022
Closest in time.
Exploring mode connectivity for pre-trained language models
Qin, Y., Qian, C., Yi, J., Chen, W., Lin, Y., Han, X., Liu, Z., Sun, M., and Zhou, J · 2022
Closest in time.
Momentum-based weight interpolation of strong zero-shot models for continual learning
Stojanovski, Z., Roth, K., and Akata, Z · 2022
Closest in time.
ID and OOD performance are sometimes inversely correlated on real-world datasets
Teney, D., Lin, Y., Oh, S. J., and Abbasnejad, E · 2022
Closest in time.
Assaying out-of-distribution generalization in transfer learning
Wenzel, F., Dittadi, A., Gehler, P. V., Simon-Gabriel, C.-J., Horn, M., Zietlow, D., Kernert, D., Russell, C., Brox, T., Schiele, B., Schölkopf, B., and Locatello, F · 2022
Closest in time.
Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms
Ye, N., Li, K., Hong, L., Bai, H., Chen, Y., Zhou, F., and Li, Z · 2022
Closest in time.
Learning useful representations for shifting tasks and distributions
Zhang, J. and Bottou, L · 2022
Closest in time.
Rich feature construction for the optimization-generalization dilemma
Zhang, J., Lopez-Paz, D., and Bottou, L · 2022
Closest in time.
Git re-basin: Merging models modulo permutation symmetries
Ainsworth, S. K., Hayase, J., and Srinivasa, S · 2023
Closest in time.
Does progress on ImageNet transfer to real-world datasets?
Fang, A., Kornblith, S., and Schmidt, L · 2023
Closest in time.
Editing models with task arithmetic
Ilharco, G., Tulio Ribeiro, M., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A · 2023
Closest in time.
Dataless knowledge fusion by merging weights of language models
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P · 2023
Closest in time.
Repair: Renormalizing permuted activations for interpolation repair
Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B · 2023
Closest in time.
Linear connectivity reveals generalization strategies
Juneja, J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N · 2023
Closest in time.
Building machine learning models like open source software
Raffel, C · 2023
Closest in time.
Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Original
Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., and Cord, M · 2023
Closest in time.
Unified model for image, video, audio and language tasks
Original
Shukor, M., Dancette, C., Ramé, A., and Cord, M · 2023
Closest in time.
lo-fi: distributed fine-tuning without communication
Wortsman, M., Gururangan, S., Li, S., Farhadi, A., Schmidt, L., Rabbat, M., and Morcos, A. S · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…