2022

MAML and ANIL Provably Learn Representations

Collins, Liam, Mokhtari, Aryan, Oh, Sewoong et al.

Understand

Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks.

However, the mechanics of GBML have remained largely mysterious from a theoretical perspective.
In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks.
Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate.

Built on

Evolutionary Principles in Self-Referential Learning, or on Learning how to Learn: the Meta-Meta-…
Jürgen Schmidhuber · 1987
Earlier work this paper cites.
Multitask Learning
Rich Caruana · 1997
Earlier work this paper cites.
A Model of Inductive Bias Learning
Jonathan Baxter · 2000
Earlier work this paper cites.
Fast Context Adaptation via Meta-Learning
Original
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson · 2000
Earlier work this paper cites.
Multi-Step Model-Agnostic Meta-Learning: Convergence and Improved algorithms
Original
Kaiyi Ji, Junjie Yang, and Yingbin Liang · 2002
Earlier work this paper cites.
Low Rank Matrix-Valued Chernoff Bounds and Approximate Matrix Multiplication
Avner Magen and Anastasios Zouzias · 2011
Earlier work this paper cites.
Random Design Analysis of Ridge Regression
Daniel Hsu, Sham M Kakade, and Tong Zhang · 2012
Earlier work this paper cites.
The Benefit of Multitask Representation Learning
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes · 2016
Earlier work this paper cites.
Optimization as a Model for Few-Shot Learning
Sachin Ravi and Hugo Larochelle · 2016
Earlier work this paper cites.
Matching Networks for One Shot Learning
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al · 2016
Earlier work this paper cites.
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Earlier work this paper cites.
Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
Original
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li · 2017
Earlier work this paper cites.
Risk Bounds for Transferring Representations With and Without Fine-Tuning
Daniel McNamara and Maria-Florina Balcan · 2017
Earlier work this paper cites.
Prototypical networks for few-shot learning
Jake Snell, Kevin Swersky, and Richard Zemel · 2017
Earlier work this paper cites.
Meta-Learning with Differentiable Closed-form Solvers
Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi · 2018
Earlier work this paper cites.
Incremental Learning-to-Learn with Statistical Guarantees
G Denevi, C Ciliberto, D Stamos, and M Pontil · 2018
Earlier work this paper cites.
Probabilistic Model-Agnostic Meta-Learning
Chelsea Finn, Kelvin Xu, and Sergey Levine · 2018
Earlier work this paper cites.
Reptile: A Scalable Meta-Learning Algorithm
Original
Alex Nichol and John Schulman · 2018
Earlier work this paper cites.

Similar

High-Dimensional Probability: An Introduction with Applications in Data Science , volume 47
Roman Vershynin · 2018
Cited alongside, same era.
Bayesian Model-Agnostic Meta-Learning
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn · 2018
Cited alongside, same era.
How to train your MAML
Antreas Antoniou, Harri Edwards, and Amos Storkey · 2019
Cited alongside, same era.
Provable Guarantees for Gradient-Based Meta-Learning
Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar · 2019
Cited alongside, same era.
Generalize Across Tasks: Efficient Algorithms for Linear Representation Learning
Brian Bullins, Elad Hazan, Adam Kalai, and Roi Livni · 2019
Cited alongside, same era.
Online Meta-Learning
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine · 2019
Cited alongside, same era.
Improving Federated Learning Personalization via Model Agnostic Meta Learning
Original
Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan · 2019
Cited alongside, same era.
Meta-Learning with Differentiable Convex Optimization
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto · 2019
Cited alongside, same era.
Meta-Learning with Implicit Gradients
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine · 2019
Cited alongside, same era.
Efficient Meta Learning via Minibatch Proximal Update
Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng · 2019
Cited alongside, same era.
Meta-Learning with Negative Learning Rates
Alberto Bernacchia · 2020
Cited alongside, same era.
Towards Better Understanding Meta-learning Methods through Multi-task Representation Learning Theory
Original
Quentin Bouniot, Ievgen Redko, Romaric Audigier, Angélique Loesch, Yevhenii Zotkin, and Amaury Habrard · 2020
Cited alongside, same era.

Then

A Sample Complexity Separation Between Non-Convex and Convex Meta-Learning
Nikunj Saunshi, Yi Zhang, Mikhail Khodak, and Sanjeev Arora · 2020
Later among the works it cites.
Is Support Set Diversity Necessary for Meta-Learning?
Original
Amrith Setlur, Oscar Li, and Virginia Smith · 2020
Later among the works it cites.
On the Theory of Transfer Learning: The Importance of Task Diversity
Nilesh Tripuraneni, Michael Jordan, and Chi Jin · 2020
Later among the works it cites.
On the Global Optimality of Model-Agnostic Meta-Learning
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang · 2020
Later among the works it cites.
When MAML Can Adapt Fast and How to Assist When it Cannot
Sébastien Arnold, Shariq Iqbal, and Fei Sha · 2021
Later among the works it cites.
How fine-tuning allows for effective Meta-Learning
Kurtland Chua, Qi Lei, and Jason D Lee · 2021
Later among the works it cites.
Exploiting Shared Representations for Personalized Federated Learning
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai · 2021
Later among the works it cites.
Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Tasks
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar · 2021
Later among the works it cites.
Meta-Learning in Neural Networks: A Survey
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J Storkey · 2021
Later among the works it cites.
Effect of Diversity in Meta-Learning
Ramnath Kumar, Tristan Deleu, and Yoshua Bengio · 2021
Later among the works it cites.
Data augmentation for Meta-Learning
Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein · 2021
Later among the works it cites.
A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning
Nikunj Saunshi, Arushi Gupta, and Wei Hu · 2021
Later among the works it cites.
Statistically and Computationally Efficient Linear Meta-Representation Learning
Kiran K Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh · 2021
Later among the works it cites.
Provable Meta-Learning of Linear Representations
Nilesh Tripuraneni, Chi Jin, and Michael Jordan · 2021
Later among the works it cites.
Representation Learning Beyond Linear Prediction Functions
Ziping Xu and Ambuj Tewari · 2021
Later among the works it cites.
How Does the Task Landscape Affect MAML performance?
Original
Liam Collins, Aryan Mokhtari, and Sanjay Shakkottai · 2022
Closest in time.
MAML is a noisy contrastive learner in classification
Chia Hsiang Kao, Wei-Chen Chiu, and Pin-Yu Chen · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…