2022

Gradient-based Bi-level Optimization for Deep Learning: A Survey

Chen, Can, Chen, Xi, Ma, Chen et al.

Understand

Bi-level optimization, especially the gradient-based category, has been widely used in the deep learning community including hyperparameter optimization and meta-knowledge extraction.

Bi-level optimization embeds one problem within another and the gradient-based category solves the outer-level task by computing the hypergradient, which is much more efficient than classical methods such as the evolutionary algorithm.
In this survey, we first give a formal definition of the gradient-based bi-level optimization.
Next, we delineate criteria to determine if a research problem is apt for bi-level optimization and provide a practical guide on structuring such problems into a bi-level optimization framework, a feature particularly beneficial for those new to this domain.

Built on

Mathematical programs with optimization problems in the constraints
Jerome Bracken and James T McGill · 1973
Earlier work this paper cites.
Stochastic bilevel programming in structural optimization
Snorre Christiansen, Michael Patriksson, and Laura Wynter · 2001
Earlier work this paper cites.
The CMA evolution strategy: A comparing review
Nikolaus Hansen · 2006
Earlier work this paper cites.
Learning fast approximations of sparse coding
Karol Gregor and Yann LeCun · 2010
Earlier work this paper cites.
Online learning for matrix factorization and sparse coding
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro · 2010
Earlier work this paper cites.
Market structure and equilibrium
Heinrich Von Stackelberg · 2010
Earlier work this paper cites.
Poisoning attacks against support vector machines
Original
Battista Biggio, Blaine Nelson, and Pavel Laskov · 2012
Earlier work this paper cites.
Learning recommender systems with adaptive regularization
Steffen Rendle · 2012
Earlier work this paper cites.
Learning to learn by gradient descent by gradient descent
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas · 2016
Earlier work this paper cites.
Scalable gradient-based tuning of continuous regularization hyperparameters
Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko · 2016
Earlier work this paper cites.
Hyperparameter optimization with approximate gradient
Fabian Pedregosa · 2016
Earlier work this paper cites.
Optimization as a model for few-shot learning
Sachin Ravi and Hugo Larochelle · 2016
Earlier work this paper cites.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Earlier work this paper cites.
Forward and reverse gradient-based hyperparameter optimization
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil · 2017
Earlier work this paper cites.
Understanding black-box predictions via influence functions
Pang Wei Koh and Percy Liang · 2017
Earlier work this paper cites.
Deep neural networks as gaussian processes
Original
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2017
Earlier work this paper cites.
Meta-sgd: Learning to learn quickly for few-shot learning
Original
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li · 2017
Earlier work this paper cites.
Learning gradient descent: Better generalization and longer horizons
Kaifeng Lv, Shunhua Jiang, and Jian Li · 2017
Earlier work this paper cites.
A review on bilevel optimization: from classical to evolutionary approaches and applications
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb · 2017
Earlier work this paper cites.
Learned optimizers that scale and generalize
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein · 2017
Earlier work this paper cites.
Neural ordinary differential equations
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud · 2018
Earlier work this paper cites.
Bilevel programming for hyperparameter optimization and meta-learning
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil · 2018
Earlier work this paper cites.
Approximation methods for bilevel programming
Original
Saeed Ghadimi and Mengdi Wang · 2018
Earlier work this paper cites.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Earlier work this paper cites.
Stochastic hyperparameter optimization through hypernetworks
Original
Jonathan Lorraine and David Duvenaud · 2018
Earlier work this paper cites.
On first-order meta-learning algorithms
Original
Alex Nichol, Joshua Achiam, and John Schulman · 2018
Earlier work this paper cites.
Learning to Reweight Examples for Robust Deep Learning
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun · 2018
Earlier work this paper cites.
Meta-learning with latent embedding optimization
Original
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell · 2018
Earlier work this paper cites.
Dataset distillation
Original
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros · 2018
Earlier work this paper cites.

Similar

Model-based reinforcement learning for biological sequence design
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell · 2019
Cited alongside, same era.
Deep equilibrium models
Shaojie Bai, J Zico Kolter, and Vladlen Koltun · 2019
Cited alongside, same era.
Meta-learning with differentiable closed-form solvers
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi · 2019
Cited alongside, same era.
λ \lambda opt: Learn to regularize recommender models in finer levels
Yihong Chen, Bei Chen, Xiangnan He, Chen Gao, Yong Li, Jian-Guang Lou, and Yue Wang · 2019
Cited alongside, same era.
Learning surrogate losses
Original
Josif Grabocka, Randolf Scholz, and Lars Schmidt-Thieme · 2019
Cited alongside, same era.
Generalized inner loop meta-learning
Original
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala · 2019
Cited alongside, same era.
Learning data manipulation for augmentation and weighting
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing · 2019
Cited alongside, same era.
Melu: Meta-learned user preference estimator for cold-start recommendation
Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung · 2019
Cited alongside, same era.
DARTS: Differentiable architecture search
Hanxiao Liu, Karen Simonyan, and Yiming Yang · 2019
Cited alongside, same era.
Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions
Original
Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse · 2019
Cited alongside, same era.
Understanding and correcting pathologies in the training of learned optimizers
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein · 2019
Cited alongside, same era.
Neural tangents: Fast and easy infinite neural networks in python
Original
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A Alemi, Jascha Sohl-Dickstein, and Samuel S Schoenholz · 2019
Cited alongside, same era.

Then

Deep extrapolation for attribute-enhanced generation
Alvin Chan, Ali Madani, Ben Krause, and Nikhil Naik · 2021
Later among the works it cites.
Generalized dataweighting via class-level gradient manipulation
Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou · 2021
Later among the works it cites.
Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning
Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang · 2021
Later among the works it cites.
Diva: Dataset derivative of a learning task
Original
Yonatan Dukler, Alessandro Achille, Giovanni Paolini, Avinash Ravichandran, Marzia Polito, and Stefano Soatto · 2021
Later among the works it cites.
Learning multi-objective curricula for deep reinforcement learning
Original
Jikun Kang, Miao Liu, Abhinav Gupta, Chris Pal, Xue Liu, and Jie Fu · 2021
Later among the works it cites.
Parameter prediction for unseen deep architectures
Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano · 2021
Later among the works it cites.
Meta-learning to improve pre-training
Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, and David K Duvenaud · 2021
Later among the works it cites.
Differentiable implicit soft-body physics
Original
Junior Rojas, Eftychios Sifakis, and Ladislav Kavan · 2021
Later among the works it cites.
Learning to purify noisy labels via meta soft label corrector
Yichen Wu, Jun Shu, Qi Xie, Qian Zhao, and Deyu Meng · 2021
Later among the works it cites.
An end-to-end framework for molecular conformation generation via bilevel programming
Minkai Xu, Wujie Wang, Shitong Luo, Chence Shi, Yoshua Bengio, Rafael Gomez-Bombarelli, and Jian Tang · 2021
Later among the works it cites.
Provably faster algorithms for bilevel optimization
Junjie Yang, Kaiyi Ji, and Yingbin Liang · 2021
Later among the works it cites.
Functionally regionalized knowledge transfer for low-resource drug discovery
Huaxiu Yao, Ying Wei, Long-Kai Huang, Ding Xue, Junzhou Huang, and Zhenhui Jessie Li · 2021
Later among the works it cites.
Roma: Robust model adaptation for offline model-based optimization
Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin · 2021
Later among the works it cites.
Neural tangent generalization attacks
Chia-Hung Yuan and Shan-Hung Wu · 2021
Later among the works it cites.
Ntopo: Mesh-free topology optimization using implicit neural representations
Jonas Zehnder, Yue Li, Stelian Coros, and Bernhard Thomaszewski · 2021
Later among the works it cites.
Unraveling model-agnostic meta-learning via the adaptation learning rate
Yingtian Zou, Fusheng Liu, and Qianxiao Li · 2021
Later among the works it cites.
Metafscil: A meta-learning approach for few-shot class incremental learning
Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang · 2022
Closest in time.
Continuous-time meta-learning with forward mode differentiation
Tristan Deleu, David Kanaa, Leo Feng, Giancarlo Kerg, Yoshua Bengio, Guillaume Lajoie, and Pierre-Luc Bacon · 2022
Closest in time.
Loss function learning for domain generalization by implicit gradient
Boyan Gao, Henry Gouk, Yongxin Yang, and Timothy Hospedales · 2022
Closest in time.
Fine-grained analysis of stability and generalization for modern meta learning algorithms
Jiechao Guan, Yong Liu, and Zhiwu Lu · 2022
Closest in time.
On enforcing better conditioned meta-learning for rapid few-shot adaptation
Markus Hiller, Mehrtash Harandi, and Tom Drummond · 2022
Closest in time.
Metamask: Revisiting dimensional confounder for self-supervised learning
Jiangmeng Li, Wenwen Qiang, Yanan Zhang, Wenyi Mo, Changwen Zheng, Bing Su, and Hui Xiong · 2022
Closest in time.
Meta-learning with self-improving momentum target
Jihoon Tack, Jongjin Park, Hankook Lee, Jaeho Lee, and Jinwoo Shin · 2022
Closest in time.
Design-bench: Benchmarks for data-driven offline model-based optimization
Original
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine · 2022
Closest in time.
Adversarial task up-sampling for meta-learning
Yichen Wu, Long-Kai Huang, and Ying Wei · 2022
Closest in time.
Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-experts
Original
Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang · 2022
Closest in time.
A comprehensive survey to dataset distillation
Original
Shiye Lei and Dacheng Tao · 2023
Closest in time.
Boosting causal discovery via adaptive sample reweighting
An Zhang, Fangfu Liu, Wenchang Ma, Zhibo Cai, Xiang Wang, and Tat-Seng Chua · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…