2019

Gradient-Based Neural DAG Learning

Lachapelle, Sébastien, Brouillard, Philippe, Deleu, Tristan et al.

Understand

We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data.

We adapt a recently proposed continuous constrained optimization formulation to allow for nonlinear relationships between variables using neural networks.
This extension allows to model complex interactions while avoiding the combinatorial nature of the problem.
In addition to comparing our method to existing continuous optimization methods, we provide missing empirical comparisons to nonlinear greedy search methods.

Built on

Early stopping - but when?
L. Prechelt · 1997
Earlier work this paper cites.
Emergence of scaling in random networks
A.-L. Barabási and R. Albert · 1999
Earlier work this paper cites.
Nonlinear Programming
D.P. Bertsekas · 1999
Earlier work this paper cites.
Causation, Prediction, and Search
P. Spirtes, C. Glymour, and R. Scheines · 2000
Earlier work this paper cites.
Optimal structure identification with greedy search
D.M. Chickering · 2003
Earlier work this paper cites.
Causal protein-signaling networks derived from multiparameter single-cell data
K. Sachs, O. Perez, D. Pe’er, D.A. Lauffenburger, and G.P. Nolan · 2005
Earlier work this paper cites.
Extremely randomized trees
P. Geurts, D. Ernst, and L. Wehenkel · 2006
Earlier work this paper cites.
A linear non-gaussian acyclic model for causal discovery
S. Shimizu, P.O. Hoyer, A. Hyvärinen, and A. Kerminen · 2006
Earlier work this paper cites.
SynTReN: a generator of synthetic gene expression data for design and analysis of structure learning algorithms
Tim Van den Bulcke, et al · 2006
Earlier work this paper cites.
Scale-free networks: a decade and beyond
A.-L. Barabási · 2009
Earlier work this paper cites.
Power-law distributions in empirical data
A. Clauset, C.R. Shalizi, and M.E.J. Newman · 2009
Earlier work this paper cites.
Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning
D. Koller and N. Friedman · 2009
Earlier work this paper cites.
Causality: Models, Reasoning and Inference
J. Pearl · 2009
Earlier work this paper cites.
On the identifiability of the post-nonlinear causal model
K. Zhang and A. Hyvärinen · 2009
Earlier work this paper cites.

Similar

Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Cited alongside, same era.
Learning Bayesian Network Structure using LP Relaxations
T. Jaakkola, D. Sontag, A. Globerson, and M. Meila · 2010
Cited alongside, same era.
Bayesian network learning with cutting planes
J. Cussens · 2011
Cited alongside, same era.
Deep sparse rectifier neural networks
X. Glorot, A. Bordes, and Y. Bengio · 2011
Cited alongside, same era.
Causal inference on discrete data using additive noise models
J. Peters, D. Janzing, and B. Schölkopf · 2011
Cited alongside, same era.
Random search for hyper-parameter optimization
James Bergstra and Yoshua Bengio · 2012
Cited alongside, same era.
Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude
T. Tieleman and G. Hinton · 2012
Cited alongside, same era.
Structural intervention distance (sid) for evaluating causal graphs
J. Peters and P. Bühlmann · 2013
Cited alongside, same era.
CAM: Causal additive models, high-dimensional order search and penalized regression
P. Bühlmann, J. Peters, and J. Ernest · 2014
Cited alongside, same era.
Identifiability of Gaussian structural equation models with equal error variances
J. Peters and P. Bühlman · 2014
Cited alongside, same era.
Causal discovery with continuous additive noise models
J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf · 2014
Cited alongside, same era.
Made: Masked autoencoder for distribution estimation
M. Germain, K. Gregor, I. Murray, and H. Larochelle · 2015
Cited alongside, same era.

Then

On estimation of functional causal models: General results and application to the post-nonlinear causal model
Kun Zhang, Zhikun Wang, Jiji Zhang, and Bernhard Schölkopf · 2015
Later among the works it cites.
Deep Learning
I. Goodfellow, Y. Bengio, and A. Courville · 2016
Later among the works it cites.
Elements of Causal Inference - Foundations and Learning Algorithms
J. Peters, D. Janzing, and B. Schölkopf · 2017
Later among the works it cites.
A million variables and more: the fast greedy equivalence search algorithm for learning high-dimensional graphical causal models, with an application to functional magnetic resonance images
J. Ramsey, M. Glymour, R. Sanchez-Romero, and C. Glymour · 2017
Later among the works it cites.
Learning from narrated instruction videos
J. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev, and S. Lacoste-Julien · 2018
Later among the works it cites.
Learning Functional Causal Models with Generative Neural Networks
O. Goudet, D. Kalainathan, P. Caillou, D. Lopez-Paz, I. Guyon, and M. Sebag · 2018
Later among the works it cites.
Generalized score functions for causal discovery
Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Schölkopf, and Clark Glymour · 2018
Later among the works it cites.
Sam: Structural agnostic model, causal discovery and penalized adversarial learning
Original
D. Kalainathan, O. Goudet, I. Guyon, D. Lopez-Paz, and M. Sebag · 2018
Later among the works it cites.
Domain adaptation by using causal inference to predict invariant conditional distributions
S. Magliacane, T. van Ommen, T. Claassen, S. Bongers, P. Versteeg, and J.M. Mooij · 2018
Later among the works it cites.
Theory of deep learning III: explaining the non-overfitting puzzle
Original
T. A. Poggio, K. Kawaguchi, Q. Liao, B. Miranda, L. Rosasco, X. Boix, J. Hidary, and H. Mhaskar · 2018
Later among the works it cites.
Dags with no tears: Continuous optimization for structure learning
X. Zheng, B. Aragam, P.K. Ravikumar, and E.P. Xing · 2018
Later among the works it cites.
The seven tools of causal inference, with reflections on machine learning
J. Pearl · 2019
Closest in time.
DAG-GNN: DAG structure learning with graph neural networks
Y. Yu, J. Chen, T. Gao, and M. Yu · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…