2015

Analysis of Testing-Based Forward Model Selection

Kozbur, Damian

Understand

This paper introduces and analyzes a procedure called Testing-based forward model selection (TBFMS) in linear regression problems.

This procedure inductively selects covariates that add predictive power into a working statistical model before estimating a final regression.
The criterion for deciding which covariate to include next and when to stop including covariates is derived from a profile of traditional statistical hypothesis tests.
This paper proves probabilistic bounds, which depend on the quality of the tests, for prediction error and the number of selected covariates.

Built on

Resume de la theorie metrique des produits tensoriels topologiques
A. Grothendieck · 1953
Earlier work this paper cites.
Stepwise regression–a backward and forward look
M. Efroymson · 1966
Earlier work this paper cites.
A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
H. White · 1980
Earlier work this paper cites.
A simple, positive semi-definite heteroskedasticity and autocorrelation consistent covariance matrix
W. K. Newey and K. D. West · 1987
Earlier work this paper cites.
Heteroskedasticity and autocorrelation consistent covariance matrix estimation
D. W. K. Andrews · 1991
Earlier work this paper cites.
A statistical view of some chemometrics regression tools
I. E. Frank and J. H. Friedman · 1993
Earlier work this paper cites.
Experiments with a new boosting algorithm
Y. Freund and R. E. Schapire · 1996
Earlier work this paper cites.
Regression shrinkage and selection via the lasso
R. Tibshirani · 1996
Earlier work this paper cites.
Micro–level estimation of poverty and inequality
C. Elbers, J. O. Lanjouw, and P. Lanjouw · 2003
Earlier work this paper cites.
Self-normalized Cramèr-type large deviations for independent random variables
B.-Y. Jing, Q.-M. Shao, and Q. Wang · 2003
Earlier work this paper cites.
Least angle regression
B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani · 2004
Earlier work this paper cites.
Greed is good: algorithmic results for sparse approximation
J. A. Tropp · 2004
Earlier work this paper cites.
Boosting for high-dimensional linear models
P. Bühlmann · 2006
Earlier work this paper cites.
High dimensional graphs and variable selection with the lasso
N. Meinshausen and P. Buhlmann · 2006
Earlier work this paper cites.
The Dantzig selector: statistical estimation when p p is much larger than n n
E. Candès and T. Tao · 2007
Earlier work this paper cites.
Forecasting economic time series using targeted predictors
J. Bai and S. Ng · 2008
Earlier work this paper cites.
Can one estimate the unconditional distribution of post-model-selection estimators?
H. Leeb and B. M. Pötscher · 2008
Earlier work this paper cites.

Similar

On sparse reconstruction from fourier and gaussian measurements
M. Rudelson and R. Vershynin · 2008
Cited alongside, same era.
The sparsity and bias of the lasso selection in high-dimensional linear regression
C.-H. Zhang and J. Huang · 2008
Cited alongside, same era.
Simultaneous analysis of Lasso and Dantzig selector
P. J. Bickel, Y. Ritov, and A. B. Tsybakov · 2009
Cited alongside, same era.
Conditional Cash Transfers: Reducing Present and Future Poverty
A. Fiszbein, N. Schady, F. Ferreira, M. Grosh, N. Keleher, P. Olinto, and E. Skoufias · 2009
Cited alongside, same era.
Elements of Statistical Learning: Data Mining, Inference, and Prediction
T. Hastie, R. Tibshirani, and J. Friedman · 2009
Cited alongside, same era.
Forward regression for ultra-high dimensional variable screening
H. Wang · 2009
Cited alongside, same era.
On the consistency of feature selection using greedy least squares
T. Zhang · 2009
Cited alongside, same era.
Variable selection in nonparametric additive models
J. Huang, J. L. Horowitz, and F. Wei · 2010
Cited alongside, same era.
Statistics for High-Dimensional Data: Methods, Theory and Applications
P. Bühlmann and S. van de Geer · 2011
Cited alongside, same era.
Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection
A. Das and D. Kempe · 2011
Cited alongside, same era.
Adaptive forward-backward greedy algorithm for learning sparse representations
T. Zhang · 2011
Cited alongside, same era.
Sparse models and methods for optimal instruments with an application to eminent domain
A. Belloni, D. Chen, V. Chernozhukov, and C. Hansen · 2012
Cited alongside, same era.

Then

When is correlation transitive? url: https://terrytao.wordpress.com/2014/06/05/when-is-correlation-transitive/, June 2014
T. Tao · 2014
Later among the works it cites.
Exact Post-Selection Inference for Sequential Regression Procedures
R. J. Tibshirani, J. Taylor, R. Lockhart, and R. Tibshirani · 2014
Later among the works it cites.
On asymptotically optimal confidence regions and tests for high-dimensional models
S. van de Geer, P. Bühlmann, Y. Ritov, and R. Dezeure · 2014
Later among the works it cites.
Confidence intervals for low dimensional parameters in high dimensional linear models
C.-H. Zhang and S. S. Zhang · 2014
Later among the works it cites.
Selective Sequential Model Selection
W. Fithian, J. Taylor, R. Tibshirani, and R. Tibshirani · 2015
Closest in time.
Inference in high-dimensional panel models with an application to gun control
A. Belloni, V. Chernozhukov, C. Hansen, and D. Kozbur · 2016
Closest in time.
Best subset selection via a modern optimization lens
D. Bertsimas, A. King, and R. Mazumder · 2016
Closest in time.
On cross-validated Lasso
D. Chetverikov, Z. Liao, and V. Chernozhukov · 2016
Closest in time.
Sequential selection procedures and false discovery rate control
M. G. G’Sell, S. Wager, A. Chouldechova, and R. Tibshirani · 2016
Closest in time.
Community detection in sparse networks via Grothendieck’s inequality
O. Guédon and R. Vershynin · 2016
Closest in time.
Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
T. Hastie, R. Tibshirani, and T. R · 2017
Closest in time.
Accumulation tests for fdr control in ordered hypothesis testing
A. Li and R. F. Barber · 2017
Closest in time.
L2 Boosting for Economic Applications
Y. Luo and M. Spindler · 2017
Closest in time.
A one covariate at a time, multiple testing approach to variable selection in high-dimensional linear regression models
A. Chudik, G. Kapetanios, and M. H. Pesaran · 2018
Closest in time.
Universal basic incomes versus targeted transfers: Anti-poverty programs in developing countries
R. Hanna and B. A. Olken · 2018
Closest in time.
Retooling poverty targeting using out-of-sample validation and machine learning
A. Nichols and L. McBride · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…