Guarding against Spurious Discoveries in High Dimensions

Understand

Many data mining and statistical machine learning algorithms have been developed to select a subset of covariates to associate with a response variable.

• Spurious discoveries can easily arise in high-dimensional data analysis due to enormous possibilities of such selections.
• How can we know statistically our discoveries better than those by chance? In this paper, we define a measure of goodness of spurious fit, which shows how good a response variable can be fitted by an optimally selected subset of covariates under the null model, and propose a simple and effective LAMM algorithm to compute it.
• It coincides with the maximum spurious correlation for linear models and can be regarded as a generalized maximum spurious correlation.