Can we trust the bootstrap in high-dimension?

Understand

We consider the performance of the bootstrap in high-dimensions for the setting of linear regression, where $p<n$ but $p/n$ is not close to zero.

• We consider ordinary least-squares as well as robust regression methods and adopt a minimalist performance requirement: can the bootstrap give us good confidence intervals for a single coordinate of $\beta$? (where $\beta$ is the true regression vector).
• We show through a mix of numerical and theoretical work that the bootstrap is fraught with problems.
• Both of the most commonly used methods of bootstrapping for regression -- residual bootstrap and pairs bootstrap -- give very poor inference on $\beta$ as the ratio $p/n$ grows.