Post-hoc $\alpha$ Hypothesis Testing and the Post-hoc $p$-value

Understand

In traditional hypothesis testing one must pre-specify the significance level $\alpha$ to bound the `size' of the test: its probability to falsely reject the hypothesis.

• Indeed, a data-dependent selection of $\alpha$ would generally distort the size, possibly making it larger than the specified level $\alpha$.
• We explore hypothesis testing with a data-dependent choice of $\alpha$ by guaranteeing that there is no such size distortion in expectation, even if the level $\alpha$ is arbitrarily selected based on the data.
• Unlike regular $p$-values, resulting `post-hoc $p$-values' allow us to `reject at level $p$' and still provide this guarantee.