What is the power of a statistical test and why use it?
Power is the probability of making a correct decision (to reject the null hypothesis) when the null hypothesis is false.
To determine it, one needs to assume a particular alternative hypothesis.
Because power focuses on “making a correct decision” it is natural to use it to report to decision-makers on the results of a test.
Unfortunately, it is not used as often as it should. Instead, major focus has been given to p-values.
A p-value is the probability of observing a result as large or larger than a test result given that the null hypothesis is true. In other words, it provides a measure of evidence supporting the null hypothesis assuming it is true. This is generally a bit harder to explain.
P-value and power are diametrically opposed in that as the p-value gets smaller (suggesting that the null hypothesis is false) power gets larger (suggesting that the alternative hypothesis is true), thus increasing the probability of making a correct decision.
In this video, Julian Parris, Learning Strategy Manager for SAS/JMP software, covers the basics of the power of a statistical test and the factors that affect it.