In an earlier blog, we discussed the power of a statistical test. We repeat that blog with the addition, following the video link, of referral to a suite of calculators that you can use to calculate the power in many applications you may face.
First, the original blog.
What is the power of a statistical test and why use it?
Power is the probability of making a correct decision (to reject the null hypothesis) when the null hypothesis is false.
To determine it, one needs to assume a particular alternative hypothesis.
Because power focuses on “making a correct decision,” it is a natural to use it to report to decision makers on the results of a test.
Unfortunately, it is not used as often as it should. Instead, major focus has been given to p-values.
A p-value is the probability of observing a result as large or larger than a test result given that the null hypothesis is true. In other words, it provides a measure of evidence supporting the null hypothesis assuming it is true. This is generally a bit harder to explain.
P-value and Power are diametrically opposed in that as the p-value gets smaller (suggesting that the null hypothesis is false) power gets larger (suggesting that the alternative hypothesis is true), thus increasing the probability of making a correct decision.
In this video, Julian Parris, Learning Strategy Manager for SAS/JMP software, covers the basics of the power of a statistical test and the factors that affect it.
Now let’s discuss a valuable resource of calculators for the power of various statistical tests.
The calculators can be found here.
This site, maintained by HyLown Consulting, LLC out of Atlanta, Georgia, is referred to as “the ultimate resource for those seeking power and sample size calculations.”
The calculators are easy to use, totally online, display the underlying formulas with references, and cover many types of comparisons from comparing a single mean to a reference value, comparing k proportions, to comparing test time to event data.
Here is the list of the calculators currently available:
- Test 1 Mean
- Comparing 2 Means
- Comparing k Means
- Test 1 Proportion
- Comparing 2 Proportions
- Comparing Paired Proportions
- Comparing K Proportions
- Test Time-To-Event Data
- Test Odds Ratio
- Test Relative Incidence in Self-Controlled Case Studies
- Other: 1-Sample Normal, 1-Sample Binomial
Let’s demonstrate the use of a calculator for the example in the Power of a Test video referred to earlier.
Julian’s example in the video was necessarily a simple one in order to cover the basics that drive the power of a test. His example used a one sample two-sided test for a mean against a hypothesized mean or reference value.
In other words: H0: µ = µ0
H1: µ ≠ µ0
So, we should use the first calculator on the list: Test 1 Mean. Under that heading there are four options:
- 1-Sample, 2-Sided Equality
- 1-Sample, 1-Sided
- 1-Sample Non-Inferiority or Superiority
- 1-Sample Equivalence
Without going into the other choices, the one needed for the example in the video is 1-Sample, 2-Sided Equality, shown here.
When you open the calculator, you will see pre-filled values as shown below. We won’t cover the associated graphs in this discussion, but they are quite useful for testing the sensitivity to your assumptions.
At the top, you can calculate Sample Size required for a given power, or Power for a given sample size. Currently, the option is on Sample Size.
In order to proceed, you must choose values that reflect your situation: Power desired, the Type 1 error rate, what the given alternative mean value or True mean is, the mean value you are testing against, the Null Hypothesis mean, and the variation or Standard Deviation you expect.
In the video, the hypothesized mean is an IQ of 100. And the alternative means are 120 or 140. While we don’t know the other values needed, we will assume reasonable values and use the calculator to illustrate Julian’s three cases where power is increased:
- Effect Size
- Population variability (Standard Deviation)
- Sample Size
For case 1, Effect Size on power, we’ll need a Type 1 error rate, say 5%, a Population Standard Deviation to start, say 40, and a sample size of say 30.
Let’s plug those values and the means 100 and 120 into the Calculator and solve for Power.
We get a Power = 0.7827. To illustrate the change in effect size on power, let’s change the True Mean to 140.
Now we see that the power has increased to Power = 0.9998.
To illustrate case 2, reduction in population variability, we’ll use the initial values of sample size = 30, error rate = 5%, means = 100 and 120, and reduce the standard deviation from 40 to 30.
This results in an increase in Power to 0.9549.
To illustrate case 3, increase in sample size, we again use the initial values but increase sample size from 30 to 40.
The resulting power is increased to 0.8859.
To illustrate another use of the calculator, suppose you wanted to know what sample size would be required to have a 90% power assuming all the other values remained as initial.
As shown, you would need to plan on a sample size of at least 42.
We have chosen to use the simplest calculator to match the discussion in Julian’s video.
We encourage you to familiarize yourself with some of the other calculators that may match conditions you will face in some of your projects going forward.