Statistical power is the probability that a test correctly rejects the null hypothesis when a true effect of a specified size exists. In experimentation, such as A/B testing, power expresses how likely an experiment is to detect a real improvement instead of missing it, and it equals one minus the Type II error rate, written 1 – beta.
Power is a property of the experiment design, not of a single result. It is fixed before data collection by four linked quantities: the sample size, the effect size to be detected, the variance of the metric and the significance level. Experiments are conventionally planned for 80% power, and 90% is common when missing an effect is costly.
What determines statistical power
Statistical power depends on how clearly a true effect stands out from random noise in the metric. Anything that enlarges the signal or shrinks the noise raises power; anything that demands stronger evidence lowers it.
| Factor | Change | Effect on power |
|---|---|---|
| Sample size | Larger groups | Increases power |
| True effect size | Larger difference between groups | Increases power |
| Metric variance | Lower spread of outcomes | Increases power |
| Significance level alpha | Stricter, such as 0.01 instead of 0.05 | Decreases power |
| Test direction | One-sided instead of two-sided | Increases power for the stated direction |
| Traffic split | Unequal allocation, such as 90/10 | Decreases power for the same total |
Because these factors interact, fixing any three determines the fourth. In practice, teams usually fix alpha, power and the minimum detectable effect, then calculate the required sample size and translate it into experiment duration based on available traffic.
Power, sample size and minimum detectable effect
The minimum detectable effect (MDE) is the smallest true effect that the experiment can detect with the planned power. It is a design choice: the MDE should reflect the smallest change worth acting on, not the change the team hopes to see.
For two equal groups, alpha of 0.05 two-sided and 80% power, a useful approximation is that each group needs about 16 times the variance divided by the squared MDE. The squared term has a strong practical consequence: halving the MDE requires roughly four times as many users.
The table illustrates this relationship for a conversion metric with a 5% baseline rate, using the same alpha and power.
| Relative lift to detect | Absolute difference | Users per group |
|---|---|---|
| 20% | 1.0 percentage point | About 7,600 |
| 10% | 0.5 percentage point | About 30,000 |
| 5% | 0.25 percentage point | About 120,000 |
| 2% | 0.1 percentage point | About 760,000 |
These figures are approximations for planning. Exact calculations depend on the test used and the variance estimate, and metrics such as revenue per user usually have far higher variance than conversion rates, which pushes sample sizes up considerably.
Power compared with related experiment concepts
Statistical power is often confused with statistical significance and with the p-value, although they describe different things. Significance concerns the observed result; power concerns the design before any result exists.
| Concept | What it measures | When it is set or known |
|---|---|---|
| Statistical power | Chance of detecting a true effect | Planned before the experiment |
| Significance level | Accepted false-positive rate | Chosen before the experiment |
| P-value | Surprise of the data under the null | Computed after the experiment |
| Minimum detectable effect | Smallest effect worth detecting | Chosen before the experiment |
| Sample size | Number of units per group | Calculated from the other inputs |
Computing post hoc power from the observed effect adds no information, because it is a direct function of the p-value. When an experiment ends without significance, the useful question is whether the confidence interval rules out effects that would have mattered.
Ways to increase power without more users
When traffic is limited, power can still be improved by reducing the variance of the metric or sharpening the comparison. Several techniques are standard in mature experimentation programs.
- Variance reduction with pre-experiment data, such as CUPED-style covariate adjustment, often cuts required samples substantially.
- Choosing a more sensitive metric, for example click-through instead of revenue, when it is a valid proxy.
- Capping or winsorizing extreme values in heavy-tailed metrics, applied by a rule fixed in advance.
- Triggering analysis only on users who actually reached the changed feature.
- Using an even traffic split, which maximizes power for a fixed total sample.
Each technique has prerequisites. Covariate adjustment needs stable pre-period data that is unaffected by the treatment, and triggered analysis must define exposure the same way in the control group and the treatment group to avoid bias.
Consequences of low power
Underpowered experiments are a common and expensive failure. They miss real improvements, and they also distort the effects they do detect.
| Problem | What happens | Mitigation |
|---|---|---|
| Missed effects | Good changes rejected as having no effect | Plan sample size from the MDE |
| Inflated winners | Significant results overstate the true effect | Adequate power, replication |
| Sign errors | Rare results point in the wrong direction | Larger samples, holdout checks |
| Early stopping | Peeking raises false positives | Fixed horizon or sequential testing |
The effect inflation, sometimes called the winner’s curse, arises because only results that cleared the significance threshold are reported, and in a low-power test only lucky overestimates clear it. Teams that celebrate many small, underpowered wins often find that the combined impact never appears in long-term metrics.
The short version
Statistical power is the chance that an experiment notices a real difference when one exists. Think of a quiet voice in a noisy room. The more people listen, the quieter the room or the louder the voice, the more likely someone hears it.
An experiment with low power is like listening for a moment in a busy restaurant. Hearing nothing does not mean nobody spoke.
Where it makes a difference
An online retailer plans to test a redesigned checkout page. The current conversion rate is 5%, and the product team decides that a relative lift of 5% or more would justify the engineering cost of keeping the new design.
A power calculation at alpha 0.05 and 80% power shows a need for about 120,000 users per group. With 20,000 eligible visitors per day split evenly, the test must run about 12 days, which the team rounds up to two full weeks to cover weekly patterns.
An earlier test of a similar change had run for only three days and was declared a failure. The power analysis shows that it had only about 30% power for the target effect, so the team treats that result as inconclusive rather than negative and runs the properly sized test.