Statistical significance is the conclusion that an observed result, such as a difference between a control group and a treatment group, would be unlikely to arise by chance alone if no real effect existed, judged against a threshold chosen before the analysis. In experimentation, a result is called statistically significant when its p-value falls below the significance level, conventionally 0.05, which means the test rejects the null hypothesis of no difference.
Significance is a statement about evidence against randomness, not about importance. A significant result can be too small to matter, and a non-significant result does not prove that an effect is absent; it may only mean that the experiment lacked the statistical power to detect it.
How significance is determined
Statistical significance is determined by a hypothesis test that compares the observed data with what would be expected under a null hypothesis, usually the assumption that the treatment has no effect. The test computes a test statistic, converts it into a p-value, and compares that p-value with the pre-set significance level, called alpha.
The p-value is the probability of observing a result at least as extreme as the one obtained, assuming the null hypothesis is true. If it is smaller than alpha, the result is declared significant. Alpha therefore sets the false positive rate the experimenter is willing to accept when there is truly no effect.
The table below lists the steps of a typical two-group significance test.
| Step | What happens | Example in an A/B test |
|---|---|---|
| State hypotheses | Null of no effect, alternative of an effect | Conversion rates are equal or differ |
| Choose alpha | Fix the false positive rate in advance | Alpha of 0.05, two-sided |
| Collect data | Run the experiment to the planned size | Randomized users in two variants |
| Compute statistic | Standardize the observed difference | z statistic for two proportions |
| Compare p-value with alpha | Decide whether to reject the null | p of 0.012 is below 0.05 |
Error types and the significance level
Every significance decision can be wrong in two distinct ways, and the design of an experiment is largely a balance between them. Lowering alpha makes false positives rarer but requires larger samples to keep false negatives under control.
| Outcome | Truth | Controlled by |
|---|---|---|
| Type I error (false positive) | No effect, but declared significant | Alpha, often 0.05 or 0.01 |
| Type II error (false negative) | Real effect, but not detected | Beta, often 0.2 or 0.1 |
| Correct rejection | Real effect, detected | Power, equal to 1 minus beta |
| Correct non-rejection | No effect, none declared | 1 minus alpha |
The common default of 80% power at alpha 0.05 is a convention rather than a law. Fields with costly false positives, such as clinical research or particle physics, use much stricter thresholds; particle physics, for instance, traditionally requires about five standard deviations before claiming a discovery.
Significance compared with related concepts
Statistical significance is closely tied to several other experimentation concepts, and confusing them leads to poor decisions. The table below separates their roles.
| Concept | What it measures | Relation to significance |
|---|---|---|
| P-value | Compatibility of data with the null | Compared with alpha to decide significance |
| Effect size | Magnitude of the difference | Independent of significance |
| Confidence interval | Plausible range of the effect | Excludes zero when significant |
| Statistical power | Chance of detecting a real effect | Determines how often real effects reach significance |
| Minimum detectable effect | Smallest effect the design can reliably detect | Sets the required sample size |
| Practical significance | Whether the effect matters for decisions | Judged separately from statistics |
With very large samples, even trivial differences become significant, because the standard error shrinks. Reporting the effect size with its confidence interval alongside the significance result lets decision makers see whether a significant change is also worth acting on.
Achieving valid significance in online experiments
A significance result is valid only if the experiment design matches the assumptions of the test. In online A/B testing, the most important requirements are proper randomization, a sample size fixed in advance and analysis of the metric that was declared before launch.
- Calculate sample size first: derive it from alpha, power, baseline rate and the minimum detectable effect.
- Run full business cycles: include at least one or two complete weeks to average out weekday effects and the novelty effect.
- Check sample ratio: a sample ratio mismatch between variants signals broken randomization and invalidates the test.
- Pre-register metrics: name one primary metric and the analysis method before seeing results.
- Use sequential methods when monitoring: if results will be checked repeatedly, apply sequential testing designed for it.
Randomization unit and analysis unit must also agree. If users are randomized but the metric is computed per page view, observations are correlated and naive standard errors are too small, which makes false significance more likely.
Common misuses of significance
Misuse of significance testing is widespread enough that many statisticians recommend de-emphasizing binary significant or not significant labels. The following patterns are the most frequent causes of misleading conclusions.
| Misuse | Why it misleads | Remedy |
|---|---|---|
| Peeking and stopping early | Inflates false positives well above alpha | Fixed horizon or sequential testing |
| Testing many metrics | Some will be significant by chance | Primary metric and corrections |
| Reading p as effect probability | P-value is not the chance the effect is real | Report effect sizes and intervals |
| Treating non-significance as no effect | Underpowered tests miss real effects | Check power and interval width |
| Slicing until significant | Subgroup findings are often noise | Pre-plan segments, replicate findings |
Explained without jargon
Statistical significance answers a simple question: could this result just be luck? If a coin lands heads six times out of ten, nobody is surprised, but ninety out of a hundred would make people suspect the coin is unfair.
A significant result is like that second case: the difference is too large to blame on chance easily. It still does not say whether the difference is big enough to care about, which is a separate judgment.
A typical use case
An online travel agency tests a simplified checkout page. With a baseline booking rate of 4% and a minimum detectable effect of a 10% relative lift, a power calculation calls for roughly 40,000 users per variant, so the test is planned for two full weeks.
After two weeks, the new page shows a booking rate of 4.4% against 4.0% for the original. The two-proportion test gives a p-value of about 0.005, below the pre-set alpha of 0.05, and the 95% confidence interval for the difference runs from roughly 0.1 to 0.7 percentage points.
Because the sample ratio was balanced, the primary metric was declared in advance and nobody stopped the test early, the team trusts the result. The interval shows that even the lower end of the effect is commercially worthwhile, so the new checkout is rolled out to all users.