A p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true. In experimentation, such as A/B testing, the null hypothesis usually states that a change has no effect, so a small p-value indicates that the observed difference would be surprising if there were truly no effect.
The p-value is a measure of compatibility between the data and a specific model, not a direct measure of truth. It does not give the probability that the hypothesis is correct, the size of the effect or its business importance, which is why it is best read together with effect sizes and confidence intervals.
How a p-value is calculated
A p-value is calculated by comparing a test statistic from the data with the distribution that statistic would follow if the null hypothesis were true. The test statistic condenses the observed difference relative to its expected noise, for example the difference in conversion rates divided by its standard error.
- State the null hypothesis, for example equal conversion rates in control and treatment.
- Choose a test that matches the data and design, such as a two-proportion z-test or a t-test.
- Compute the test statistic from the experiment data.
- Find the probability, under the null distribution, of a statistic at least that extreme.
- Compare the p-value with a significance level chosen before the experiment.
Tests can be one-sided or two-sided. A two-sided test counts extreme results in both directions and is the usual default, because a change can harm as well as help. For a symmetric test statistic, a one-sided p-value is half the two-sided value, which is why switching to one-sided after seeing the data is a form of manipulation.
Interpreting p-values and significance levels
The significance level, written alpha, is the threshold below which a result is called statistically significant. It equals the false positive rate the experimenter accepts when the null hypothesis is true. The conventional alpha of 0.05 is a convention, not a law of nature, and many fields use stricter thresholds.
| Alpha | False positive rate under the null | Typical context |
|---|---|---|
| 0.10 | 1 in 10 | Exploratory tests, low-risk product changes |
| 0.05 | 1 in 20 | Standard A/B tests, many scientific studies |
| 0.01 | 1 in 100 | Costly or irreversible decisions |
| 0.005 | 1 in 200 | Proposed stricter default for new discoveries |
| About 0.0000003 | Five standard deviations | Particle physics discovery claims |
A p-value of 0.03 means that, if the change truly had no effect, results this extreme would occur about 3 percent of the time through random variation alone. It does not mean there is a 97 percent chance that the change works.
P-value compared with related experimentation concepts
The p-value is one piece of a larger framework that also includes statistical power, sample size, confidence intervals and the control group. Each answers a different question, and confusing them causes most interpretation errors.
| Concept | Question it answers | Relationship to the p-value |
|---|---|---|
| P-value | How surprising are the data under no effect? | The quantity itself |
| Significance level | What false positive rate is acceptable? | Threshold the p-value is compared with |
| Statistical power | How likely is detecting a real effect? | Low power makes significant results less reliable |
| Confidence interval | What effect sizes fit the data? | Excludes zero exactly when p is below alpha |
| Effect size | How large is the difference? | Independent of the p-value |
| Bayesian posterior | How probable is each hypothesis? | Different framework, needs a prior |
With very large samples, tiny and irrelevant effects become significant, while with small samples important effects may not reach significance. Planning the sample size for a minimum detectable effect, typically with 80 percent power, keeps p-values meaningful.
Common misinterpretations and misuse
P-values are among the most frequently misread statistics. The table below lists the most common errors and the correct reading.
| Misconception | Why it is wrong | Correct reading |
|---|---|---|
| P is the chance the null is true | P assumes the null is true | P measures surprise under the null |
| Significant means important | Large samples detect trivial effects | Judge the effect size separately |
| Not significant means no effect | Test may lack power | Report the confidence interval |
| P = 0.049 and P = 0.051 differ greatly | Evidence changes continuously | Report exact values, avoid hard cliffs |
| Repeated peeking is harmless | Each look adds false positive chances | Use fixed horizons or sequential methods |
Peeking at a running A/B test and stopping as soon as p falls below 0.05 can inflate the real false positive rate several times over. Multiple comparisons, such as testing twenty metrics or segments, create the same problem, because one in twenty null effects will appear significant by chance at alpha 0.05.
Using p-values responsibly in experiments
Responsible use of p-values depends on decisions made before the data arrive. Proper randomization, a declared primary metric, a fixed sample size or a valid sequential design, and a chosen alpha remove most opportunities for accidental bias.
- Declare a single primary metric and treat other metrics as exploratory.
- Correct for multiple tests with methods such as Bonferroni or Benjamini-Hochberg.
- Use sequential testing methods when early stopping is a business requirement.
- Always report the effect size with a confidence interval next to the p-value.
- Check for sample ratio mismatch, which signals broken randomization before any p-value is trusted.
A non-technical view
Imagine someone claims a coin is fair, and you flip it 20 times and get 16 heads. The p-value answers one narrow question: if the coin really were fair, how often would a result this lopsided happen just by luck?
If the answer is very rarely, you start doubting the claim that the coin is fair. A p-value never proves the coin is unfair, and it does not say how unfair it is. It only measures how surprising the result would be if nothing special were going on.
A concrete example
An online travel company tests a simplified booking form against its current form. Before launch, the team declares booking conversion as the primary metric, sets alpha at 0.05 and power at 80 percent and calculates that it needs about 26,000 visitors per group to detect an increase from 4.0 to 4.5 percent.
After two full weeks, with about 31,000 visitors per group, the control converts at 4.02 percent and the new form at 4.41 percent. A two-sided two-proportion z-test gives a p-value of about 0.015, and the 95 percent confidence interval for the lift runs from roughly 0.07 to 0.71 percentage points. The team ships the new form, but because the interval is wide, it monitors conversion after launch and avoids promising the exact observed gain. The decision rests on effect size and uncertainty, not on the p-value alone.