A/B testing is a randomized controlled experiment in which users are randomly assigned to two versions of a product, page or message, version A (the control) and version B (the variant), so that any difference in a chosen metric can be attributed to the change itself. In experimentation, it is the standard method for establishing cause and effect in digital products, marketing and pricing.
Its power comes from randomization: because chance alone decides who sees which version, both groups are statistically equivalent in everything except the tested change. Its main constraint is statistics, since real effects are often small and must be separated from random noise with an adequate sample size and a disciplined analysis.
How an A/B test works
An A/B test follows a fixed sequence, and skipping the planning steps is the most common source of invalid results. The experiment is designed before any data is collected, not adjusted while it runs.
- Formulate a hypothesis, for example that a shorter checkout form will increase completed purchases.
- Choose one primary metric that decides the test, plus guardrail metrics that must not deteriorate, such as refund rate or page load time.
- Calculate the sample size from the baseline rate, the minimum effect worth detecting, the significance level and the desired power.
- Randomize at the right unit, usually the user rather than the page view, so each person consistently sees one version.
- Run for the planned duration, ideally whole weeks to cover weekday and weekend behavior.
- Analyze once at the end with a significance test and a confidence interval, then decide to ship, iterate or discard.
The analysis is a hypothesis test. The null hypothesis states that both versions perform equally; the p-value expresses how surprising the observed difference would be if that were true. A result is called statistically significant when the p-value falls below the preset significance level, conventionally α = 0.05.
Key statistical parameters
Four parameters determine how large an A/B test must be and how trustworthy its conclusion is. They are fixed before the test starts, and changing them afterward undermines the result.
| Parameter | Meaning | Common setting | Effect of a stricter setting |
|---|---|---|---|
| Significance level (α) | Accepted risk of a false positive | 0.05 | Fewer false positives, larger sample |
| Statistical power (1 – β) | Chance of detecting a real effect of the planned size | 0.80 | Fewer missed effects, larger sample |
| Minimum detectable effect (MDE) | Smallest effect worth detecting | Often 2-10% relative lift | Smaller MDE needs a much larger sample |
| Baseline rate | Current value of the primary metric | Measured from history | Lower rates need larger samples |
For conversion rates, the required sample per variant is approximately n = (z₁₋α/₂ · √(2p̄q̄) + z₁₋β · √(p₁q₁ + p₂q₂))² / δ², where δ is the absolute difference to detect. With a baseline of 4.0%, a target of 4.4% (a 10% relative lift), α = 0.05 and 80% power, the formula gives about 39,500 users per variant, or roughly 79,000 in total.
Because δ is squared in the denominator, halving the minimum detectable effect roughly quadruples the sample size. This is why low-traffic products often cannot detect small improvements at all and should test bolder changes.
A/B testing compared with other experiment designs
The classic two-arm A/B test is one of several designs for online experimentation. The alternatives trade statistical simplicity against speed, breadth or revenue during the test.
| Design | What it compares | Strength | Weakness |
|---|---|---|---|
| A/B test | One control and one variant | Simple, clear causal answer | Tests one idea at a time |
| A/B/n test | Control and several variants | Several ideas in one run | Needs more traffic and multiple-comparison correction |
| Multivariate test | Combinations of several elements | Measures interactions between elements | Sample size grows with every combination |
| Multi-armed bandit | Variants with adaptive traffic allocation | Less traffic spent on weak variants | Harder to obtain unbiased effect estimates |
| Before-and-after comparison | One period against another | Requires no traffic split | Not randomized, confounded by seasonality |
A before-and-after comparison is not an A/B test, even though it is often presented as one. Without a concurrent control group, a holiday, a competitor’s campaign or a marketing email can produce the observed change.
Validity threats and common mistakes
Most failed A/B tests do not fail on arithmetic but on violated assumptions. The table below lists the threats that experienced experimentation teams monitor routinely.
| Mistake | What goes wrong | Remedy |
|---|---|---|
| Peeking | Stopping as soon as p < 0.05 inflates false positives well above 5% | Fixed horizon, or sequential methods designed for monitoring |
| Sample ratio mismatch | A 50/50 split arrives as, for example, 50.8/49.2, revealing an assignment or logging bug | Chi-square check on group sizes before analysis |
| Many metrics, no correction | With 20 independent metrics, a false positive somewhere has a probability of about 64% | One primary metric, corrections for the rest |
| Novelty or primacy effect | Early behavior reflects curiosity or habit, not lasting preference | Run longer and inspect the effect over time |
| Wrong randomization unit | Users see both versions, contaminating the groups | Randomize by user or account, not by page view |
| Underpowered test | Real effects are missed or wildly overestimated | Compute sample size before launch |
A non-significant result does not prove that the versions are equal. It means that the test could not distinguish the effect from noise, and the confidence interval shows which effect sizes remain plausible.
Interpreting the result correctly
A good decision rests on the effect size and its confidence interval, not on the p-value alone. A statistically significant lift of 0.1% may not justify the engineering cost, while a non-significant but promising result may justify a larger follow-up test.
- Report the absolute and relative effect with a 95% confidence interval.
- Check guardrail metrics before shipping, because a revenue gain can hide a rise in cancellations.
- Segment cautiously. Post-hoc segment findings are hypotheses for the next test, not conclusions.
- Keep a holdout group for major launches to measure the long-term effect of several shipped changes together.
In everyday language
A/B testing is a fair contest between two versions of something. Half of the visitors, chosen at random, see the current version, and the other half see the new one. Because the split is random, the two groups are alike in every way except the version they see.
It resembles a taste test in which nobody knows which cup is which. If noticeably more people prefer one cup, and the difference is too large to be luck, then the drink itself made the difference. The trick is to ask enough people so that luck cannot explain the result.
Seen in the real world
An online store with a checkout conversion rate of 4.0% wants to know whether removing an optional account-creation step increases purchases. The team sets α = 0.05 and 80% power with a 10% minimum relative lift, which requires about 39,500 users per variant. With 20,000 checkout visitors a day, that takes about four days, but the team runs the test for two full weeks to cover weekly shopping patterns.
After 14 days, each group has about 140,000 visitors and the sample ratio check passes. The control converts 5,600 visitors, or 4.00%, and the variant converts 6,020, or 4.30%, a relative lift of 7.5%. The p-value is below 0.001, and the 95% confidence interval for the difference runs from about 0.15 to 0.45 percentage points.
Refund rates and support contacts stay flat, so the guardrails hold. The store ships the shorter checkout, and the randomized design rules out seasonality as the explanation for the gain.