Sample size is the number of independent units, such as users, sessions, orders or devices, that an experiment observes in each group. In experimentation, the required sample size is calculated before the test starts so that the experiment has a high probability of detecting a real effect of a chosen size while keeping the risk of a false positive at an agreed level.
Sample size links four quantities: the significance level, the statistical power, the minimum detectable effect and the variability of the metric. Fixing any three determines the fourth, which is why sample size planning is the point where business ambition meets statistical reality.
How sample size is determined
Sample size is determined by a power analysis, which asks how many units are needed so that a true difference of a given size would produce a statistically significant result with a chosen probability. Larger samples reduce the standard error of the estimated difference, and that error shrinks with the square root of the sample size.
For comparing two means with equal groups, the required size per group is approximately n = 2 (z₁₋α/₂ + z₁₋β)² σ² / δ², where σ is the standard deviation of the metric and δ is the minimum detectable effect. With a two-sided significance level of 0.05 and power of 0.80, the constant term is about 7.85, which gives the well-known shortcut n ≈ 16 σ² / δ² per group.
The square-root relationship has an important consequence: halving the detectable effect quadruples the sample size. The table below lists the inputs to the calculation and their typical values.
| Input | Typical value | Effect of tightening it |
|---|---|---|
| Significance level (α) | 0.05 two-sided, sometimes 0.01 | Stricter α requires more units |
| Statistical power (1 – β) | 0.80 to 0.90 | Higher power requires more units |
| Minimum detectable effect | 1% to 10% relative lift | Smaller effects require far more units |
| Metric variance | Estimated from historical data | Noisier metrics require more units |
| Allocation ratio | 50/50 between two groups | Unequal splits require more units in total |
Sample size for different metric types
The metric type determines how variance enters the formula. For a conversion rate, the variance follows from the rate itself as p(1 – p), so low baseline rates need large samples to detect small relative changes. For continuous metrics such as revenue per user, variance must be estimated from historical data and is often inflated by a few large values.
| Metric type | Variance source | Practical consequence |
|---|---|---|
| Binary conversion | p(1 – p) from the baseline rate | Rare events need very large samples |
| Continuous mean | Historical standard deviation | Heavy tails raise required size sharply |
| Ratio metric | Delta method or bootstrap | Unit of analysis must match randomization |
| Count per user | Often overdispersed | Poisson assumptions understate variance |
| Percentile metric | Bootstrap estimates | Usually needs more data than the mean |
Variance reduction techniques can lower the required sample size without lowering standards. CUPED, which adjusts the outcome using a pre-experiment covariate, and capping extreme values are common choices; both can reduce variance substantially when the pre-period metric correlates well with the outcome.
How sample size relates to neighboring concepts
Sample size sits at the center of experimentation, connecting the design of an A/B test to the interpretation of its p-value. The table below clarifies how it differs from concepts it is often confused with.
| Concept | What it describes | Relation to sample size |
|---|---|---|
| Statistical power | Chance of detecting a true effect | Increases as sample size grows |
| P-value | Surprise of the data under no effect | Becomes small for tiny effects in huge samples |
| Randomization | How units are assigned to groups | Defines the unit that sample size counts |
| Control group | Baseline for comparison | Needs its own share of the total sample |
| Test duration | Calendar time of the experiment | Sample size divided by daily eligible traffic |
A frequent confusion is to treat traffic as sample size when the unit of randomization is the user. Ten page views from one user are not ten independent observations, and counting them as such produces overconfident results and false positives.
Planning sample size in practice
In practice, teams translate the required sample size into test duration by dividing it by the number of eligible units per day, then rounding up to whole weeks so that weekday and weekend behavior is represented evenly. A test that would need many months usually signals that the minimum detectable effect is unrealistic for the available traffic.
| Situation | Recommended action | Reason |
|---|---|---|
| Required duration exceeds eight weeks | Accept a larger detectable effect | Long tests suffer from drift and novelty |
| Metric is highly variable | Apply CUPED or capping | Reduces variance before adding traffic |
| Many variants are planned | Reduce the number of arms | Each arm needs its own full sample |
| Low traffic product | Use a more sensitive proxy metric | Earlier funnel steps convert more often |
| Several metrics are decisive | Adjust α for multiple comparisons | Prevents inflated false positive rates |
The sample size should be fixed and recorded before launch. Deciding afterward that a test has “enough data” because the result looks favorable defeats the purpose of the calculation.
Common mistakes with sample size
Most sample size errors either stop a test too early or misjudge the variance. Both lead to results that do not replicate when the change is rolled out.
- Peeking and stopping early when the p-value first drops below 0.05, which can multiply the false positive rate several times unless a sequential testing method is used.
- Underpowered tests, where a nonsignificant result is read as proof of no effect even though the test could not have detected a realistic one.
- Inflated effect estimates from small significant tests, because only the lucky overestimates cross the threshold.
- Ignoring sample ratio mismatch, where the observed split differs from the planned split and signals an assignment or logging defect.
- Using the wrong unit, such as sessions when users were randomized.
Explained without jargon
Sample size is simply how many people or cases you need to observe before you can trust a comparison. Flip a coin ten times and getting seven heads proves very little. Flip it ten thousand times and a seventy percent result would be very convincing.
Experiments work the same way. If the difference you are looking for is large, a small group may reveal it. If the difference is small, you need many more observations to separate it from random luck. The key rule is that smaller differences require much bigger samples, so the size must be decided before the test begins, not adjusted once the results look appealing.
A typical use case
An online retailer wants to test a redesigned checkout page. The current checkout conversion rate is 4.0%, and the product team decides that a relative lift of 10%, from 4.0% to 4.4%, is the smallest improvement worth shipping. Using a two-sided significance level of 0.05 and power of 0.80, the analyst calculates a requirement of roughly 39,500 users per variant, about 79,000 in total.
The checkout receives about 6,000 eligible users per day, so the test needs around 13 days. The team rounds up to 14 days to cover two full weeks and commits to analyzing only at the end. The final result shows 4.38% against 4.02%, a significant difference, and the redesign is rolled out with confidence that the lift is unlikely to be random noise.