A control group is the set of experimental units, such as users, sessions, stores or patients, that does not receive the treatment being tested and is otherwise handled exactly like the treatment group. In experimentation, it supplies the counterfactual baseline: an estimate of what would have happened to the treated units without the change.
The causal effect of a change is measured as the difference between the treatment group and the control group. Without a properly formed control group, a measured improvement can simply reflect seasonality, marketing activity, novelty or chance rather than the change itself.
How a control group isolates a causal effect
Every outcome is influenced by many factors at once. A control group works because both groups experience the same external influences during the same period, so these influences cancel out when the groups are compared. Only the treatment differs systematically.
The key condition is exchangeability: before the treatment, the groups must be comparable in every characteristic that affects the outcome. Randomization achieves this on average by assigning units by chance, which balances both measured and unmeasured characteristics as the sample grows.
In A/B testing, the control group is usually called variant A or the baseline and sees the current experience. The estimated effect is the difference in a metric between groups, together with a confidence interval and a p-value that express the uncertainty caused by random variation.
Types of control groups
The appropriate control depends on what the experiment must rule out. The following table lists the most common forms.
| Control type | What it receives | What it rules out |
|---|---|---|
| No-treatment control | Nothing new | Effects of time and external factors |
| Placebo control | An inert imitation of the treatment | Expectation and attention effects |
| Active control | The current standard or best alternative | Whether the new option beats the existing one |
| Holdout group | No exposure to a program for a long period | Cumulative effect of many changes |
| A/A control | Identical experience in both groups | Bugs and bias in the testing system |
| Synthetic control | Weighted combination of untreated units | Trend differences when randomization is impossible |
In product experimentation, a long-term holdout of a few percent of users is a common way to measure the combined effect of a whole roadmap, because individual tests often overestimate gains that do not persist.
Designing the control group
Design decisions determine whether the control group delivers a valid comparison. The most important are the unit of randomization, the allocation ratio and the sample size.
| Parameter | Typical values | Effect of the choice |
|---|---|---|
| Allocation ratio | 50/50, sometimes 90/10 | Equal split gives the highest statistical power |
| Randomization unit | User, session, account, region | Must match how the treatment spreads |
| Sample size per group | Hundreds to millions | Set by baseline rate and minimum detectable effect |
| Test duration | At least one or two full weekly cycles | Captures weekday and weekend behavior |
| Holdout share | About 1 to 10 percent | Trades lost benefit against long-term insight |
An unequal split such as 90/10 limits risk from an untested change, but the smaller group dominates the variance, so the experiment needs considerably more total traffic to reach the same power. When users interact, as in marketplaces or social products, randomizing by cluster or region prevents treatment effects from leaking into the control group.
Threats to a valid control group
A control group can be compromised in ways that silently bias results. Experienced teams check for these threats routinely before reading any effect.
| Threat | Symptom | Remedy |
|---|---|---|
| Sample ratio mismatch | Group sizes deviate from the planned split | Investigate assignment and logging before analysis |
| Contamination | Control units receive the treatment | Stricter assignment, cluster randomization |
| Interference between units | Treated users affect control users | Randomize by cluster, region or time |
| Selection bias | Groups formed by choice, not chance | Use random assignment |
| Novelty effect | Early lift that fades | Run longer, analyze effect over time |
| Peeking | Stopping when results look good | Fixed horizon or sequential testing methods |
A sample ratio mismatch is a particularly strong warning sign. Even a small but statistically significant deviation from the planned split usually means that assignment or tracking is broken, and the results should not be trusted until the cause is found.
Control groups in observational studies
When random assignment is impossible, analysts construct a comparison group from untreated units. Methods such as matching, difference-in-differences and synthetic control try to make this group resemble the treated one, but they rely on assumptions, such as parallel trends, that cannot be fully verified. Results from such designs deserve more caution than a randomized control group.
In everyday language
A control group is the group that stays the same so that a change can be judged fairly. If a gardener wants to know whether a new fertilizer helps, half of the plants get it and half do not. Both halves get the same sun and water.
If the fertilized plants grow taller, the difference can be credited to the fertilizer, because the other plants show what would have happened anyway. Without them, the gardener could not tell whether good weather or the fertilizer made the plants grow.
Seen in the real world
A subscription streaming service wants to know whether a new recommendation layout increases weekly viewing time. It randomly assigns 500,000 active accounts, half to the new layout and half to a control group that keeps the existing layout, randomizing by account so that every device in a household sees the same version.
After two full weeks, an automated check confirms that the groups match the planned 50/50 split. The treatment group watches about 3 percent more per week, with a confidence interval that excludes zero. A smaller 2 percent long-term holdout keeps the old layout for three more months.
The holdout shows that the gain stabilizes at about 2 percent once the novelty wears off. The service ships the layout with a realistic estimate of its lasting effect instead of the inflated early figure.