A treatment group, also called a variant group, is the set of units in an experiment that receives the intervention being tested, such as a new feature, price, message or algorithm. In experimentation, its outcomes are compared with those of a control group that does not receive the change, and the difference between the two estimates the causal effect of the intervention.
The validity of that estimate depends almost entirely on how units enter the treatment group. When assignment is random and independent of outcomes, the treatment and control groups differ only by chance and by the intervention itself, which is what allows an A/B test to support a causal conclusion.
How a treatment group produces a causal estimate
The logic of a treatment group rests on the counterfactual question: what would have happened to the same units without the change. Because nobody can observe both outcomes for the same user at the same time, an experiment uses a randomized control group as a stand-in for the unobserved counterfactual.
The resulting quantity is the average treatment effect, the mean outcome in the treatment group minus the mean outcome in the control group. A confidence interval and a p-value describe how much of that difference could plausibly come from random variation in assignment rather than from the intervention.
The components that define a treatment group are summarized below.
| Element | Meaning | Typical choice |
|---|---|---|
| Randomization unit | What is assigned to a group | User, session, device, store or region |
| Allocation ratio | Share of traffic in treatment | 50/50, or 10 to 20 percent for risky changes |
| Exposure definition | When a unit counts as treated | First time the change is actually shown |
| Assignment mechanism | How units are placed | Deterministic hash of the unit identifier |
| Duration | How long the group is observed | At least one or two full weekly cycles |
Treatment group compared with related groups
Several experimental groups share a structure but serve different purposes. Confusing them leads to wrong comparisons and misread results.
| Group | Receives | Purpose |
|---|---|---|
| Treatment group | The new intervention | Measure the effect of the change |
| Control group | The current experience | Provide the counterfactual baseline |
| Holdout group | No new features for a long period | Measure cumulative long-term impact |
| Multiple treatment arms | Different variants of the change | Compare several options at once |
| A/A test groups | The same experience | Validate the experimentation system |
In multivariate testing, each combination of factor levels forms its own treatment group, so the number of groups grows multiplicatively and each one receives a smaller share of traffic. In a multi-armed bandit, allocation to treatment arms changes during the test in favor of better performers, which improves short-term outcomes but complicates unbiased effect estimation.
Sizing and allocating a treatment group
The size of a treatment group determines the statistical power of the experiment, its probability of detecting a true effect of a given magnitude. Power depends on the minimum detectable effect, the variance of the metric, the significance level and the allocation ratio.
For a fixed total sample, a 50/50 split maximizes power. Unequal splits, such as 10 percent treatment and 90 percent control, reduce risk exposure but require a noticeably larger total sample to reach the same power. The table below shows typical design parameters.
| Parameter | Common value | Effect of a stricter setting |
|---|---|---|
| Significance level | 0.05, sometimes 0.01 | Larger sample required |
| Statistical power | 0.80, sometimes 0.90 | Larger sample required |
| Minimum detectable effect | Relative lift of about 1 to 5 percent | Halving it roughly quadruples the sample |
| Metric variance | Depends on the metric | Higher variance needs more units |
| Allocation ratio | 50/50 for efficiency | Unequal splits need more total units |
Variance reduction methods such as CUPED, which adjusts outcomes using pre-experiment behavior, can shrink the required sample considerably for metrics that are stable over time.
Threats to a valid treatment group
A treatment group is only useful when it is comparable to the control group in everything except the intervention. Several practical failures break that comparability.
| Threat | Symptom | Mitigation |
|---|---|---|
| Sample ratio mismatch | Observed split differs from planned split | Chi-square check, investigate assignment |
| Interference between units | Treated users affect control users | Randomize by cluster or region |
| Novelty effect | Early lift fades over time | Run longer, analyze by exposure week |
| Contamination | Control units see the change | Consistent hashing, exposure logging |
| Peeking | Stopping when results look good | Fixed horizon or sequential testing |
A sample ratio mismatch is a particularly strong warning sign. When a planned 50/50 split arrives as 50.8/49.2 with millions of users, the deviation is very unlikely to be chance and usually signals a bug in assignment, logging or filtering that invalidates the comparison.
Practical guidance for working with treatment groups
- Analyze by assignment, not by usage, an intention-to-treat approach that preserves randomization.
- Assign only units that can be exposed, which reduces dilution from users who never see the change.
- Fix the primary metric in advance together with guardrail metrics such as latency or error rate.
- Keep assignment sticky, so the same user stays in the same group across sessions and devices.
- Record the exact variant definition, including configuration values, for later reproducibility.
These habits keep the treatment group a clean, comparable sample, which is the condition under which an experiment can claim more than correlation.
A non-technical view
Imagine a bakery that wants to know whether a new recipe sells better. It offers the new bread in some randomly chosen shops and keeps the old bread in the others. The shops with the new bread are the treatment group. The shops with the old bread are the control group.
Because the shops were chosen by chance, the two sets are alike in every important way except the recipe. If sales rise only where the new bread is sold, the bakery can be fairly confident that the recipe caused the difference, not the weather or the season.
A concrete example
A subscription news service wants to test a shorter sign-up form. Its experimentation platform hashes each visitor identifier and places half of eligible visitors in the treatment group, which sees the three-field form, and half in the control group, which sees the existing seven-field form. The team fixes completed sign-ups as the primary metric, page load time as a guardrail and a two-week duration to cover weekly patterns.
On day three, a routine check shows a split of 52.1 to 47.9 percent, a clear sample ratio mismatch. Engineers find that a bot filter removed more control visitors because the old form loaded an extra script. After fixing the filter and restarting, the test runs cleanly. The treatment group converts at 6.3 percent versus 5.6 percent, a statistically significant lift, and the shorter form is rolled out to all visitors.