A holdout group is a randomly selected portion of users or customers deliberately excluded from a feature, campaign or set of changes, so that their outcomes can be compared with everyone else’s to measure the true incremental effect. In experimentation, it extends the idea of a control group beyond a single A/B test: it is often kept for weeks or months and may cover many launches at once.
Holdouts answer questions that short experiments cannot, such as whether the combined effect of a quarter’s launches is real, whether a marketing program causes purchases that would not have happened anyway, and whether an early gain persists after the novelty effect fades.
How a holdout group works
A holdout is created by randomization before exposure. A fixed share of the eligible population, often selected by hashing a stable user identifier, is assigned to the holdout and receives the old experience or no treatment. Everyone else receives the new experience, and the difference in outcomes between the two groups estimates the causal lift.
Unlike a typical A/B test, where traffic is often split evenly, a holdout is usually small, commonly about 1 to 10 percent of users, because withholding a beneficial change has a cost. The small size means the holdout must run longer or cover a large population to detect effects with adequate statistical power.
| Parameter | Typical range | What it affects |
|---|---|---|
| Holdout share | About 1-10 percent of users | Cost of withholding and precision |
| Duration | Several weeks to a few quarters | Ability to see long-term effects |
| Assignment unit | User, account, household or region | Contamination between groups |
| Scope | One feature, a program or all launches | Which effect is measured |
| Primary metric | Revenue, retention, engagement | What counts as success |
Types of holdout groups
Holdouts are used at different levels of granularity, from one feature to an entire marketing channel. The type determines what the measured difference means.
| Holdout type | What is withheld | Question answered |
|---|---|---|
| Feature holdout | One shipped feature after its A/B test | Does the effect persist over time |
| Universal holdout | All launches during a period | What is the combined effect of the roadmap |
| Campaign holdout | Marketing messages or ads | How many conversions are truly incremental |
| Model holdout | Personalization or recommendation model | How much value the model adds over a baseline |
| Geographic holdout | Treatment in selected regions | Effect when user-level randomization is impossible |
A universal holdout is particularly valuable because individual A/B tests tend to overstate their effects, through winner selection and novelty. Summing the measured lifts of many experiments usually gives a larger number than the effect the universal holdout actually shows, and the holdout is the more trustworthy figure.
Campaign holdouts play a similar role in marketing. Attribution reports credit a campaign with every conversion that followed a message, but a campaign holdout measures incrementality directly: the conversions in the exposed group minus the conversions the holdout shows would have happened without any message. The gap between attributed and incremental conversions is often substantial.
Holdout group compared with related experiment terms
Holdout, control and treatment groups are related but differ in purpose and duration. Using the terms precisely avoids confusion when results are reported.
| Concept | Definition | Difference from a holdout |
|---|---|---|
| Control group | Unexposed group in one experiment | Usually as large as the treatment and short-lived |
| Treatment group | Group receiving the change | The opposite side of the comparison |
| A/B testing | Randomized comparison of variants | Decides whether to launch; a holdout validates after launch |
| Multi-armed bandit | Adaptive allocation toward winners | Optimizes during the test, does not keep a stable baseline |
| Validation holdout set | Data withheld to evaluate a model | Concerns data in model training, not users |
The last row deserves attention: in machine learning, a holdout set is a portion of data withheld from training, which is a different concept. In experimentation, the holdout is always a group of real units that experiences the world without the change.
Designing and running a holdout
A reliable holdout requires stable assignment, clean exclusion and a predefined analysis plan. Each element protects the comparison from bias.
- Assign deterministically by hashing a persistent identifier, so users stay in the same group across sessions and devices where possible.
- Enforce exclusion everywhere: email, push messages, in-app features and paid audiences must all respect the holdout flag.
- Check the sample ratio regularly; a share that drifts from the planned value signals assignment or logging bugs.
- Calculate the minimum detectable effect in advance to set the holdout size and duration.
- Refresh long-running holdouts periodically, so the same users are not deprived for years.
Analysis compares the full holdout with the full exposed population, including people who never noticed the feature. This intention-to-treat comparison is diluted but unbiased, whereas comparing only users who engaged with a feature mixes the effect with self-selection.
Costs, limitations and pitfalls
The main cost of a holdout is the value withheld from its members, and the main risk is that the comparison quietly loses its validity. Teams should weigh both before committing to a long holdout.
- Contamination: holdout users hear about the feature from others or receive it through another channel.
- Survivorship: users who churn leave both groups differently, which biases metrics measured only on active users.
- Ethical and fairness limits: safety, security or legally required improvements should never be withheld.
- Low power: a 1 percent holdout may need millions of users to detect a lift of a fraction of a percent.
- Technical debt: maintaining an old experience for the holdout adds engineering cost.
Understanding it simply
A holdout group is a small set of people who are deliberately left out of something new. Later, everyone can see what difference the new thing really made. Think of a school trying a new reading program. It keeps a few classes on the old program and compares results at the end of the year.
Without such a comparison, nobody knows whether things improved because of the change. They might have improved anyway. The holdout gives an honest reference point, especially for effects that take months to show.
Applied in practice
A subscription streaming service sends personalized reminder emails to members who have not watched anything for a week. Click rates look excellent, but the team suspects that many of those members would have returned anyway. It creates a campaign holdout of 5 percent of eligible members, selected by hashing account IDs, who receive no reminder emails for three months.
At the end of the period, the service compares 90-day retention between the holdout and the emailed group. The difference is positive but considerably smaller than the click data suggested, because many clickers would have come back on their own. The team keeps the program, reallocates effort toward segments where the incremental lift is largest, and maintains a small rolling holdout for ongoing measurement.