Sequential testing

On this page 7

Sequential testing is a family of statistical methods that analyze accumulating data repeatedly during an experiment and allow a valid decision to stop as soon as the evidence is strong enough, while keeping the overall false positive rate under control. Experimentation teams in Data and product analytics use it when they want to look at an A/B testing result before the planned end date without breaking the guarantees of the test.

The core problem it solves is the peeking problem: a classical fixed-horizon test assumes that the data is analyzed exactly once, at a sample size chosen in advance. Checking a standard p-value every day and stopping at the first significant result can push the real error rate from 5% to well above 20%. Sequential designs replace that single look with a stopping rule built for many looks.

How sequential testing controls error across repeated looks

Sequential testing controls error by spending the significance budget across interim analyses instead of using it all at one final analysis. Every additional look at the data is another chance for random noise to cross a threshold, so the method either raises the bar at each look or uses a test statistic whose behavior is valid at any moment.

Two broad families exist. Group sequential designs plan a fixed number of interim analyses, for example after 25%, 50%, 75% and 100% of the target sample, and assign each one a boundary through an alpha spending function. Always-valid methods such as the mixture sequential probability ratio test produce confidence sequences and p-values that remain correct no matter how often, or when, the analyst checks them.

The following table shows how the false positive rate grows when a naive fixed-horizon test is checked repeatedly at the nominal 5% level, which is the behavior sequential methods are designed to prevent.

Number of looksNominal alphaApproximate actual error rate
15%5%
25%about 8%
55%about 14%
105%about 19%
Continuous monitoring5%approaches 100% over time
Approximate false positive rate under naive repeated peeking

Main designs and stopping boundaries

The choice of design determines how conservative early stopping is and how much statistical power remains for the final look. Practitioners select a boundary shape that matches how strongly they want to protect the final analysis.

DesignHow boundaries behaveTypical use
O’Brien-FlemingVery strict early, near nominal at the endExperiments where early stops should be rare
PocockEqual threshold at every lookCases where early stopping is a priority
Lan-DeMets spendingFlexible timing of interim looksUnequal or unplanned analysis dates
mSPRTAlways-valid p-value at any timeContinuous dashboards in online platforms
Bayesian sequentialStops on posterior probability thresholdsTeams using decision-theoretic rules
Common sequential designs and their characteristics

An O’Brien-Fleming boundary might require a z-score above roughly 4 at the first of four looks but only about 2.02 at the last, so the final threshold stays close to the familiar 1.96. A Pocock boundary uses a constant value of about 2.36 at every look, which makes early stops easier but costs more power at the end. Futility boundaries can be added as well, letting a team stop early for lack of effect when the treatment is clearly not going to reach the minimum detectable effect.

Sequential testing compared with fixed-horizon tests and bandits

Sequential testing sits between the classical fixed-horizon test and the multi-armed bandit, keeping hypothesis-testing guarantees while adding flexibility in timing. The comparison below helps decide which approach fits a given question.

PropertyFixed-horizon testSequential testMulti-armed bandit
Number of analysesOneSeveral or continuousContinuous
Early stoppingNot validValid by designTraffic shifts gradually
Error controlExact at one lookExact across all looksNot the main goal
Sample needed if no effectPlanned sizeUsually larger maximumOpen-ended
Main objectiveInferenceInference with speedReward maximization
Comparison with neighboring experimentation approaches

The key trade-off is that sequential designs typically require a maximum sample size 5-30% larger than an equivalent fixed test to keep the same power, yet the expected sample size is often lower because strong effects end the test early. When the true effect is large, savings of 30-50% of traffic are realistic; when the effect is null or tiny, the test usually runs to the maximum.

Planning and running a sequential experiment

A sequential experiment must be specified before launch, just like a fixed test, because the guarantees only hold when the rules are set in advance. The plan covers the metric, the minimum detectable effect, the target power, the maximum sample size, the spending function and the timing of interim looks.

  1. Define the primary metric and the smallest effect worth detecting.
  2. Choose the alpha spending function or always-valid method.
  3. Compute the maximum sample size with the inflation factor of the design.
  4. Schedule interim analyses by information fraction, not by calendar alone.
  5. Record each look and apply the stopping rule exactly as written.

The table below summarizes typical decisions a team faces when it sets up the design.

SituationRecommended choiceReason
Stakeholders check results dailyAlways-valid mSPRTAny look remains statistically valid
Few planned reviewsO’Brien-Fleming spendingPreserves nearly full final power
Risky change that may harm usersAdd a harm boundaryStops quickly on negative impact
Strong weekly seasonalityMinimum duration of one or two weeksAvoids stopping on a biased day mix
Many metrics monitoredSequential test on one primary metricLimits multiple comparison inflation
Decision guide for configuring a sequential test

Limitations and common mistakes

Sequential testing removes the peeking problem but does not remove other sources of bias, and early stopping introduces a few of its own. The most important one is the winner’s curse: a test that stops early on a strong signal tends to overestimate the true effect size, so reported lifts need bias-adjusted estimates or confirmation.

  • Stopping before full weekly cycles captures an unrepresentative user mix and can reverse later.
  • Novelty effects inflate early results, so a quick stop may lock in a temporary gain.
  • Changing the plan mid-test, such as adding looks without a spending function, breaks error control.
  • Ignoring sample ratio mismatch at interim looks lets a broken randomization drive the decision.
  • Treating a non-significant interim result as no effect confuses insufficient evidence with a null finding.

Sequential methods also assume independent observations and a stable effect over time. Metrics with heavy tails, such as revenue per user, may need variance reduction or longer minimum durations before interim boundaries behave as intended.

Stripped to the essentials

Imagine tasting soup while it cooks. If you decide it is perfect after one lucky spoonful, you may be wrong. Sequential testing is a set of fair rules that lets you taste as often as you like while still being fair about the verdict.

In an online experiment, teams want to check results early instead of waiting weeks. Looking too often with ordinary statistics makes false wins far more likely. Sequential testing sets stricter rules for early looks, so a team can stop as soon as the evidence is truly convincing and save time without fooling itself.

Where it is applied

A subscription streaming service tests a redesigned checkout page against the current one, with a planned maximum of 400,000 visitors per variant and four interim analyses using an O’Brien-Fleming spending function. The primary metric is checkout conversion, currently around 3.2%, and the team wants to detect a relative lift of 4%.

At the second interim look, after about 50% of traffic, the z-score is 3.1, which exceeds the boundary of roughly 2.96. The team stops the experiment two weeks early and ships the new page. Because early stops tend to overstate effects, analysts report a bias-adjusted lift of about 5% rather than the raw 6.3%, and they keep a small holdout group for a month to confirm that the gain persists.