Project · Experimentation

A/B Tests With 32% Fewer Users: Variance Reduction & Safe Early Stopping

How many users does a test really need, and what happens when you check it early? Variance reduction and sequential testing on a real 200,000-customer experiment, verified against simulations where the true effect is known.

13–22%

fewer users needed for the same precision, using each customer's purchase history from before the test

38%

of tests with no real effect declared a win when checked 100 times and stopped at the first p < 0.05 (target: 5%)

−32%

users per arm on average, with 7 points more power, from combining both ideas (86% power at ~25,000 users vs. 79% at 37,000)

1.33pp

average lift reported by tests that stopped early with a win, when the true lift is 1.0pp (1.7pp under naive peeking)

Every experimentation team feels two pressures: tests take too long and use too many users, and people want to look at results while the test is still running. There are standard tools for both, variance reduction (CUPED and its relatives) and sequential testing. I wanted to know how much they actually deliver on real data, and whether they stay valid. The data is a real randomized SMS campaign from the retailer X5, from the public RetailHero competition: about 200,000 customers split 50/50, a purchase rate of 60.3% in control and 63.7% in treatment (+3.3pp), and 4 million transactions of purchase history from before the campaign.

Part 1 · The data

Can the experiment be trusted?

Before any analysis I checked the randomization. The split is 100,058 vs. 99,981 customers (p = 0.86 against 50/50), and a direct test of each of the 19 pre-campaign features flags none of them. Those features come from 45.8 million purchase lines, collapsed to 4.0 million transactions: counts and spend over several windows, recency, tenure, redemption history, age and gender.

Two details would have broken the analysis silently. In the raw file, purchase_sum is the transaction total repeated on every line item, so summing it over lines inflates spend several-fold (I verified this on all 8.0 million transactions). And a clients table captured after the campaign contained a redemption date running to November 2019, which made one feature post-treatment. An audit after my first full draft caught it: that feature was the only one failing a balance test (p = 2e-7). I replaced it with "redeemed before the campaign" and reran everything. The numbers moved by less than half a point.

Part 2 · Variance reduction

What past behavior is worth

If a customer's past behavior predicts whether they buy afterward, you can subtract the predictable part and compare what is left. That is the idea behind CUPED (Deng, Xu, Kohavi & Walker, 2013), and it stays valid in a randomized test because the features are measured before assignment. I compared five estimators on the full data.

Fig. 6 — fewer users needed for the same precision, with 95% bootstrap intervals.

Method	Fewer users	95% interval
CUPED, one covariate (past transactions)	12.7%	12.5 to 13.0
Regression adjustment, 19 covariates	17.8%	17.5 to 18.0
Post-stratification on a predicted score	21.7%	21.4 to 22.0
CUPAC (CUPED with a machine-learned score)	22.0%	21.7 to 22.3

To detect a lift of	Plain test	With CUPAC
3.3pp (the observed effect)	3,352 per arm	2,616
1pp	36,945	28,833
0.5pp	147,779	115,330

A more precise estimator is worthless if it is biased or its intervals are too narrow, and real data cannot reveal the true effect. So I built experiments where it is exact: take control customers only, assign random fake arms, and inject a known effect. With no injection this is a true A/A test. Across 500, 2,000 and 5,000 users per arm, all five methods were unbiased (under 0.08pp), their 95% intervals covered the truth 93.6% to 95.6% of the time, and A/A false positives stayed between 4.7% and 5.6%.

Fig. 7 — same effect, same users: at 2,000 per arm, power rises from 58.8% (plain) to 68.8% (CUPAC).

Part 3 · Peeking

What checking early does to a test

I simulated tests with no real effect and a planned 5,000 users per arm, checking the data at evenly spaced points and declaring a win at the first p < 0.05.

Fig. 8 — false alarms climb with every extra look unless the error budget is spent across the looks.

Number of looks	1	2	5	10	20	50	100
Naive peeking	5.2%	9.0%	14.9%	19.8%	24.7%	32.6%	38.0%
O'Brien-Fleming	5.2%	5.3%	5.3%	5.2%	5.2%	5.2%	5.3%

The fix is not "never look". It is to spend the 5% error budget across the looks. I implemented Lan-DeMets alpha-spending boundaries by numerical integration and validated them by simulation: 4.9% false positives with O'Brien-Fleming boundaries and 5.1% with Pocock-type, against 25.1% for repeated 1.96 thresholds. Then I compared the rules on a test with a true 1pp lift, a planned 37,000 users per arm and 20 looks.

Rule	Power	Average users per arm
Fixed horizon (one look at the end)	79.5%	37,000
O'Brien-Fleming	76.3%	27,325 (−26%)
Pocock-type	68.8%	25,366
mSPRT (always valid)	47.2%	29,779
Naive peeking	88.7%	16,224, with 25% false positives

O'Brien-Fleming gives up about 3 points of power but stops 26% earlier on average, because clear wins are declared early. mSPRT (Johari et al.) needs no planned horizon and is valid under continuous monitoring, but when the horizon is known it pays a large power price, and it is sensitive to its prior.

Part 4 · Both together

Fewer users and more power, plus a caution

Each CUPAC point in the chart sits above and to the left of its plain counterpart: more power and fewer users.

Fig. 9 — for a true 1pp lift, CUPAC plus O'Brien-Fleming moves a test from 79.5% power at 37,000 users to 86.2% at about 25,000.

Setup	Power	Average users per arm
Plain, fixed horizon	79.5%	37,000
CUPAC, fixed horizon	88.8%	37,000
CUPAC + O'Brien-Fleming	86.2%	24,966 (−32.5%)

The result held across 10 independent seeds (new score model, new folds, new simulated experiments each time): 86.2% power (seed-to-seed SD 1.8 points) with 25,064 users per arm on average. It also does not depend on a complex model: a shallow tree model or a plain logistic regression gave 21.5% and 21.1% fewer users, against 22.0% for gradient boosting. The value comes from using purchase history at all.

One caution. Among tests that stopped and declared a win, the average estimated lift was well above the true 1.0pp.

Fig. 10 — stopping early overstates the winning effect: 1.13pp (fixed horizon), 1.33 (O'Brien-Fleming), 1.58 (Pocock-type), 1.67 (naive peeking), 1.70 (mSPRT).

Selecting on "statistically significant" inflates the estimate even in a fixed-horizon test, and stopping early on a lucky streak inflates it further. A team that stops early should report a bias-adjusted estimate or a confidence sequence, not the raw number at the stopping point.

Scope & Assumptions

One experiment, one population. The 13–22% saving reflects how well past purchases predict future purchases for this retailer. Where history is less predictive, the saving will be smaller.
Simulated arrival and effects. The data do not record the order customers entered the experiment, so sequential tests use random arrival order. Injected effects are the same for everyone, while real effects vary.
Approximations. The CUPED coefficient is re-estimated from the data seen so far at each look (false positives held at 5.0% in the simulations, but this is not a proof). mSPRT uses plug-in variances and only 20 looks, so truly continuous monitoring would raise its error rate, still bounded by 5% in theory.
Simulation error. Power figures in the 1pp scenario carry about ±1 point of Monte Carlo error; seed-to-seed variation is reported above.
Assumed campaign date. The campaign date is not given, so I used the day after the last recorded transaction (2019-03-19). Features that depend on it (recency, redemption before the campaign) would shift slightly under a different date.
Data terms. Public RetailHero competition data, obtained from a public mirror. Its license is unconfirmed, so it is analyzed and cited here, not redistributed.