Project · Causal inference

Advertising Incrementality & Uplift Modeling

Does an ad campaign cause incremental conversions — and whose? A complete causal-inference project: a network-interference simulation, a randomization audit, and uplift modeling on 14 million real users, with uncertainty on every number.

+59%

relative lift in conversion, 0.194% → 0.309% (95% CI +56% to +63%)

37%

of conversions in the treated group were incremental — the rest would've happened anyway

+3.20pp

causal lift for users actually exposed to the ad; the naive estimate overstates this by ~60%

82%

of incremental conversions captured by targeting just the top 20% by predicted uplift

Most A/B-testing tutorials stop at "is the difference significant?" Two harder questions matter in practice: can you trust the experiment's design, and who did the treatment actually work for — not just on average. Motivated by Meta Research's Jobs-on-Facebook experiment (Karrer, Shi & Bhole, 2021), where a naive user-randomized test measured 71.8% lift and a cluster-randomized version of the same test measured 49.7% (read the paper →), this project does two things of its own: a simulation of that mechanism, and a full causal analysis of a real, public 14-million-user ad dataset.

Part 1 · Simulation

How network interference biases a test

Built a synthetic network of 20,000 users in 400 tightly-knit clusters, where each user's outcome depends on a direct effect (were you treated?) and a spillover effect (were your neighbors treated?) — set to 1.0 and 0.6, for a true combined effect of 1.6. Randomized individuals and randomized whole clusters, 300 times each.

Fig. 1 — individual randomization recovers only the direct effect; cluster randomization recovers close to the true combined effect.

Design	Estimated lift	What it measures
Randomize individuals	1.001 ± 0.045	Direct effect only
Randomize whole clusters	1.570 ± 0.043	Direct + spillover (true total: 1.6)

Checked two ways this isn't a one-parameter trick: with competitive spillover (e.g. job seekers competing for the same openings) the bias flips sign and individual randomization overstates the effect — the direction of Meta's actual result; and cluster estimates degrade as cross-cluster ties increase (1.605 → 1.463 as ties per user grow from 0.05 to 3.0). This is a toy model with assumed parameters — it demonstrates the mechanism, not any real product's effect.

Part 2 · Real data

Is the experiment trustworthy?

The Criteo Uplift Prediction Dataset (Diemert et al., 2018, AdKDD; CC-BY-NC-SA 4.0): 13,979,592 real users randomized into an ad campaign (85%) or control (15%), with 12 anonymized features. Before measuring any effect, I audited the randomization itself: no missing values or impossible rows, a sample-ratio check matching the stated 85/15 design (chi-square p = 0.999), and covariate balance across all 12 features (max standardized difference 0.049, under the usual 0.1 threshold).

Fig. 2 — every feature balanced between treatment and control; the test was well-randomized.

Part 3 · Did the ad work?

The headline, and the exposure trap

Design stated before looking at results: conversion primary, visit secondary, Bonferroni-adjusted α = 0.025. Comparing everyone assigned to the campaign against everyone assigned to control:

Outcome	Control	Treatment	Lift	95% CI
Conversion	0.194%	0.309%	+0.115pp	0.108 to 0.122
Visit	3.82%	4.85%	+1.03pp	1.01 to 1.06

With 14M users, significance was never in doubt (z = 28.5) — the useful number is effect size: about 1.15 extra conversions per 1,000 users assigned, of which roughly 37% were incremental.

Only 3.6% of assigned users were actually shown an ad, and exposure is self-selected (exposed treated users convert at 5.4%, unexposed at 0.12% — mostly selection, not effect). Using random assignment as an instrument for exposure gives the causal effect on users exposed because they were assigned:

Fig. 3 — the naive exposed-vs-control comparison overstates the causal effect by about 60%.

For exposed users, conversion rose 3.20pp (95% bootstrap CI 3.01 to 3.39) — versus a naive 5.18pp. This rests on one untestable assumption: that assignment affects outcomes only through exposure.

Part 4 · For whom did it work?

Uplift modeling finds the 20% who matter

An average lift of 0.115pp could mean everyone benefits a little, or a few people benefit a lot. I fit three causal meta-learners (S-, T- and X-learner, on gradient boosting) on 70% of the data; every number below comes from the held-out 30% (4.19M users), with 95% CIs from 200 bootstrap resamples.

Fig. 4 — ranking users by the X-learner's predicted uplift captures incremental conversions far faster than random targeting. Hover the chart to read each curve.

Users targeted	X-learner captures	Random targeting
Top 10%	71% (66–75)	11%
Top 20%	82% (78.5–84.8)	21%
Top 30%	88% (86–91)	31%

Checked directly rather than trusting the model's own word: grouping test users into ten bins by predicted uplift and comparing treated vs. control within each bin.

Fig. 5 — the measured lift is concentrated almost entirely in the model's top decile (+0.80pp vs. an overall average of +0.115pp).

The top decile also has a much higher baseline conversion rate (1.8% vs. ~0.1% elsewhere), so part of its large absolute lift is just that these users convert more in general — absolute and relative lift tell different stories here. In practice: most of the campaign's incremental conversions come from a small slice of the audience, so a budget-limited advertiser could target the model's top 20–30% and keep most of the benefit.

Scope & Assumptions

Two estimands, two assumptions. The +0.115pp result holds under random assignment alone, verified above. The +3.20pp exposed-user result additionally assumes assignment affects outcomes only through exposure — standard for this kind of estimate, and not something that can be verified from the data directly.
One split, default hyperparameters. The headline comparison is between meta-learner architectures (X vs. T vs. S), a contrast that's robust to light tuning; CIs here reflect test-set sampling, not retraining variance. A multi-seed or k-fold check would sharpen the point estimates further.
Anonymized features, standard for public ad-exposure data — enough to show the high-uplift segment exists and is targetable, not enough to say who's actually in it.
Likely a subsample of Criteo's full release (14M rows vs. ~25M often cited elsewhere), with 11.8% of rows sharing identical feature vectors — kept, since there's no user ID to confirm they're duplicates rather than distinct users with similar profiles.
Two sources, kept separate. The interference simulation uses parameters I chose; Meta's published result motivates the question but isn't reproduced. Every quantitative result here runs on Criteo's public data only.