Project · Marketplace experimentation

Testing a Ride-Hail Promotion When Riders Compete for Drivers: Switchback Experiments on 19.7M NYC Trips

When riders compete for the same drivers, a standard A/B test of a promotion measures the wrong thing. A simulation built from 19.7 million real NYC ride-hail trips shows how large the error is, and what a time-window (switchback) design recovers and costs.

3.3×

overstatement by a user-level A/B test: it reports +15.0% in completed rides, while the true rollout effect is +4.6%

11×

lower error from a well-designed switchback (about 1pp vs. 10.4pp): 60–120 minute windows, a 30-minute washout, adjusted for expected demand

~5%

false positives kept by a randomization test at every window length, where the usual tests reach 15%

2.4%

smallest lift in completed rides detected in 14 days (1.8% in 28 days): small effects need long experiments

The usual A/B test assumes one user's treatment does not change another user's outcome. In a marketplace it does: riders requesting rides compete for the same drivers, so a promotion for some riders changes what happens to the others. I wanted to see how bad the bias is, how it depends on the market, and what the fix costs. There is no real marketplace I can experiment on, so this is a simulation study of the mechanism: a ride-hail market whose demand rhythm, trip durations and demand noise come from 19.7 million real trips (NYC TLC data), and whose matching logic is validated against an exact formula. It does not measure any company's system.

Part 1 · The mechanism

What the promotion does, and what a user-level test sees

A rider promotion makes treated riders request 15% more often. The truth is simulated directly: run the whole market with everyone treated, run it again with no one treated (same demand shocks), and compare completed rides over 14 days, averaged over 400 repetitions. Then run a standard user-level test with half the riders treated.

Fig. 11 — the user-level test reports the demand lift, not what a rollout delivers.

	Result
Effect of the promotion on requests	+15.0%
Requests served, everyone untreated vs. everyone treated	91.4% vs. 83.1%
True effect on completed rides	+4.60% (about 267 more rides on a base of 5,813 per day)
User-level A/B test	+14.98%, 3.3× the truth
Tests whose 95% interval contains the truth	0%
Tests declaring a significant positive effect	100%

The reason: an unserved request is lost at random, so treated and control riders have the same chance of getting a ride. The competition for drivers hits both groups equally and cancels in the comparison. The test measures the direct effect on requests (+15%) and is blind to the shared supply limit that decides what happens at rollout.

Fig. 12 — the gap depends on how tight supply is: 13.4pp at 77% of requests served, 10.4pp at 91%, 3.8pp at 99%, none with abundant drivers.

Two of my written predictions were wrong, and I kept the corrections. I expected the gap to grow with the share of riders treated; it does not (the user-level estimate is 15.3%, 14.9% and 14.8% at 10%, 50% and 90% treated). I also expected it to peak at rush hour; it does not, because supply in the model scales with demand all day.

Part 2 · The simulator

Built from real trips, checked against an exact formula

From the real data I took three things: the demand rhythm (requests per minute swing about 11-fold across a typical week), how long a driver is occupied (22.7 minutes on average, from dispatch to dropoff), and the demand noise (15-minute demand varies about 10% around the weekly rhythm, and persistently: autocorrelation 0.95 at 15 minutes and 0.75 at three hours). The file records only completed trips, so demand is a lower bound; I used the data for rhythm and noise, never for effect sizes.

The market is one zone at 1% of citywide demand. Riders request at the weekly rate times a persistent demand shock. A request is matched to an idle driver if there is one, otherwise the rider is lost. A matched driver is busy for a realistic random time and then returns. The driver pool follows a schedule equal to expected demand and does not react to treatment.

Validation: Erlang's loss formula gives the exact share of lost riders in a market like this, whatever the distribution of trip times. At 15-second time steps, the simulator is within 0.10 to 0.27 percentage points of the formula across five market sizes (a 2 to 4% relative error), and exponential and three-phase trip times give the same answer, as the formula says they should. The check caught two bugs of mine: returning drivers became matchable one step late (about 4% extra load), and a crude half-minute correction I tried only moved the error to the other side. The principled fix was finer time steps.

Part 3 · The switchback

What it recovers, and what it costs

A switchback flips the whole market between treatment and control in time windows, so treated and control riders never compete at the same moment. I ran 28 combinations of window length (15 minutes to a day), assignment (each window flipped independently, or paired windows with one treated and one control), washout (dropping the first minutes of each window from the analysis), and analysis (a plain comparison, or a regression that adjusts for each window's expected demand), with 250 simulated experiments each.

Carryover is the first cost: drivers busy at the start of a window were dispatched under the previous window's treatment.

Fig. 13 — short windows overstate the effect; a 30-minute washout removes the bias. The user-level test is off by +10.4pp.

The second cost is noise. A plain comparison of treated and control windows is extremely noisy (SD 2.8 to 12.9 percentage points), because the treated windows land on different parts of the daily cycle than the control windows. Pairing adjacent windows helps only when the paired windows are similar (1.1 points at 15 to 30 minutes, but 13.4 at eight hours). Adjusting for each window's expected demand, which is known in advance from the demand profile, brings it to 0.8 to 1.6 points at every window length.

Fig. 14 — adjusting for expected demand cuts the noise 3 to 11 times.

The best combinations (windows of 60 to 120 minutes, a 15 to 30 minute washout, adjusted) have an error (RMSE) of 0.9 to 1.0 percentage points, about 11 times lower than the user-level test. The recommended design, paired 120-minute windows with a 30-minute washout, has bias -0.02 and RMSE 1.00.

Part 4 · Honest inference

Which tests to trust, and how small a lift you can see

The usual standard errors are 8% to 21% too small, so 95% intervals cover the truth only 87% to 93% of the time. With no real effect, the usual tests produce false positives 6% to 15% of the time under complete randomization, and 15% when there are only 14 daily windows. A randomization test avoids this: it compares the observed estimate with what the estimator gives under re-drawn random assignments of the experiment's own design, so it needs no variance formula and stays valid however demand noise is correlated (Bojinov, Simchi-Levi & Zhao also propose exact randomization p-values for switchbacks).

Fig. 15 — with no real effect, the randomization test rejects 4.0% to 6.4% of the time in nine of ten cells. The tenth (8.8%) was noise: a rerun with 1,000 experiments gave 4.4%.

The price is power with few windows: at the true +4.6% effect, the test detects it 95% to 100% of the time for windows up to eight hours, but only 78% to 81% for daily windows.

Fig. 16 — the smallest lift detected with 80% power is about 2.4% in 14 days and 1.8% in 28 days.

Stress tests: the switchback's bias stays between -0.02 and +0.13pp at every supply level, while the user-level bias runs from +13.4 to +0.6pp. With trips twice as long, carryover needs about twice the window length, and a 30-minute washout keeps the bias within 0.25pp. With demand noise made as persistent as the data, the randomization test stays at 5.0% to 7.3% false positives. In practice: windows of about five trip-lengths, a washout of about one trip-length, adjust for expected demand, test by randomization, and size the experiment before running it.

Scope & Assumptions

A simulation of a mechanism. One zone, no waiting queue, a supply schedule that ignores the promotion, a promotion that changes only request rates, and riders served at random when supply is short. Real marketplaces have spatial matching, dynamic pricing and supply response, so effect sizes here say nothing about any company.
The "user-level equals the demand lift" result follows from the matching rule (unserved requests are random), a stylized mechanism.
Carryover only through busy drivers. No behavioral carryover (riders changing habits), and effects are the same within a window. The expected-demand covariate is assumed known exactly; in practice it is estimated from history.
Simulation error. 250 experiments per cell gives about ±1.4 points of noise on coverage and false-positive rates. The demand-noise model is slightly less persistent at long lags than the data (0.66 vs. 0.75 at three hours), so real standard errors could be worse.
Data. Only completed trips are observed, so demand is a lower bound. January 2024 weather events are left in as real noise.

Reproduced end-to-end by one script that writes every figure above and a results file with every number quoted here, plus step-by-step notebooks for each stage.