A/B Testing for Analysts: Sample Size First, Result Second

1. What an A/B test is actually deciding

Before the definition: you change a button and conversions go up. Name the specific reason that is not yet evidence the button did it, before reading on.

Conversions move on their own. Different days, different traffic sources, different weather. An A/B test removes those explanations by splitting traffic randomly and at the same time, so the only systematic difference between the two groups is the change you made. Everything else, known and unknown, is distributed evenly between them by the coin flip.

That randomization is the whole engine. It is what turns "the number went up" into "the change caused the number to go up", and it is why an A/B test can support a causal claim when a before-and-after comparison cannot. The general version of that argument is in correlation vs causation; a controlled experiment is the tool that resolves it.

The vocabulary, once, so the rest of the page reads cleanly. The control is the current version. The variant or treatment is the new one. The metric is the one number you decided in advance to judge on. And the split should be genuinely random per visitor, held constant so a returning visitor sees the same version, and running over the same period for both.

2. The worked test: 4.0 percent against 5.0 percent

Before the arithmetic: here is a finished test. Decide whether you would ship the variant, before any statistics.

The test statistic is built the same way as any other: an observed gap divided by how much gap random splitting produces. For two rates, the standard error uses a pooled rate, because the null hypothesis says both arms come from the same underlying rate.

pooled rate = (160 + 200) / (4000 + 4000) = 0.045

standard error = sqrt( 0.045 × 0.955 × (1/4000 + 1/4000) )
               = sqrt( 0.045 × 0.955 × 0.0005 )
               = 0.0046355

z = (0.05 − 0.04) / 0.0046355 = 2.157
p = 0.031

import numpy as np
from scipy import stats

nc, kc, nv, kv = 4000, 160, 4000, 200
pc, pv = kc/nc, kv/nv
pool   = (kc + kv) / (nc + nv)
se     = np.sqrt(pool * (1 - pool) * (1/nc + 1/nv))
z      = (pv - pc) / se
p      = 2 * (1 - stats.norm.cdf(abs(z)))
print(round(z, 3), round(p, 4))     # 2.157 0.0310

So p = 0.031, under the usual threshold. Before you ship it, section five asks a question that changes the answer, and it is the question most tests never get asked.

3. Reading the result: absolute, relative, and the interval

Before the numbers: the rate went from 4.0 percent to 5.0 percent. Say how big that improvement is, in a sentence, before reading on. There are two correct answers and they sound very different.

It is 1.0 percentage point in absolute terms and 25 percent in relative terms. Both are true, they are the same fact, and the second one sounds roughly twenty-five times more exciting. This is the single most common way test results get oversold, and usually nobody involved intends it.

Report the two rates and both lifts, and say "percentage points" out loud when you mean them. A slide that says "up 25 percent" next to a rate of 5 percent has told the reader nothing about which 25 percent it means.

Then add the interval, which is the number the meeting actually needs.

unpooled se = sqrt( 0.04 × 0.96 / 4000 + 0.05 × 0.95 / 4000 ) = 0.004634
95% interval on the difference = 0.01 ± 1.96 × 0.004634
                               = 0.0009 to 0.0191
                               = +0.09 to +1.91 percentage points

So the data is consistent with a gain of a tenth of a percentage point and with a gain of nearly two. In relative terms that is roughly 2 percent to 48 percent, and the width of that range is the honest state of knowledge. If shipping only pays off above a 10 percent relative gain, this test has not answered the question, even though its p-value cleared 0.05.

Say out loud why the interval is more useful here than the p-value. The p-value said "probably not nothing". The interval says "somewhere between barely worth it and excellent", which is the sentence a decision can be built on.

4. Sample size, decided before the test starts

Before the calculation: you convert at 4 percent and you would ship anything that improves it by a fifth. Guess how many visitors per arm that needs, before reading on. Almost everybody guesses low.

It needs 10,317 per arm, or 20,634 in total. At a thousand visitors a day per arm that is a ten-day test, and at a hundred a day it is a hundred-day test that should never be started.

Four inputs decide it, and three of them are choices you make.

1. Baseline rate. What you convert at now. 4 percent here. This one is a measurement, not a choice.
2. Minimum detectable effect. The smallest improvement worth shipping. A fifth of 4 percent is 0.8 percentage points, so the target rate is 4.8 percent.
3. Significance level. How often you will accept a false alarm. Conventionally 5 percent.
4. Power. How often you want to catch a real effect of that size. Conventionally 80 percent, which means one real effect in five goes undetected.

from scipy import stats
import numpy as np

def per_arm(p0, p1, alpha=0.05, power=0.80):
    za, zb = stats.norm.ppf(1 - alpha/2), stats.norm.ppf(power)
    pbar = (p0 + p1) / 2
    num = (za * np.sqrt(2 * pbar * (1 - pbar)) +
           zb * np.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
    return np.ceil(num / (p1 - p0) ** 2)

per_arm(0.04, 0.048)      # 10317.0

Here is the table worth pinning somewhere. Visitors needed per arm, at 5 percent significance and 80 percent power.

Two patterns are worth reading off it. Smaller effects are dramatically more expensive: halving the effect you want to detect roughly quadruples the traffic you need, which is the same square root relationship as the margin of error. And low baseline rates cost more than high ones, which is why checkout-page tests finish and homepage-signup tests do not.

Now picture your own site's traffic on the page you most want to test. Take your monthly visitors, halve it, and find the nearest cell in that table. What is the smallest improvement you could actually detect in a month?

5. What your test can and cannot detect

Before the number: our finished test had 4,000 per arm, and the table says a 20 percent lift needs 10,317. Predict what that mismatch means for the result we already read.

It means the test was underpowered. With 4,000 per arm, the chance of detecting a true 20 percent lift was 41 percent. If that improvement were genuinely there, this test would have missed it more often than found it.

That does not make our p = 0.031 wrong, and it does change how to read it. An underpowered test that comes back significant has, by construction, observed an effect larger than the truth, because only large observed effects clear the bar at that sample size. Ours observed 25 percent. The true effect, if any, is probably smaller, and the interval already told us so by stretching down to 2 percent.

The name for this is the winner's curse, and it is why organizations that run many small tests see their measured wins fail to show up in the quarterly numbers. Each test was analysed correctly. The set of tests that got shipped was selected for having overshot.

The fix is the discipline in section four, and there is no clever analysis that substitutes for it. Decide the sample size, run to it, then look. If the traffic is not there, test something with a bigger expected effect, or test on a page with more traffic, or accept that this particular question is not answerable by experiment at your scale.

6. Peeking, and the 24 percent

Before the simulation: you set the test to run for 10 days and you check it every day. Say why that is different from checking once at the end, before reading on.

Checking once means running one test. Checking twenty times and stopping the first time it looks good means running twenty tests and keeping whichever one crossed the line. The p-value threshold was designed for the first situation.

Here is what that costs, measured rather than asserted. I simulated 2,000 tests where both versions convert at exactly 4 percent, so any winner is a false alarm. Each test ran to 4,000 visitors per arm, checked every 200 visitors, twenty checks in total.

Five percent is exactly what the 0.05 threshold promises. Twenty-four percent is what you get instead when you watch. Nearly one test in four declares a winner that does not exist, and every one of those results will look completely ordinary in the write-up.

The chart at the top of this page is one of those runs. Both versions convert at exactly 4 percent. The p-value dipped under 0.05 at checks eight, nine and ten, bottoming out at 0.022, then climbed back and finished at 0.375. Three consecutive days of "significant" in a test where the two versions are identical.

Three ways to handle it, in order of how much I like them.

Fix the sample size and do not look. Free, correct, and the reason section four exists. Look at data quality during the run if you must look at something.

Use a method built for continuous monitoring. Sequential testing and always-valid intervals exist precisely so that watching is legitimate. They cost some power in exchange, and several testing platforms implement them.

Peek but only act at the planned end. Workable if the organization can tolerate seeing a number without acting on it. In my experience it cannot, which is why the first option is the one I actually recommend.

7. The failures that are not statistics

Before the list: the arithmetic on this page is all correct and the test can still be worthless. Name one way, before reading mine.

The split is not even. You expected 50/50 and got 4,000 against 3,780. That is a sample ratio mismatch, and it means the randomization or the logging is broken. It invalidates the test, because whatever caused the imbalance may also have caused the difference you are measuring. Check it before you look at the metric, every time, and treat a mismatch as a bug rather than a rounding issue.

Novelty and primacy. Regular users react to a change because it is a change. A new design can win in week one because it is unfamiliar and lose in week four when it stops being new. Run for whole weeks, and if you can, check whether the effect is different for new and returning visitors.

Slicing until something is significant. Twenty segments at 5 percent gives a 64 percent chance one of them fires by accident. Reporting "no overall effect but a strong win for mobile users in Canada" is the multiple comparison problem in its natural habitat, covered in the p-values guide. Decide segments in advance or treat them as ideas for the next test.

Testing the wrong metric. Click-through on a button is easy to move and easy to move at the expense of what happens next. If your metric is not the thing you actually want, you will optimize your way somewhere you did not intend, which is why the metric goes in writing before the test starts and why defining metrics is worth reading first.

Contamination between arms. If one user can see both versions, on two devices or via a shared account, the difference between the arms shrinks toward nothing and the test understates whatever is there.

Edge cases that catch people out

Six that each cost somebody a quarter.

Running until it works. Extending a test that has not reached significance is the same error as peeking, in slow motion. Extend only if you decide the new sample size in advance and for a stated reason.

Testing three variants and comparing them all. Three arms give three pairwise comparisons and the false alarm rate rises accordingly. Compare each variant with the control only, and adjust the threshold for the number of variants.

Conversion rate on the wrong denominator. Conversions divided by sessions, visitors, or users are three different metrics that move differently. Pick one, write it down, and make sure both arms use the same one.

Weekday and weekend traffic. A test run Monday to Friday is a test on weekday visitors. Run whole weeks so both arms see the same mix, which they will automatically, but so that the result generalizes.

Very low conversion counts. Rates built on fewer than about 30 conversions per arm are unstable no matter how many visitors you had, and the normal approximation behind the z-test starts to strain. Check the conversion counts, not just the visitor counts.

Shipping on a metric that did not move. If the primary metric came back flat and a secondary one moved, that is a hypothesis for the next test, not a result. Write it down as such and it stays useful.