A/B Testing: Design, Sample Size, and Result Analysis
Published:
Last Updated:
Category: Web Analytics, Ad Measurement
Authors: Shusaku Yosa
An A/B test randomly assigns eligible participants to two alternatives and compares their outcomes. Running the alternatives concurrently helps separate the change being tested from seasonality and shifts in incoming traffic.
A higher observed conversion rate is not sufficient to select a winner. You also need reliable assignment and measurement, adequate data, an estimate of uncertainty, and a result worth implementing.
Start with an observed problem
Build the hypothesis from behavior, errors, interviews, or support questions. “Change the button color” is an action; “users cannot understand what happens after submitting” is a problem you can investigate.
Design field | Hypothetical form test |
|---|---|
Observation | Mobile visitors frequently fail phone-number validation |
Hypothesis | An example and specific error guidance will reduce failures |
A | Current form |
B | Form with clearer input and error instructions |
Assignment unit | Eligible user, with assignment retained |
Primary outcome | Share of assigned users completing an inquiry |
Guardrails | Valid inquiry rate, opportunity rate, error rate |
Testing a whole page is possible, but the result describes the combined alternative. It cannot identify the effect of each changed element separately.
Align the assignment and analysis units
If assignment is by user, do not count repeated visits as independent users. Keep assignment stable where possible and document limits such as cookie deletion and cross-device use.
Choose one primary decision metric in advance. Button clicks may be useful diagnostics but can improve without generating more valid inquiries. See the CVR guide for denominator choices.
Calculate sample size before starting
Sample requirements depend on the baseline rate, minimum effect of interest, significance level, statistical power, sidedness, and allocation ratio.
For a hypothetical increase from 3% to 3.6%, a conventional two-sided normal approximation with 5% significance, 80% power, equal allocation, and independent binary user outcomes requires approximately 13,914 users per group, or 27,828 in total. Different methods and corrections can produce different requirements.
With 500 eligible unique users per day across both groups, that is about 56 days before allowing for conversion delays and other design considerations. Five hundred visits are not necessarily 500 eligible users.
The example uses p1=0.03, p2=0.036, and p=(p1+p2)/2. The per-group calculation is {1.959964×√[2p(1−p)] + 0.841621×√[p1(1−p1)+p2(1−p2)]}² ÷ (p2−p1)², rounded up.
If the required traffic is unrealistic, reconsider the business-relevant detectable effect, choose a higher-traffic opportunity, or use interviews and usability observation to refine the proposal. Do not manufacture certainty from an underpowered test.
Validate the experiment before interpreting it
Test both variants, their completion events, and the assignment behavior. Check for duplicate events, missing data, internal traffic, and unexpected sample allocation.
A substantial mismatch from the planned allocation can indicate implementation or eligibility problems. Investigate that before interpreting significance. Record operational changes during the experiment, especially changes affecting only one group.
Read uncertainty and business value together
A p-value is not the probability that B is better. Under the test's assumptions, it describes how incompatible the observed evidence is with the specified null hypothesis.
A 95% confidence interval describes the coverage behavior of the interval procedure under repeated sampling. It does not guarantee a minimum future improvement or mean that the observed fixed interval has a frequentist 95% probability of containing the parameter. NIST confidence interval explanation
Result | Decision approach |
|---|---|
Credible, commercially useful improvement | Review guardrails and implementation costs |
Detectable but tiny effect | Assess whether the benefit justifies maintenance |
Interval includes meaningful harm and benefit | Treat the result as uncertain |
Missing samples or measurement failures | Resolve validity before deciding |
“Not significant” does not prove equality. Distinguish a precise estimate of a small effect from a wide interval caused by insufficient data.
Avoid three common errors
Do not repeatedly inspect a fixed-sample test and stop only when it becomes significant. If interim decisions are necessary, use a method and stopping rule designed for them.
Do not search many outcomes and segments, then present the best result as if it were the original hypothesis. Label exploratory findings and validate them separately.
Do not interpret a simple before-and-after comparison as a randomized experiment. Traffic, prices, and seasonality may have changed at the same time.
Keep a reusable test record
Record the hypothesis, audience, variants, assignment, primary metric, sample plan, dates, actual counts, effect and interval, guardrails, decision, and next question. Keep inconclusive results too. A useful testing program accumulates evidence rather than only a collection of wins.
