A/B Testing: Design, Sample Size, and How to Interpret Results
Published:
Last Updated:
Category: Marketing Glossary
Authors: Shusaku Yosa
A/B testing is a randomized experiment that compares alternative versions to evaluate the effect of a change. Plan the hypothesis, primary metric, assignment unit, detectable effect, sample size, and stopping rule before launch. A higher observed rate, or a p-value below 0.05, is not a complete rollout decision.
Random assignment helps make groups comparable on average. It does not remove random variation, broken tracking, exposure to both variants, or interference between users. Causal interpretation depends on the design and its execution. All numerical examples in this guide are fictional, not improvement benchmarks or company case studies.
Decide what the comparison should teach you
Changing only a headline can make learning about that change easier. You can also compare two complete page designs through randomized assignment. That remains an A/B test, but the result estimates the package's effect rather than the contribution of each element. A single changed element is a choice about learning, not a universal requirement.
Approach | What changes | What the result can identify |
|---|---|---|
Single-element A/B | One headline or another defined change | The effect of that version difference |
Whole-page A/B | A package of changes | The package effect, not each component |
Multivariate design | Combinations of several factors | Main effects or interactions supported by the design |
Separate URLs, client or server delivery | How versions are served | Implementation alone does not establish randomization |
Registering several ad assets and allowing a platform to optimize delivery is not automatically a randomized experiment. Audiences, auctions, and exposure opportunities may differ. Check the platform's formal experiment design before interpreting an asset comparison as causal. Email opens also have measurement limitations, including privacy features; combine them with clicks, accepted outcomes, and unsubscribes rather than equating opens with human reading.
Begin with a landing page audit and a plausible explanation for the problem. For button changes, use the CTA design guide. Fix known submission failures or incorrect prices before treating them as uncertain optimization hypotheses.
Write the experiment brief before building variants
A useful hypothesis connects a problem, a proposed mechanism, and an outcome: explaining why a form asks for a phone number may reduce uncertainty and increase accepted inquiries. Choose a primary metric and separate guardrails for lead quality, revenue, errors, or customer harm.
Plan item | Specify | Why it matters |
|---|---|---|
Hypothesis and audience | Problem, mechanism, intervention, eligibility | Connect the change to a decision |
Metrics | Primary outcome and quality/safety guardrails | Avoid optimizing clicks at the expense of outcomes |
Units and allocation | User or organization; ratio; persistence | Address dependence, contamination, and interference |
Statistical settings | Baseline, absolute/relative MDE, alpha, power | Estimate sample and uncertainty |
Timing and decision | Maximum duration, outcome delay, stopping and rollout rules | Prevent result-driven changes |
Assignment and analysis units must agree with the statistical method. If users are assigned but their repeated sessions are analyzed as independent observations, uncertainty can be understated. Check cross-device identification, login changes, exclusions, and exposure persistence. A revenue metric with a skewed distribution needs a suitable analysis; a binary-conversion formula cannot simply be reused.
Microsoft Research's preparation guidance covers pre-experiment design. Use the KPI guide to distinguish business and diagnostic metrics, and the conversion counting guide to define success consistently.
Translate sample size into feasible calendar time
Required sample size depends on the baseline rate, minimum detectable effect (MDE), significance level, power, allocation, and test method. State MDE as both an absolute percentage-point difference and a relative change. It should represent a decision-relevant difference, not an optimistic effect chosen merely to make the required sample small.
Fictional plan: independently assigned users enter two equal groups, with accepted inquiry measured once as a binary outcome per user. The baseline is 2.0%, and the desired detectable rate is 2.4%: +0.4 percentage points, or +20% relatively. A two-sided normal approximation with 5% significance and 80% power requires approximately 21,109 users per group, or 42,218 total.
At 1,000 eligible new users per day across both groups, traffic alone takes about 43 days, plus conversion-maturation time. This is not a universal test duration. Recalculate when clustering, repeat observations, allocation, multiple comparisons, or the metric distribution changes. The 5% and 80% settings belong to this example; they are not mandatory values for every decision.
If the required period exceeds the business deadline, consider a larger decision-relevant change, user research, usability observation, or measurement work. An underpowered test cannot establish that no improvement opportunity exists.
Validate delivery and monitor safety during the experiment
- Test both variants across devices, languages, consent states, reloads, navigation, and successful submission.
- Verify persistent assignment, eligible audiences, and unintended exposure to both versions.
- Reconcile assignment, exposure, success events, and business-side records; check missing data by group.
- Monitor predefined safety and quality guardrails, including severe errors or inability to purchase.
- Investigate unexplained allocation discrepancies before reading the treatment effect.
The last issue is sample ratio mismatch (SRM): a statistically inconsistent observed allocation relative to the planned ratio. An ordinary random difference in counts is not automatically SRM. Microsoft Research's SRM guidance treats it as a data-quality diagnosis that comes before trusting the outcome comparison.
Separate operational monitoring from opportunistic winner selection
With an ordinary fixed-horizon test, checking the p-value daily and stopping on the first significant result increases false-positive risk beyond the original design. That does not mean you should stop monitoring errors, safety, or allocation. If interim efficacy decisions are required, plan an appropriate sequential method and stopping rules in advance. Research on continuous monitoring explains this distinction.
Extending only disappointing tests or selecting only successful segments after seeing the data changes the analysis. Record departures from the plan and distinguish confirmatory conclusions from exploratory findings. A later experiment may be needed to verify an interesting segment pattern.
Read effect size, uncertainty, and business value together
Fictional result: A produces 100 successes among 5,000 users, or 2.0%. B produces 110 among 5,000, or 2.2%. The absolute difference is +0.2 percentage points. The relative change is 0.2 ÷ 2.0 = +10%, not a ten-percentage-point improvement.
Measure | A | B or difference |
|---|---|---|
Users | 5,000 | 5,000 |
Accepted successes | 100 | 110 |
Rate | 2.0% | 2.2% |
Absolute difference | Reference | +0.2 percentage points |
Relative difference | Reference | +10% |
95% interval for difference | Unpooled normal approximation | About −0.362 to +0.762 percentage points |
A simple unpooled normal approximation gives a 95% interval for B minus A of approximately −0.362 to +0.762 percentage points. It includes zero, so this example does not establish a winner. Consider the range of deterioration and improvement consistent with the data, the prespecified MDE, implementation cost, and outcome quality. Another appropriate estimator may produce a somewhat different interval; record the method used.
A p-value is the probability, under the null hypothesis and statistical assumptions, of obtaining a result at least as extreme as the observed one. It is not the probability that B is correct or that the observed difference happened by chance. In the usual frequentist interpretation, 95% confidence describes the long-run coverage of the procedure, not a 95% probability attached to this particular fixed true effect.
The research overview of online controlled experiments discusses assumptions, contamination, and multiple comparisons. Avoid searching many metrics and segments until one looks favorable. Label unplanned analysis exploratory and account for multiplicity when making confirmatory claims.
Record rollout, rejection, or an inconclusive result
A rollout decision needs practical value, maintenance cost, quality, and safety checks as well as statistical evidence. A nonsignificant result does not mean the true effect is exactly zero. If the interval is too wide for the decision, record the result as inconclusive. Plan observation for novelty or sustained-use effects when those matter instead of inventing a new window after seeing results.
- Experiment ID, hypothesis, changes, eligibility, assignment, and analysis unit.
- Primary metric, MDE, sample requirement, period, maturation date, and method.
- Group counts, successes, effect interval, SRM, missingness, and guardrails.
- Rollout, reject, or inconclusive decision; owner, rationale, and review date.
- Plan deviations, exploratory findings, and the next hypothesis.
Feed the record into the next PDCA cycle. For initiative ownership, execution plans, costs, and monthly budget-versus-actual reviews, explore Xtrategy's features. Preserve the reasoning behind the decision alongside the work it creates.
