A/B Testing Cheat Sheet

A practical guide to designing, running, and statistically analyzing A/B tests, covering sample size, significance testing, and common pitfalls.

2 PagesIntermediateMar 10, 2026

Sample Size & Z-Test

Power analysis and two-proportion z-test.

python

import numpy as npfrom statsmodels.stats.power import NormalIndPowerfrom statsmodels.stats.proportion import proportions_ztest# Sample size needed to detect an effect (power analysis)analysis = NormalIndPower()n_per_group = analysis.solve_power(    effect_size=0.2,   # standardized effect size (Cohen's h)    alpha=0.05,         # significance level    power=0.8,          # desired statistical power    ratio=1.0,)print(f"Required sample size per group: {n_per_group:.0f}")# Two-proportion z-test for conversion rate comparisonconversions = np.array([120, 150])   # [control, treatment]visitors = np.array([2000, 2000])z_stat, p_value = proportions_ztest(conversions, visitors)print(f"z = {z_stat:.3f}, p = {p_value:.4f}")

T-Test for Continuous Metrics

Compare means and build a confidence interval.

python

from scipy import statsimport numpy as np# For continuous metrics (e.g. revenue per user), use a t-testcontrol = np.random.normal(50, 10, 1000)treatment = np.random.normal(52, 10, 1000)t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)print(f"t = {t_stat:.3f}, p = {p_value:.4f}")# 95% confidence interval for the difference in meansdiff = treatment.mean() - control.mean()se = np.sqrt(control.var(ddof=1) / len(control) + treatment.var(ddof=1) / len(treatment))ci = (diff - 1.96 * se, diff + 1.96 * se)print(f"Difference: {diff:.2f}, 95% CI: {ci}")

A/B Testing Concepts

Key vocabulary for designing experiments.

Null hypothesis (H0)- assumes no difference between control and treatment
Statistical significance (p-value)- probability of seeing this result (or more extreme) if H0 were true
Statistical power- probability of correctly detecting a true effect (typically target 80%)
Minimum Detectable Effect (MDE)- smallest effect size the test is designed to reliably detect
Sample Ratio Mismatch (SRM)- unequal group sizes vs. the intended split, signals a bug in randomization
Novelty effect- short-term behavior change from users noticing something new, not a lasting effect
Peeking problem- repeatedly checking significance before the planned sample size inflates false positive rate
Multiple comparisons- testing many metrics/segments increases chance of a false positive; use a correction (e.g. Bonferroni)

Test Design Checklist

Steps to run a trustworthy experiment.

Define one primary metric- pick a single primary success metric before launch to avoid p-hacking
Compute required sample size- run a power analysis before starting, don't stop early
Randomize consistently- assign users to variants deterministically (e.g. hash of user ID)
Check for SRM- verify observed group sizes match the intended split before trusting results
Run for full business cycles- avoid day-of-week effects by running at least 1-2 full weeks
Pre-register the analysis plan- decide the test and segments in advance to avoid fishing for significance

Pro Tip

Decide your sample size and test duration up front using a power analysis, and don't stop the test early just because you see a significant p-value — 'peeking' at results repeatedly inflates your false positive rate far above 5%.

Was this cheat sheet helpful?

ROC & AUC Cheat Sheet

Hypothesis Testing Cheat Sheet

Explore Topics

#ABTesting#ABTestingCheatSheet#DataScience#Intermediate#SampleSizeZTest#TTestForContinuousMetrics#ABTestingConcepts#TestDesignChecklist#MachineLearning#Testing#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services