A/B test calculator

Plans an A/B test (sample size and duration) and checks whether the difference in conversion rate between two or more variants is real or just random variation, with the size of the difference and its confidence interval. For anyone running an A/B test who needs a straight answer before shipping a variant.

How to use it

Choose Plan a test before you launch to learn how many visitors you need, or Analyze a test with the visitors and conversions of each variant as your testing or analytics tool reports them. The result updates as you type.

1. Plan first: enter the baseline conversion rate and the smallest change worth detecting, and note the sample size per variant.
2. Run the test until every variant reaches that size, using the same date range and the same conversion for all variants.
3. Enter the numbers in Analyze, with the first variant as the control. Read the verdict, then the intervals and the findings: they say how big the difference probably is and whether the data can be trusted.

How Reavlo tests this

The calculator uses a frequentist method, in your browser, and declares no winner without statistical significance.

Planning. With baseline p₁, target p₂ (the baseline plus your minimum detectable effect, relative or absolute), significance α and power 1 − β, the visitors needed per variant are n = (z_1−α/2 × √(2 p̄ (1 − p̄)) + z_1−β × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)², where p̄ = (p₁ + p₂) ÷ 2, rounded up. With more than two variants, α is divided by the number of comparisons with the control (Bonferroni), which keeps the plan valid for the Holm procedure used when analyzing. Duration is the total sample divided by your daily visitors, rounded up to whole days.

Analyzing. Each variant is compared with the control (the first variant):

1. Conversion rate p = conversions ÷ visitors.
2. Pooled two-proportion z-test: z = (p_variant − p_control) ÷ √(p̂ (1 − p̂) (1 ÷ n_control + 1 ÷ n_variant)), with the two-sided p-value from the standard normal distribution.
3. Holm correction across all comparisons: sort the p-values, multiply the smallest by the number of comparisons, the next by one fewer, and so on, keeping the adjusted values non-decreasing. A variant is significant when its adjusted p-value is below 1 − confidence.
4. The interval for the difference uses the unpooled standard error and the critical z of your confidence level. It is not corrected for multiple comparisons.
5. Lift is (p_variant − p_control) ÷ p_control.
6. The winner is the variant with the highest conversion rate among those significantly better than the control. If none is, there is no winner.

Sample ratio mismatch (SRM). A chi-square test compares the visitors per variant with the planned split (equal when you give none). A p-value below 0.001 is flagged as critical: the groups are not comparable and the other results should not be trusted.

Findings.

• srm.detected: the SRM test above.
• sample.too-small: a variant has fewer visitors than the planned sample size, or fewer than 100 when none is given.
• normal-approximation.weak: a variant has fewer than 5 conversions or 5 non-conversions, where the normal approximation is unreliable.
• peeking.warning: you looked at the results more than once before the planned end. A fixed-sample test assumes one read at the end.

Distribution functions (normal, chi-square) are computed from the regularized incomplete gamma function and checked against SciPy and statsmodels to within 1e-6.

Questions