A/B Test Calculator — Statistical Significance
Determine whether the difference in conversion rates between your control (A) and variant (B) is statistically significant using a two-proportion z-test.
Significance level (α)
Not statistically significant — cannot reject the null hypothesis
- 1
Rate A (control)
50 ÷ 1,000 = 0.05 - 2
Rate B (variant)
65 ÷ 1,000 = 0.065 - 3
Pooled rate
(50 + 65) ÷ (1,000 + 1,000) = 0.0575Assumes equal conversion rate under the null hypothesis. - 4
Standard error
√(0.0575 × 0.9425 × (1/1,000 + 1/1,000)) = 0.01041 - 5
z-statistic
(0.065 − 0.05) ÷ 0.01041 = 1.4408 - 6
p-value (two-tailed)
2 × (1 − Φ(|1.4408|)) = 0.1496Φ is the standard normal CDF; smaller p means stronger evidence against the null.
How does this calculator work?
Pool the two conversion rates (p̄), compute SE = √[p̄(1−p̄)(1/nA+1/nB)], then z = (p₂−p₁)/SE. The two-tailed p-value is 2·Φ(−|z|). If p < α, the difference is statistically significant. Relative lift = (p₂−p₁)/p₁ × 100%. At least 10 events per group are recommended for the normal approximation to hold.
Formula
How this is calculated
The calculator runs a two-sided two-proportion z-test. It computes the conversion rate for each group (p₁ = conversions A / visitors A and p₂ = conversions B / visitors B) and then pools them under the null hypothesis that both groups share the same underlying rate: p̄ = (xA + xB) / (nA + nB).
The standard error under the null is SE = √[p̄ × (1 − p̄) × (1/nA + 1/nB)], and the z-statistic measures how many standard errors the observed difference (p₂ − p₁) is away from zero. A two-tailed p-value is obtained from the standard normal distribution: p = 2 × (1 − Φ(|z|)). If p is below the chosen significance level α, the test is declared significant — you can reject the null that the two rates are equal.
Relative lift = (p₂ − p₁) / p₁ × 100% shows how much better (or worse) variant B performs relative to control A. The z-test assumes independent random samples, binary outcomes, and large enough group sizes for the normal approximation to hold. A common rule of thumb is at least 10 expected successes and 10 expected failures in each group.
Frequently asked questions
It means the observed difference in conversion rates is unlikely to be due to random chance alone, given the chosen significance level α. At α = 0.05, there is a 5% chance of falsely declaring significance when the rates are actually equal (Type I error). It says nothing about whether the effect is practically large or economically important.
Sample size depends on the baseline conversion rate, the minimum detectable effect, and your chosen α and statistical power (typically 80%). The z-test is valid when each group has at least 10 expected conversions and 10 expected non-conversions. Use a sample-size calculator to determine the required experiment duration before you start.
This calculator uses a two-tailed test, which is appropriate when you care about any direction of change. A one-tailed test has more power to detect a specific direction but must be pre-specified before collecting data to prevent inflated false-positive rates.
Also known as
TG we-Calculate Editorial Team. (2026). A/B Test Calculator — Statistical Significance [Online calculator]. TG we-Calculate. https://we-calculate.com/calculator/ab-test-calculator
TG we-Calculate Editorial Team. "A/B Test Calculator — Statistical Significance." TG we-Calculate. 2026. https://we-calculate.com/calculator/ab-test-calculator.
TG we-Calculate Editorial Team, "A/B Test Calculator — Statistical Significance," TG we-Calculate, 2026. [Online]. Available: https://we-calculate.com/calculator/ab-test-calculator
@misc{wecalculate_ab_test_calculator, title = {A/B Test Calculator — Statistical Significance}, author = {{TG we-Calculate Editorial Team}}, howpublished = {\url{https://we-calculate.com/calculator/ab-test-calculator}}, year = {2026}, note = {TG we-Calculate} }
Did this calculator help you?