Question 1

What is statistical significance in A/B testing?

Accepted Answer

Statistical significance tells you if the difference between your test variants is likely real or just due to random chance. Typically, a p-value below 0.05 (95% confidence) indicates significance.

Question 2

What is a p-value?

Accepted Answer

A p-value is the probability of seeing your test results (or more extreme) if there's actually no difference between variants. Lower p-values = stronger evidence of a real difference.

Question 3

What sample size do I need for an A/B test?

Accepted Answer

It depends on your baseline conversion rate, the minimum effect you want to detect, desired statistical power (usually 80%), and significance level (usually 5%).

Question 4

What is a confidence interval?

Accepted Answer

A confidence interval gives a range of values likely to contain the true effect. A 95% CI means if you ran the test 100 times, 95 of them would capture the true difference.

Question 5

When should I stop an A/B test?

Accepted Answer

Stop when you've reached your pre-calculated sample size AND achieved statistical significance. Never stop early just because results look good (peeking increases false positive rates).

Question 6

What is the minimum detectable effect (MDE)?

Accepted Answer

MDE is the smallest difference between variants you want to reliably detect. Smaller MDE requires larger sample sizes. Most tests target a 10-20% relative improvement.

🧪 A/B Test Calculator

Test Results Analysis

Control (A)

Variant (B)

📊 Interpretation

Sample Size Calculator