A/B test calculator: significance and sample size
Check whether one version of a page really beat the other, with the p-value and an interval for the difference, and work out how many visitors a test needs before you start it. Every step of the math is shown.
How many visitors does an A/B test need?
Work it out before the test starts, from your conversion rate now and the smallest lift worth catching1. It opens with the example from our guide to A/B testing a small site.
How does the A/B test calculator work?
It runs the standard test for two conversion rates, a two-proportion z-test, and shows each step under the result:
- The two rates. Each version's conversions ÷ its visitors.
- The p-value. How often a gap at least this big would turn up by chance if both versions really converted the same. Under 0.05 counts as significant at 95%.
- The interval. The range the real difference likely sits in. When it runs from below zero to above it, B could be worse than A.
- The sample size. Evan Miller's formula, the one behind his sample size calculator1, at the significance and power you pick. At 5% and 80% it also shows the rule of thumb, 16 × p × (1 − p) ÷ δ², with p your rate now and δ the lift in points2.
Don't stop a test the moment it looks significant
Results drift up and down while a test runs. If you check a running test and stop as soon as it looks significant, you'll declare far more false winners than the confidence level suggests: checked ten times, what reads as 1% significance is really 5%3. Fix the sample size before the test starts, run it in whole weeks and for at least one, and judge the result once, at the end. How long an A/B test should run explains why whole weeks matter.
Not enough visitors for a test?
At a 2% rate and 3,000 visitors a month, catching a 20% lift takes over a year, and few stores can hold everything else still for that long. Compare equal weeks before and after one change instead, and watch replays from both sides. It isn't a controlled test, but it shows whether visitors notice and use the change. Run a before-and-after check with replays shows what to keep the same on both sides.
See what visitors do with a change
Session Recordings records visits before and after you change a page, so you can watch whether shoppers use it. Star the replays worth keeping past 30 days. Free forever: 2,000 recordings a month.

A/B testing, answered
What does statistical significance mean in an A/B test?
That a difference as big as the one you saw would rarely turn up by chance if the two versions really converted the same. At 95%, rarely means less than 1 time in 20: a p-value under 0.05. It doesn't say how big the real difference is; the interval does that.
How many visitors does an A/B test need?
It depends on your conversion rate now and the smallest lift you want to catch. At a 2% rate, catching a rise to 2.4% takes 19,784 visitors per version at 5% significance and 80% power, 39,568 in all. Lower rates and smaller lifts need far more.
How long should an A/B test run?
Until it reaches the sample size you chose before it started, and for at least one full week, in whole weeks, so every day of the week is in it. Don't stop the moment the result looks significant: checking a running test and stopping on a good result makes false winners far more likely.
Can a small store run A/B tests?
Often not with a clear answer. At 3,000 visitors a month, the test above takes over a year. Compare equal weeks before and after one change instead, and watch replays from both sides: it isn't a controlled test, but it shows whether visitors use the change. How to run a before-and-after check goes step by step.
Is this a one-tailed or a two-tailed test?
Two-tailed: it asks whether B differs from A in either direction. That's the usual choice, since a change can make things worse as well as better.
What's the difference between relative and absolute lift?
Relative lift is the change as a share of your rate now: 2% to 2.4% is a 20% lift. Absolute lift is the change in percentage points: 0.4 points. The sample size calculator takes the relative lift.
See your next visit through your visitor's eyes
Install it on Shopify and switch it on in your theme, or add it to your Wix site. Recording starts with your next visitor.