by myuser.ai
Get it free on Shopify or Wix
Free tool

A/B test calculator: significance and sample size

Check whether one version of a page really beat the other, with the p-value and an interval for the difference, and work out how many visitors a test needs before you start it. Every step of the math is shown.

Your test

Version A (the original)
Version B (the change)

It opens with a made-up test. Enter the numbers once your test has reached the sample size you chose before it started. Nothing you enter leaves your browser.

The result

Version A converts2%
Version B converts2.6%
B against A+30%
p-value0.073

B converted at 2.6% against A's 2%, but at p = 0.073 that isn't significant at 95%: if the two versions really converted the same, a gap at least this big would turn up by chance about 7 times in 100. Keep the test running to the sample size you chose before it started.

The arithmetic, step by step

  1. Rates: A 80 ÷ 4,000 = 2%; B 104 ÷ 4,000 = 2.6%.
  2. Difference: 2.6% − 2% = +0.6 points, +30% on A's rate.
  3. Pooled rate: (80 + 104) ÷ (4,000 + 4,000) = 2.3%, the rate if both versions were the same.
  4. Standard error: √(0.02300 × 0.97700 × (1/4,000 + 1/4,000)) = 0.00335.
  5. z-score: 0.00600 ÷ 0.00335 = 1.79, so the two-sided p-value is 0.073.
  6. 95% interval for B − A: +0.6 points ± 1.96 × 0.00335 = −0.06 points to +1.26 points.

How many visitors does an A/B test need?

Work it out before the test starts, from your conversion rate now and the smallest lift worth catching1. It opens with the example from our guide to A/B testing a small site.

Before you start

The lift is a share of your rate now: 20% takes 2% to 2.4%. Nothing you enter leaves your browser.

Sample size

Visitors per version19,784
Visitors in all39,568
How longAbout 13.2 months
Rule of thumb19,600 per version

To catch a rise from 2% to 2.4% at 5% significance and 80% power, each version needs 19,784 visitors, 39,568 in all. At 3,000 visitors a month, split evenly, that's about 13.2 months: too long for most stores to hold everything else still.

The arithmetic, step by step

  1. Rates to tell apart: 2% now and 2.4% with a 20% lift, a difference of 0.4 points.
  2. z-values: 1.96 for 5% significance (two-sided) and 0.84 for 80% power.
  3. Visitors per version: (1.96 × √(2 × 0.0200 × 0.9800) + 0.84 × √(0.0200 × 0.9800 + 0.0240 × 0.9760))² ÷ 0.0040² = 19,784.
  4. Rule of thumb: 16 × 0.0200 × 0.9800 ÷ 0.0040² = 19,600 per version, close to the full formula.
  5. How long: 39,568 visitors ÷ 3,000 a month = 13.2 months.

How does the A/B test calculator work?

It runs the standard test for two conversion rates, a two-proportion z-test, and shows each step under the result:

  • The two rates. Each version's conversions ÷ its visitors.
  • The p-value. How often a gap at least this big would turn up by chance if both versions really converted the same. Under 0.05 counts as significant at 95%.
  • The interval. The range the real difference likely sits in. When it runs from below zero to above it, B could be worse than A.
  • The sample size. Evan Miller's formula, the one behind his sample size calculator1, at the significance and power you pick. At 5% and 80% it also shows the rule of thumb, 16 × p × (1 − p) ÷ δ², with p your rate now and δ the lift in points2.

Don't stop a test the moment it looks significant

Results drift up and down while a test runs. If you check a running test and stop as soon as it looks significant, you'll declare far more false winners than the confidence level suggests: checked ten times, what reads as 1% significance is really 5%3. Fix the sample size before the test starts, run it in whole weeks and for at least one, and judge the result once, at the end. How long an A/B test should run explains why whole weeks matter.

Not enough visitors for a test?

At a 2% rate and 3,000 visitors a month, catching a 20% lift takes over a year, and few stores can hold everything else still for that long. Compare equal weeks before and after one change instead, and watch replays from both sides. It isn't a controlled test, but it shows whether visitors notice and use the change. Run a before-and-after check with replays shows what to keep the same on both sides.

Session Recordings

See what visitors do with a change

Session Recordings records visits before and after you change a page, so you can watch whether shoppers use it. Star the replays worth keeping past 30 days. Free forever: 2,000 recordings a month.

Questions

A/B testing, answered

What does statistical significance mean in an A/B test?

That a difference as big as the one you saw would rarely turn up by chance if the two versions really converted the same. At 95%, rarely means less than 1 time in 20: a p-value under 0.05. It doesn't say how big the real difference is; the interval does that.

How many visitors does an A/B test need?

It depends on your conversion rate now and the smallest lift you want to catch. At a 2% rate, catching a rise to 2.4% takes 19,784 visitors per version at 5% significance and 80% power, 39,568 in all. Lower rates and smaller lifts need far more.

How long should an A/B test run?

Until it reaches the sample size you chose before it started, and for at least one full week, in whole weeks, so every day of the week is in it. Don't stop the moment the result looks significant: checking a running test and stopping on a good result makes false winners far more likely.

Can a small store run A/B tests?

Often not with a clear answer. At 3,000 visitors a month, the test above takes over a year. Compare equal weeks before and after one change instead, and watch replays from both sides: it isn't a controlled test, but it shows whether visitors use the change. How to run a before-and-after check goes step by step.

Is this a one-tailed or a two-tailed test?

Two-tailed: it asks whether B differs from A in either direction. That's the usual choice, since a change can make things worse as well as better.

What's the difference between relative and absolute lift?

Relative lift is the change as a share of your rate now: 2% to 2.4% is a 20% lift. Absolute lift is the change in percentage points: 0.4 points. The sample size calculator takes the relative lift.

See your next visit through your visitor's eyes

Install it on Shopify and switch it on in your theme, or add it to your Wix site. Recording starts with your next visitor.

Add to Shopify Add to Wix Free forever · Shopify and Wix · Form typing masked in the browser