A/B test sample size and significance calculator
Plan a test before you build it, or read one that has finished. The calculator runs in your browser and sends nothing anywhere.
Why many B2B tests never finish
A page with a 3% conversion rate needs about 13,900 visitors in each version to detect a 20% relative lift at 95% confidence and 80% power. At 2,000 visits a week that is 14 weeks for one test, and a test stopped early often crowns noise as the winner.
Two things help. Test bigger changes, because the sample size falls with the square of the lift you want to detect. And start from the strongest draft you can write, so the test you can afford to run is worth running.
A Behavly check writes that draft from your live page, with the reason for each change.
How the numbers are worked out
Sample size uses the standard two-proportion formula for a two-sided test at the confidence and power you pick. A result is read with a pooled two-proportion z test; the interval around each rate is the normal approximation. The figures assume an even split, a fixed sample decided in advance and one look at the end.
Checking a running test every day and stopping at the first significant reading raises the false positive rate well above the confidence you picked.
Questions
Reading the calculator
What is the smallest lift worth detecting?
The relative change you would act on. If a 10% lift would not change what you ship, plan for a larger one: the test finishes sooner and a smaller effect would not have mattered.
What does the p value mean here?
The chance of seeing a difference at least this large if A and B convert at the same rate. Below 0.05 is significant at 95% confidence.
Can I stop the test as soon as it is significant?
Not with these figures. They assume you decide the sample size first and read the test once at the end. Stopping at the first significant reading makes false winners far more likely.
Does the calculator store what I type?
No. It runs in your browser and nothing you type leaves the page.