A/B test significance calculator

A/B testing: the math is easy, the discipline is hard

The math behind statistical significance hasn't changed in 100 years — it's a Z-test on proportions for conversion-rate tests, or a T-test on continuous metrics for revenue. What breaks 80% of the A/B tests I audit isn't the math; it's the process around the math. Teams peek at results early. They stop tests when they "look good." They run too many tests simultaneously on overlapping audiences. They declare winners at n=200 conversions per variant because "the lift is huge." And then the "winner" reverts to baseline the following quarter and everyone wonders what happened.

This calculator gives you the frequentist p-value (the 95% confidence industry standard) and tells you whether the test is conclusive. Below, I'll walk through the rules that turn a calculator result into a real business decision — and the traps that turn significance into self-deception.

Sample size planning (do this before you launch)

The #1 reason tests don't reach significance is under-powered sample sizes. Use a power-analysis calculator (Evan Miller's sample-size calculator is the industry standard) and plug in: baseline conversion rate, minimum detectable effect (MDE), and statistical power (usually 80%). Typical B2C ecommerce test with 3% baseline conversion and 10% MDE requires ~14,000 visitors per variant. At 8,000 daily visits total split 50/50, that's ~3.5 days to adequately powered — but most teams need 14–21 days to cover a full business cycle regardless.

The 10 rules of honest A/B testing

1. Decide sample size and test duration BEFORE launching. Document the commitment. This prevents early-stopping temptation.
2. Run for a full business cycle. Minimum 7 full days. For B2B, usually 14–21 days to capture weekday-vs-weekend behavior. Longer if your cycle is monthly (subscription renewals, salary-pay cadence).
3. Don't peek. Checking early and stopping when "significance is reached" inflates your false-positive rate from 5% to as high as 30%. Academic papers have been written on this (Johari et al., 2017).
4. One variable at a time. If you change the headline AND the hero image AND the CTA color, a win tells you nothing about which change worked.
5. Equal traffic allocation. Don't run 70/30 unless you have a specific reason (e.g., exposure-limited test). 50/50 maximizes statistical power.
6. Watch for interaction effects. If you run 5 tests simultaneously on the same page, they can interact. Use a proper experimentation platform (Optimizely, VWO, Statsig, GrowthBook) that handles this.
7. Segment post-hoc only to generate hypotheses, not to declare winners. "It won for mobile but lost for desktop" could be real or could be p-hacking. Re-run the test segmented if the segment effect matters.
8. Declare ties. If p > 0.20 after adequate sample, the variants are truly equivalent. Ship the cheaper/simpler one.
9. Check for sample ratio mismatch. If variant A got 50,000 and variant B got 48,000 visitors, your randomization is broken. Throw out the test.
10. Replicate critical winners. Before committing engineering resources to permanent implementation, re-run the winning variant against control for another cycle. "Regression to the mean" is real.

Minimum detectable effect: the number that decides everything

MDE is the smallest lift you can reliably detect given your traffic. If you have 5,000 visits per variant per month and a 2% baseline, you can only detect 30%+ lifts with reasonable power. Most "real" lifts from landing-page tweaks are 5–15%, so that test is permanently under-powered. Two responses: (1) only test bold changes that plausibly move the needle 30%+ (different hero concepts, different offer structure), or (2) aggregate smaller pages into one test by running a site-wide change.

Most small-business owners I consult with are testing at traffic levels where only massive wins are detectable. They burn 12 weeks on a test that looks inconclusive and conclude "A/B testing doesn't work for us." The real diagnosis: their traffic is too low for the effect sizes they're trying to measure. Either test bigger changes or test less often with longer cycles.

Frequentist vs. Bayesian — which to use

This tool is frequentist (p-value based, the classical approach). Bayesian testing — used natively by VWO, Statsig, GrowthBook — answers a different question: "what's the probability that B is actually better than A?" rather than "if there was no real difference, how likely is this much variation?"

Both are valid. Frequentist is standard in academic and regulated settings. Bayesian is more intuitive for business stakeholders ("B has a 94% chance of being better") and handles early stopping more gracefully. If you're running a modern experimentation platform, Bayesian with proper priors is usually the better default. For hand-calculated tests (this tool), frequentist is simpler and well-understood.

What to test vs. what to leave alone

Tests have diminishing returns. The first few tests you run on a new landing page or email template will produce big wins — that's typically where the page has glaring problems. By test 10–15, you're into the 3–8% lift range. By test 30+, you're often running tests that cost more in engineering and ops time than the lift recovers.

Prioritize tests by ICE score (Impact, Confidence, Ease). High-traffic pages with clear hypotheses backed by session-replay data or user interviews. Avoid: low-traffic pages, tests driven by "let's just try X," and aesthetic tweaks (button color, font size) that rarely produce meaningful lifts.

Three A/B test archetypes with full sample-size and MDE math

Archetype 1: DTC landing-page headline test (high traffic)

Baseline add-to-cart rate: 4.2%. Desired MDE: 8% relative lift (practical threshold — anything smaller doesn't change roadmap). Power 80%, significance 0.05. Required sample per variant: ~28,400 sessions. At 6,000 daily paid sessions split 50/50 = 3,000/variant/day, adequately powered in ~9.5 days. Plan to run 14 days to cover the full weekly cycle. Expected outcome: if true lift is 10%+, test reaches significance in the planned window 83% of the time. Tool: VWO at $199/month for SMB tier or Optimizely at $4k/month for enterprise.

Archetype 2: B2B SaaS signup-flow test (low traffic)

Baseline signup rate: 2.1%. Traffic 800 sessions/day. Desired MDE 15% relative lift. Required sample per variant: ~29,000 sessions. At 400/variant/day, 72 days to power — which is too long given SaaS release cycles. Solutions: (1) increase MDE to 25% (only big wins detectable — requires ~10,400 per variant, 26 days), (2) switch to Bayesian inference via Statsig (can call practical significance earlier with proper priors), (3) use CUPED variance reduction (Microsoft Research methodology) to cut required sample ~30%. Most PLG SaaS teams run (2) + (3) combined.

Archetype 3: Email subject-line test (Klaviyo, 50k list)

Baseline open rate 32%. Klaviyo native A/B test on 20% of list (10,000 recipients, 5,000 per variant). Desired MDE on open rate: 5 percentage points (37% vs 32%). With n=5,000 per variant, MDE at 80% power and α=0.05 is ~1.8 percentage points — so 5-point MDE is over-powered and you will detect smaller effects reliably. Klaviyo automatically sends the winner to the remaining 80% after a 4-hour significance window. Cost: already bundled in Klaviyo $150/month at that list size. This is the cleanest, cheapest A/B testing that most DTC brands never fully leverage.

Testing-stack reference, April 2026

Decision framework: which experimentation platform fits your org

Pick based on team composition and scale. Under 500k monthly visitors with 1 marketer and 0 engineers: VWO for SMB — visual editor lets non-technical users ship tests without engineering bottleneck. 500k-5M monthly visitors with 2-3 engineers: Statsig or GrowthBook — both expose feature-flag API, and engineers will ship experiments as part of normal releases. Above 5M monthly visitors with dedicated experimentation team: Optimizely, Statsig Enterprise, or build in-house on top of LaunchDarkly + Mixpanel. Skip the "free with GA4" path (GA4 experiments were sunset in 2023 alongside Optimize) — you need a real platform with proper sequential testing and variance reduction if you want to run more than 5 tests/month without false-positive drift.