A store experiment result is trustworthy when enough users saw each variant, the test ran across full weekly cycles, the lift exceeds noise, and the difference matters in practice. Do not stop early on a good-looking number, and treat inconclusive results as information, not failure.
What significance means
Statistical confidence estimates how likely a difference is due to chance. A result with high confidence still may be too small to matter, while a large lift on tiny traffic may be noise.
Platforms present results differently, such as confidence intervals or probability to beat control, so learn how yours reports.
Sample size and duration
Low-traffic apps need longer tests to detect small lifts. Run through at least one full weekly cycle so weekday and weekend behavior are both included, and avoid stopping when numbers first look good.
Estimate required traffic in advance using baseline conversion and the smallest lift you care about.
Practical significance
Decide the minimum lift worth acting on, considering cost of change and brand effects. A 1 percent relative lift may be real but not worth a redesign.
Look at ranges, not only point estimates.
Avoid common errors
Do not run overlapping tests that share audiences, change variants mid-test, or check results repeatedly and stop at the best moment. Beware of seasonal or campaign shifts that change traffic mix.
Document inconclusive tests and learn from them.
Common mistakes with A/B test significance ASO
Stopping the moment a variant looks ahead is the most common error, because early swings are mostly noise. Running tests for only a few days misses weekday and weekend differences. Testing several things at once makes the winner uninterpretable.
Another mistake is ignoring practical size. A tiny lift on a huge page may be real but not worth a brand change.
A/B test significance ASO: a worked example
A team with moderate traffic wants to detect a 5 percent relative lift. It estimates that the test will need several weeks, so it plans around a release and avoids a major campaign during the window.
At the end, the lift interval includes zero, so the team records the test as inconclusive and designs a bolder variant for the next round.
A quick calculation (illustrative numbers)
Say the control converts at 20%. To reliably detect a 10% relative lift (to 22%) you need roughly 6,500 visitors per variant at 95% confidence and 80% power. A page with 500 views a week would need about six months to get there.
If traffic is low, test bolder changes with bigger expected effects, or accept directional evidence and combine it with other signals.
How to do it, step by step
- Pick the primary metricUse conversion from page views to first-time downloads.
- Estimate required trafficUse baseline conversion and the smallest lift you care about.
- Plan the durationRun at least one full weekly cycle and avoid campaigns.
- Do not peek and stopDecide the end date in advance.
- Read intervals and practical sizeCheck the range, not only the point estimate.
Before and after examples
Illustrative examples written for a generic app in this category, not client results. Swap in your own product, audience, and data.
Stop on day two because B is ahead
Wait until the planned end after full weekly cycles
Why it works: Early gaps are mostly noise.
B wins by 4%
B shows +4% with a range from -2% to +10%
Why it works: The range includes zero, so the result is inconclusive.
Any lift is a win
Act only if the lift exceeds the minimum worth the change
Why it works: Practical size matters.
Action plan
- Estimate needed traffic before testing
- Run full weekly cycles
- Do not stop early
- Define a minimum meaningful lift
- Read intervals, not just points
- Avoid overlapping tests
Metrics to monitor
- Test duration and sample size
- Lift with interval
- Share of conclusive tests
- Post-launch confirmation
Record the baseline, release date, audience, markets, and external changes before interpreting movement. Rankings and conversion are useful signals, but a decision is stronger when activation, retention, and business value confirm that the page attracted the right users.
Frequently asked questions
How long should a test run?
At least a full weekly cycle and until reliable data accumulates.
What if the result is inconclusive?
Record it and decide whether to retest with a bolder variant.
Is 95 percent confidence required?
It is a common threshold, but consider context, cost, and risk.
Keep reading
More in Measurement
Browse all 9 Measurement guides
