A/B Test
What is an A/B test?
An A/B test is a method of comparing two variants (A and B) of a page, ad, headline, or any other element to determine which one better achieves a goal — higher conversion, higher CTR, lower CPC. Traffic is randomly split between the variants, and results are evaluated statistically.
A/B tests are the foundation of CRO (Conversion Rate Optimization) — they enable data-driven decisions instead of relying on intuition.
Why does it matter?
- Higher conversion — changing a single element (headline, CTA, button color) can increase conversion by 20-50%
- Data-driven decisions — eliminates guesswork and opinions
- Continuous optimization — every test is a step toward better results
- Budget savings — better conversion = lower customer acquisition cost
How to run an A/B test?
1. Hypothesis
Define what you want to test and why. For example: "Changing the headline from descriptive to a question will increase conversion by 15%."
2. Variants
Create variant B with a single change. Test one thing at a time!
3. Traffic split
Random 50/50 split between variant A (control) and B (test).
4. Data collection
Collect data long enough — a minimum of 100 conversions per variant for statistical significance.
5. Analysis
Is the difference statistically significant (p < 0.05)? If so — implement the winner.
Best practices
- Test one change at a time — if you change the headline, CTA, and image simultaneously, you won't know what worked
- Sufficient traffic — without at least 1,000 visits per variant, results won't be reliable
- Be patient — don't end the test after 24 hours; wait a minimum of 7-14 days
- Prioritize — test elements with the greatest impact: headline > CTA > button color
- Document everything — record every test and result to build a knowledge base
- Separate mobile — behavior on mobile and desktop may differ
More on conversion optimization in the article why your page doesn't convert.
Related terms
- Conversion — goal completion on a page
- Landing page — destination page
- CTR — click-through rate
- CPC — cost per click
- ROAS — return on ad spend