By Omri Yacubovich | Head of Marketing @ Commerce Sciences
A/B testing is a critically important part of your conversion optimization strategy, but simply running tests just isn’t enough. Marketers need to make sure they’re running the right tests, and that they’re scientifically accurate so the results can be relied upon to drive real change, backed by solid data.
Here are five things you can do to ensure that every A/B test is accurate, meaningful, and produces results.
1. Test Entirely Different Variations
This is a must regardless of whether you are testing offers, headlines, or copy. Sometimes small changes generate big results. Other times, this does nothing to move the needle.
For the most accurate testing, focus on two entirely different variations. This will give you a clear understanding of which direction to move. Sites with higher traffic levels should consider multi variant testing (MVT), to create all possible variations to find the best options for change.
2. Track and Let Tests Run
What good is A/B testing if you don’t track the results? You need to know what’s working, what isn’t, and how you can use the results to adjust your approach in the future.
Track testing dates carefully and let them run until you have a decent sample size. For some, those with high traffic websites, a test of a few days may be more than enough to generate meaningful results. For others, it may take a period of two to three weeks to receive enough impressions. This is important — you don’t want to skew your results with a too-small sample size, or to see an early lead as a win when perhaps it was just an anomaly.
Once you’ve established that your sample size and test length were sufficient, a conversion calculator can be used to assess your results.
3. Reduce ‘Noise’ and Distractions for More Accurate A/B Tests
You don’t want an outside influence skewing the results of your test. While it’s not usually not possible to control every factor that might affect content that’s already live on your site, try to reduce noise and distractions as much as possible.
For example, weekends or holidays can serve as a distraction. If you run one test from December 1 to December 14 and the other from December 15 to December 28, you might see entirely different results, especially in e-commerce. The holiday is sure to get in the way of the second test, thus impacting the results.
Another factor that can create a lot of noise are your other promotions and campaigns. For example, a targeted PR campaign consisting of a press release, some social media ads and a Google AdWords campaign could drive an entirely different type of traffic than usual, so you want to make sure you aren’t reading those variations as the result of your on-site CRO efforts.
Teams need to work together to ensure elements like social media marketing, PPC campaigns, PR activities and more aren’t skewing the results.
4. Avoid Common A/B Testing Fallacies
A/B testing isn’t foolproof. In fact, there are a number of common issues you can avoid if you’re aware of them.
Common testing fallacies include the beliefs that:
- You need a large budget to produce meaningful results.
- A/B testing only works on websites with a lot of traffic.
- If one test immediately generates results you can stop there.
- Correlation and causation are one in the same.
Remember, best practices and conventional wisdom are there are guidelines, but they aren’t gospel. Test them using your own data and see if the results play out the way they should, or if that specific type of test maybe just isn’t right for your e-commerce site, and your business.
5. Track Performance Across Multiple KPIs
Conversion rate is certainly one important metric to consider in your A/B test, but it’s important to look at other key performance indicators to make sure they are being affected positively, as well. Typically, a true winning variant has an impact on several key performance indicators.
In e-commerce CRO, you’re also going to want to track Average Order Value (AOV), Revenue Per Visitor (RPV) and Lifetime Value (LTV), at the very least. You need to make sure that driving up conversion rates keeps your quality in check; that it’s also driving positive change where it actually matters: in revenue.
Remember, though, that it’s important to remove outliers when tracking across KPIs, to ensure they don’t skew your results. For example, sites with a dedicated section for wholesalers don’t want those visits to skew the average order value of all visitors, since they aren’t typical converting visitors.
Now that you know your e-commerce site’s a/b tests are scientifically accurate, learn how to prioritize them to ensure you reap maximum benefit!