11 A/B Testing Best Practices for Digital Marketers


A/B testing is the practice of comparing two variables to see which variable more effective at maximizing a desired outcome. For example, a web designer can A/B test two landing pages to see which page has a higher click-through-rate on an advertisement. Or, an email marketer can compare two email subject lines to determine which one has a higher open rate. Once you have a solid A/B testing strategy in place you can really start to create content that is optimized for engaging your audience and increasing conversions.

We asked 11 digital marketers, what are some must-know digital marketing best practices for A/B testing?

1. Segment your traffic for A/B tests

Sergey Alakov, Humberview Group, @Sergey_Alakov
A lot of marketers run A/B tests without segmenting desktop and mobile, when in fact, that’s probably one of the most important things to do for your results to be reliable. Sample size is important, but the thing is, you need to have a big enough sample size for both desktop and mobile. Some things work for desktop, and don’t for mobile, and vise versa. So, if you don’t segment desktop and mobile you might really miss out.

2. Set weekly goals

Melissa Gosse, CanIRank, @CanIRank
Each week set a new goal tactic that cannot overlap with other goals previously tackled. For example, week 1 = FB ads, week 2 = Leadpages and week 3 = affiliate program. The marketing techniques that are going to work for you all depend on your niche, your target audience, your geographical location, etc. To really know which will work for your business and which will not you have to trial and error. One week is a preferred time length for testing as it is long enough to implement the strategy and get results plus short enough to try multiple marketing techniques over a couple months span. At the end of each month, look back at the marketing tactics you tried and look at your analytics (i.e. did your sales rise or decline? How many man-hours did it cost you to implement and manage that strategy? What was your engagement?). After answering those questions you will have a pretty good idea of what techniques will work for your brand without having to use an expensive complicated software.

3. Alter your images

Tory Kalousek, Blue Compass, @ToryKalousek
If your brand is launching a new product or service, you have to know what your audience responds to better. A/B testing, whether it's through mediums like Facebook, Bing or Google Adwords, provides an excellent way to see what your customers really want. One tip is to alter your images. For example, have the text in the center of the image on test A and the text off to the side on test B. Something as small as text can yield surprising results.

4. Focus on the elements that have the biggest impact on conversion

Sarit Neundorf, Integrate & Automate, @SmBizAutomation
Don’t sweat the small stuff. Follow the Pareto Principle (also known as the 80/20 rule) and identify the 20% of your web-pages or communication pieces that generate 80% of your results. Then, when you design your testing strategy for those assets, focus on the elements that have the biggest impact on conversion, such as headline, sub-headline, images and calls to action.

5. Include a chat widget on your site

Henry Butler, CanIRank
One A/B test to experiment with is including a chat widget on your website. Do users spend more time on your site when there is a chat box? Is it better to have the chat option pop up or hidden until users press a button?

A chat box allows for the first online interaction between a business and its customers. By testing different strategies, you can discover the ideal way to greet potential clients who are peering into your company’s virtual window.

6. Prioritize your tests

Dylan Whitman, BVAccel, @DylanWhitman
The key to testing is setting yourself up to get an actual ROI on your efforts. Avoid jumping in and running tests randomly. Always rank your potential tests by level of effort to implement, potential impact, and confidence of impact, and regularly re-prioritize tests based on new ideas and results.

Bonus tip: Check out Behave for actionable ideas!

7. Create a pre-test checklist

Amelia Willson, HostGator, @HostGator
When you're first setting up a test, start by giving yourself a pre-test checklist. If you can't answer the following 4 questions, refine your campaign until you can.

  1. What information are you hoping to learn from the test?
  2. Which variable are you changing?
  3. Can you generalize the test results? i.e. can you apply the learnings from this email to future emails, or better yet, can you apply it to other marketing efforts such as the design/UX of your website?
  4. Is your test sample statistically significant?

8. Test one variable at a time

Kristian Rivera, Fit Small Business, @FitSmallBiz
Don’t make A/B Testing more complicated than it is. When it comes to testing, be sure only to test one variable at a time. If you’re looking to get a better open rate, then begin with testing the subject line of your email. That’s the first thing your audience is going to see and will make or break your campaign. If you’re looking to increase the click through rate on purchases, then take a look at the content of your email. It can be many different things that affect your results. Take a look at the copy, images, font, buttons and other potential factors keeping you from getting the results you need. However, you don’t want to test more than factor at a time. If you do so, you won’t be able to find out where the source of the problem is and how to fix it.

9. Get ideas from your competitors

Ali Khan, SatchCorp Ltd
A step-by-step approach to quick, performance-led A/B tests:

  • Run search queries using keywords you and your competitors target.
  • Take the most highly ranked content and study it meticulously.
  • Create your first test creative to match your competitor's as closely as possible, in structure, content optimization and user experience.
  • Then draft your second test creative using your own ideas.
  • This is performance-led (your competitor is giving you the first creative). Monitor the first 1,000 visitors and discard losers quickly!

10. Create dramatic variations between your tests

Allison VanNest , Parse.ly Don't run headline tests for too long if you're running them for 100% of your audience. Create dramatic variations between your tests. The more dramatic you make the changes the more likely you'll find a meaningful winner. If your find a winner in ~1 hour, implement the winner. If the experiment hasn't found a meaningful winner in an hour, pause the experiment and move on, trying more dramatic variations next time. Lastly, set meaningful goals for your organization.

11. Study the data your business generates

Bryan Clayton, GreenPal
Sometimes big data sources can be right at your fingertips. Leverage your data, combined with publicly available census data, for marketing insights to come up with A/B tests for landing pages and pay channels.

For example, in a recent campaign we ran in Nashville, TN , we ran pay-per-click (PPC) Adwords campaign with one ad targeting the entire metro Nashville area. The headline read ‘Local Lawn Pros in Nashville are a click away.’ and the performance of the ad was good with a click through rate of over 1% and conversion rate of over 10% on the Nashville landing page but we wanted to improve on it.

We set out to make our brand more contextual and relevant to the viewer. So we researched census data, looking at the average income and home values throughout the Nashville area.

We found that an up-and-coming neighborhood, was populated with more working class, and a creative class demographic and we hypothesized this customer segment would be price sensitive but still not want to cut their own lawns. So we segmented those zip codes and only ran a specific ad for them, with a headline ‘The Cheapest Lawn Mowing in Nashville. Lawn mowing from $20.’

We then created a matching landing page. After running the ad for one month we saw over a 200% lift in the click through rate and and a 30% lift in on-page conversion. Studying the data your own business generates can tell you which of your online marketing campaigns works best. Do the ads appeal to your target market or another market altogether? The data may also point to completely new areas of customer interest.