5 tips: How to do A/B testing well as a marketer

Home > Blog > 5 tips: How to do A/B testing well as a marketer

I’ve talked before here about how important—and easy—it is to do things wrong quickly with digital marketing. That is, to try an idea out and then adjust it or even abandon it if it’s not working. One of the ways to do this is through A/B testing. Here are five tips:how to do A/B testing well as a marketer.

A/B testing is one of those things that is super simple to do but often isn’t done and that’s a real shame because testing is the only way to know if you’re marketing is really succeeding.

If you’re not familiar with A/B testing, it’s exactly what it sounds like: two versions of the same element (A and B) compared against each other simultaneously. For example, to determine which of two landing pages is better, you might send a link to one in an email to a small percentage of your mailing list and a link to the other to an equal portion of the list. You’ll know which ever page gets the most attention is the better page and you can then use that page with the remainder of your mailing list.

Here are some tips for successful A/B testing:

1. Test for your goals. The most important part of any test is knowing what to measure. It’s easy to measure things like clicks and page views, but are those metrics important to achieving your goals? Do they directly translate into sales? Do those things directly translate into sales for you? What if you run a campaign that significantly boosts your page views, but sales remain stagnant, or even drop? If you’re only measuring page views, you would say that campaign was a success when it really wasn’t. Test for your business goals and measure what matters.

2. Do one thing at a time. On the old TV show “M*A*S*H,” the finicky surgeon Major Charles Winchester once said “I do one thing at a time. I do it very well, and then I move on.” This is a good approach to take with A/B testing. Unless you have substantial experience and expertise with statistical research, test only one variable at a time.

3. Control for contaminating variables. While it’s usually best to only test for one thing at a time, a sloppy marketer might test multiple things at once by accident and ruin the experiment. A contaminating variable statistical speak for any variable that you’re not measuring, but which could interfere with the results (thereby contaminating them). For example, if you’re testing the effectiveness of two different landing pages, direct people to the different pages the same way. If you direct people to one page via Twitter and another via an email blast, then you won’t be able to gauge the effectiveness of either page since two different populations (email recipients vs. Twitter followers) and two different transmission methods have been introduced into the equation. If you want to measure all those things, share both landing pages via email and via Twitter.

4. Test A and B version simultaneously. While it may be tempting to do a control group and then a variation, that would actually introduce contaminating variables (time period, etc.) to the experiment. Always test both versions simultaneously, directing some traffic to A and some to B.

5. Gut feelings have no place in A/B tests. The purpose of an A/B test is to show empirically which element is better. The most effective element may not be the best looking design or have the cleverest turn of phrase in the copy, but if it works, it’s the winner. Don’t let a gut feeling override what the data says (or if you’re going to do that, then just skip the hassle of the test altogether).

There you have it, five tips to make your next test a success. Good luck!

Get Access to Our Ideas, Tips & Best Practices


50 East Ridgewood Avenue
Suite 304
Ridgewood, NJ 07450

Copyright © 2016-2024 OnDemandCMO, Inc. All Rights Reserved.

Get Our eBook

7 Steps of Marketing

onDemandCMO 7 Steps of Marketing eBook cover

OnDemandCMO has authored 7 Steps of Marketing, the only marketing guide book you’ll need to either get your marketing started properly, or stay on track strategically.

It features best practices on branding, messaging, social media, lead generation and much in between.

Please let us know who you are, and we'll share a few of our secrets (we don't sell or trade your info)!