The Problem With A/B Testing
This week we set up an elaborate A/B test on subject lines. I liked "How 1.75 Billion Mobile Users See Your Website" and my client manager liked "Business Cards are No Longer the First Impression." We learned long ago not to be a focus group of two, but our testing also proves something else I've been saying for years—A/B tests do not stand alone.
For our Mobile Users campaign, we dropped in an actual screenshot of every recipient's website as viewed on an iPhone 6 (see image), because we knew this level of personalization could add a sizeable bump to engagement. It's one thing to tell a recipient their website looks awful on a mobile device; it's another thing to show them.
At the end of the campaign, we will have sent under 10,000 emails, but before we get to the balance, we felt it was important to know which of the two subject lines would perform better. All of us want to have the very best chance of success, so this was a necessary step. Ensure our subject line would foster a higher open rate.
For our initial test, we sent 600 emails, half to each subject line. One subject line performed best with opens, the other subject line performed best for clicks to the form. What that means is we now have a new question: is it better for us to get more people to open and see the message, or is it better to get fewer people to open, but to have accurately set their expectation about what was inside so they would click?
The open rate differed by more than 10 percent, and the CTR by about 2 percent.
Should I stop my analysis here and answer the only question I started with (which subject line should we use), or would it be better to take a look at other factors and try to improve the overall success in any way we can? For me, the problem I see with many marketers' A/B tests is they ask one question, answer it, and then move on. In fact, many email automation systems are set up in precisely this manner: send an A/B test of two subject lines, and whichever performs better, use it to send the balance. What about the open rate and the CTR combined? Isn't that far more important in this case (and many others)? Let's take it one step further: what about the open rate, CTR and form completion rate combined? Now we're on to something.
Email marketing is the most effective way to increase sales, improve service, and keep your customers engaged. Email campaigns are best bolstered through an integrated strategy that crosses channels and meets your constituents where they congregate and in the media they prefer. “The Integrated Email” provides best practices and ideas for developing strategies and deploying email campaigns and initiatives while keeping an eye on revenue attributable to marketing.
Cyndie Shaffstall, founder, Spider Trainers, is a successful entrepreneur and prolific author, with many books, dozens of eBooks, and hundreds of articles to her credit. She is the former founder of ThePowerXChange, editor and publisher of X-Ray Magazine, and the current founder and managing member of Spider Trainers, a managed automated email services provider for companies around the world. Connect with Cyndie on LinkedIn, Twitter, Google+, or join her LinkedIn Group, the Marketing Resource Library for daily links to marketing-critical resources.