The Problem With A/B Testing
There are many factors at work here: time of day, past engagement, lifecycle and more. The subject line is a good place to start, but I can't afford to ignore what we've gleaned from other campaigns.
This then becomes the hardest part of testing—be that A/B or multivariate—isolating what we've actually learned, and that usually means I cannot analyze just this one campaign. It must be an aggregate.
For our campaign, I took our test results and put those into a spreadsheet of 2014 campaign results and started to look for patterns. We've all read Thursday mornings are good (as an example), but does that hold true for my list? Were my open rates affected by time of day, by date, by day, by business type, by B-to-C vs. B-to-B? These are all analytics we track because we've found each of these does, in fact, influence open rate.
So, yes, we did learn which of the two subject lines performed better for opens, but what we also learned is that a repeat of the test to another 600 recipients on Tuesday morning instead of Thursday morning resulted in almost exactly opposite performance.
A/B tests can be hard. If they were easy, everyone would do them. Our simple one-time test was not enough information to make decisions about our campaign. It took more testing to either prove or disprove our theories, and it took aggregating the data with other results to paint the full picture.
We did find a winner: an email with a good subject line to get it opened, good presentation of supporting information inside, that led recipients to a form they actually completed, and all sent on the right day at the right time, from the right sender,
While you're not privy to all of the data we have, on the top of the subject lines alone, which do you prefer?
Email marketing is the most effective way to increase sales, improve service, and keep your customers engaged. Email campaigns are best bolstered through an integrated strategy that crosses channels and meets your constituents where they congregate and in the media they prefer. “The Integrated Email” provides best practices and ideas for developing strategies and deploying email campaigns and initiatives while keeping an eye on revenue attributable to marketing.
Cyndie Shaffstall, founder, Spider Trainers, is a successful entrepreneur and prolific author, with many books, dozens of eBooks, and hundreds of articles to her credit. She is the former founder of ThePowerXChange, editor and publisher of X-Ray Magazine, and the current founder and managing member of Spider Trainers, a managed automated email services provider for companies around the world. Connect with Cyndie on LinkedIn, Twitter, Google+, or join her LinkedIn Group, the Marketing Resource Library for daily links to marketing-critical resources.