Catch the E-mail Testing Bug!
3. Determine which testing methodology will provide necessary insights.
Three primary methodologies can be leveraged in e-mail testing. Each of these methodologies has advantages and disadvantages you’ll want to consider when preparing to design an e-mail test:
• Split testing (i.e., A/B, A/B/C) involves testing a single factor, such as subject lines, images or price points. The advantage is the speed and ease of split testing. To conduct the test, devise different options of the factor (i.e., two subject lines, three price points) to present to test groups. For the results to be valid, nothing else about the messages should be different. For example, if you’re testing subject lines, send the messages at the same time, and make sure the e-mail content is identical—only the subject lines would change. After most responses come in, compare the response rates to see if the difference between results is significant. If so, you’ve found your winning subject line.
• Multivariate testing allows you to simultaneously look at several factors, such as price points and offers, and evaluate the interactions of those factors by creating a test grid. For a simple example, let’s assume you’re selling a single product in an e-mail. You want to test two price points and have a long- and short-copy version of the e-mail. The test grid would contain four test cells. You then create four e-mails and four randomized test groups. Then run the test.
When analyzing the results, you may find that e-mails with the short copy have a better conversion rate than long copy, and the lower price point performed better than the higher one. Had you run sequential A/B split tests of these factors, you may have decided to send e-mails with the lower price point and short copy.
However, when looking at the interaction of price point and copy, you find that short copy wasn’t always the winner. When combined with the high price point, the conversion rate actually was poor. The same goes for long copy and low price point. Most importantly, the long copy version combined with the higher price point performed just slightly worse than the winner.