5 Important Answers to Your Email Testing Questions
Marketers have been proclaiming the importance of A/B testing for years, but we don't always practice what we preach as deadlines and last-minute edits get in the way of our best-laid marketing plans.
I recently was a presenter for the Direct Marketing IQ Brunch & Learn webinar, "7 Email Tests to Run Today (and Tomorrow)," and if you had the chance to attend (or viewed the webinar on-demand, which is still available), you can probably tell I am very passionate about email testing and could have spoken on the topic for far more than 25 minutes.
Email testing can provide continuous boosts to your revenue and improves your knowledge of your audience-two major wins that are hard to ignore, particularly with how easy A/B testing can be.
Although we ran out of time to answer all of the questions live, I've answered as many as I could below. If you still want to chat about email testing, you can tweet at me anytime!
1. What is the minimum test segment quantity, to be statistically significant? In my direct mail days, it was 12,500 minimum, but we NEVER have anywhere near those quantities in our email campaigns. They are small, VERY targeted.
For this, I recommend using a segment size calculator and a statistical significance calculator. Determining a proper sample size is more about your baseline conversion rate, and the minimum percent improvement you'd like to be able to see.
It's definitely harder to test as your segments get smaller and smaller. I'd recommend finding places to test where you can, and extrapolating those learnings into your smaller sends.
2. How do you decide who is in the "A" group, and who is in the "B" group?"
The subscribers that go into each test group should be completely and totally random! If you handpick people for either group, your results are going to be biased.