Straight Talk: Test Talk
For Tammy Nelson, marketing manager of The American Express Property Casualty companies, testing is not just a good idea, it's a way of life. All of Nelson's marketing efforts for the financial services giant's auto and home insurance division are done through direct channels. This means that each and every communication needs to be more clear, more persuasive and more actionable than her competitors', who can rely on field agents, TV promos and space advertising to support their message. To ensure that her direct mail efforts are up to par, Nelson relies heavily on testing. "We drop at least every week, all year round, and we have at least one test in every drop," Nelson asserts. "And it's not just one test, it's a test in every single segment we are marketing to." Nelson spoke with me recently about how preplanning, number crunching, vigilance and business sense can improve both the present and future of your testing program.
TG: What are some of the most common testing mistakes marketers make?
TN: One, they don't do a lot of preplanning. We get excited about some new insert opportunity or, for example, Customized MarketMail ... and we just jump in and test it. But really there should be some preplanning to decide [if] it is even possible [to] get a result that would break even with the control package. That was the case when we looked at Customized MarketMail. When we did the number crunching, we ended up choosing to not move ahead and test it because the lift we needed to cover the cost difference was just too huge.
A second thing is to set up valid tests and controls to isolate the variables you are testing and then choose a sample size that will yield statistically valid results. People always ask, "What is the rule of thumb? How many mailings do you need to drop to have a statistically valid result?" And my advice is to go away from rule-of-thumb numbers and calculate the number of responses you would need to truly evaluate a test.
Finally, I hear a lot of marketers say, "Our testing budget always gets cut. No one sees the value in it." As marketers, we have the power, we have the numbers in our hands to show the value of the testing that we do. Over a large amount of mailings, even with a small improvement, if you even find one or two winners out of 10 and you actually do the math, you can show the benefits of testing, either by showing how many more sales you can get with that improved package, or how much money you can save by dropping fewer packages. We do the test, but we fail to ... put it into business terms that can help us maintain our testing budget further down the road.
TG: Do you see any current trends that will impact how marketers test?
TN: One of the big challenges right now is multichannel. We have found without a doubt that multichannel is absolutely the way to go ... but trying to track that is an absolute nightmare. I try to balance wanting a crisp, clean rigor around measuring each single element with the business results. There are times when ... even if you can't measure every single element individually and understand the interactions between them, you still know it's the right thing to do for the business. Sometimes, it's hard to find that balance ... you just have to be comfortable with a little bit of ambiguity knowing that the multichannel efforts are the right way to go.
Also, an important testing trend, and something that we are looking to do, is ... multivariate testing. It's using advanced scientific techniques to test many variables at once while still controlling for each individual element. I think that's really going to be a big plus in the future. Right now when we do a simple A-B split, it can be a full year from the time we have an idea to the time we actually rollout that package. Grid testing would certainly speed that process up.
TG: How will a postal rate hike affect your testing program?
TN: A rate increase will affect our overall marketing mix, but I don't intend to limit testing because of it. Higher rates will make it more important than ever to have the best-performing package possible and the only way to get that is through testing. We are preparing for the postal increase by ... building more predictive models to fine tune our direct mail targeting so we can reduce volumes.
TG: What testing advice would you offer other mailers?
TN: First, do it, and do it often. And then turn those results into business results so that you can hold on to that testing budget and can prove to people, if they ever try to take your money away, how important it is. Second, take your testing budget and allocate it between tweaking your control packages and trying brand new things. Finally, have rigor around your tests and controls and sample sizes, but balance that with the fact that we are trying to run a business and get results for the business.