|
Scientific multivariable testing is gaining traction in the marketing world because it offers deeper insights with greater accuracy, smaller sample sizes and more efficient use of resources than common split-run techniques. One big challenge: Multivariable testing is not one thing, but a collection of diverse (and sometimes complex) statistical techniques and test designs developed over the last 80 years.
The primary difference between a split-run and scientific multivariable test is the number of variables changed in each test cell. To test 15 elements in your direct mail package (or catalog cover or e-mail effort), you can use 15 one-variable splits plus your control. Or you can use 16 packages to run one multivariable test. With a multivariable approach, you still need to mail a number of different packages. The key difference is each "recipe" (version of the package) includes some combination of all the variables in the test and gives an additional data point - another perspective - on every test element. The complex statistics underlying the selected test design defines which specific combinations you need to mail to have a clear read on results. (For example, download this case study.)
The Broccoli of Direct Marketing
Multivariable testing should be a part of every healthy marketing program, but you can have too much of a good thing. A good strategy is to plan your objectives for key campaigns over the next six to 12 months. After you decide what questions you want to answer, the right test design can be selected for each objective and campaign.
Planning a Healthy Testing Program
The full range of test strategies can be grouped into three main classes. At one extreme are one-variable split-run tests. The other end of the spectrum is large "screening" tests of 10 to 25 variables designed to screen the few important marketing elements from a pile of new ideas. In between are smaller "refining" tests focused on a few important elements, tested in more combinations to increase accuracy and better quantify interactions - where the impact of each element may change depending upon how others are set. These three test designs are summarized in the chart at the end of the story.
Generally, a good approach is to consider a cycle of testing, asking:
1. Does it work? Run a split-run test to see if a program (or list, new creative or offer) is viable.
2. What can we change to make it better? Run a large, screening multivariable test of the current control to gather lots of information quickly about the most important variables and low-cost opportunities for improvement.
3. Now that we know what's important, how can we squeeze out even better results? Run a smaller, refining test of a few key marketing-mix elements to leverage insights from previous tests and analyze interactions. These focused tests can answer questions like:
- Since a teaser on the envelope helps, can we come up with a stronger teaser?
- Can we raise price further if we offer a great premium?
- What combination of price and shipping discounts, deadlines, and terms will maximize profit?
4. Where do we go from here? Continue the cycle of testing - with ongoing A/B splits of new creative and another wave of testing - as the marketplace changes.
Multiple Test Strategies for a Multitude of Questions
For best results, someone on your team should have a good understanding of the full range of available test designs and techniques. So often you read of "factorial," "optimal" or "Taguchi" tests. But these are only three out of dozens of available testing strategies. Your choice should be driven by your unique objectives and marketing programs, rather than a favorite flavor of the month. Every test has its pros and cons - no one design is the perfect answer for every question.
"Testing is still the best way to find true breakthroughs." This quote from the 8th edition of "Successful Direct Marketing Methods" by Bob Stone and Ron Jacobs sums it up. Testing gives you the power to prove what sells. And scientific multivariable testing offers new tools to build insights with greater speed, flexibility and pinpoint accuracy. What questions are most pressing? Start with your objectives, list ideas, and find the right blend of test designs and strategies to achieve your goals.
Gordon Bell is president of LucidView. His research, writing and consulting over the last decade has focused on bringing advanced testing techniques from academia to the front lines of direct marketing, Internet and retail programs. He can be reached at (888) 582-4384 x707.
|
The Business of Database Marketing