Scientific multivariable testing is gaining traction in the marketing world because it offers deeper insights with greater accuracy, smaller sample sizes and more efficient use of resources than common split-run techniques. One big challenge: Multivariable testing is not one thing, but a collection of diverse (and sometimes complex) statistical techniques and test designs developed over the last 80 years.
The primary difference between a split-run and scientific multivariable test is the number of variables changed in each test cell. To test 15 elements in your direct mail package (or catalog cover or e-mail effort), you can use 15 one-variable splits plus your control. Or you can use 16 packages to run one multivariable test. With a multivariable approach, you still need to mail a number of different packages. The key difference is each "recipe" (version of the package) includes some combination of all the variables in the test and gives an additional data point - another perspective - on every test element. The complex statistics underlying the selected test design defines which specific combinations you need to mail to have a clear read on results. (For example, download this case study.)
The Broccoli of Direct Marketing
Multivariable testing should be a part of every healthy marketing program, but you can have too much of a good thing. A good strategy is to plan your objectives for key campaigns over the next six to 12 months. After you decide what questions you want to answer, the right test design can be selected for each objective and campaign.
Planning a Healthy Testing Program
The full range of test strategies can be grouped into three main classes. At one extreme are one-variable split-run tests. The other end of the spectrum is large "screening" tests of 10 to 25 variables designed to screen the few important marketing elements from a pile of new ideas. In between are smaller "refining" tests focused on a few important elements, tested in more combinations to increase accuracy and better quantify interactions - where the impact of each element may change depending upon how others are set. These three test designs are summarized in the chart at the end of the story.
Generally, a good approach is to consider a cycle of testing, asking:
1. Does it work? Run a split-run test to see if a program (or list, new creative or offer) is viable.
Social Media ROI
Email Marketing that Works (2nd Edition)