Strategy : Tested and Proven

As with sample sizes, maintaining control groups - those that get either the "established" winner version or get no promotion at all - is important to understand what would have happened if you'd simply done business as usual - no special offers, no revisions to the "base" approach. It's often not enough to know one offer outpulls another. You need to know what no offer at all would produce.

Holdout groups serve a similar role as controls, insomuch as they allow the marketer to observe what happens to those customers who get no test stimulus. This group may get no contacts at all ("what happens when we do nothing?") or may get the base contact strategy but no special or ancillary mailings. Simply put, these are records that are "held out" of a campaign in an effort to observe the effects of the campaign on customer behaviors, including migration and cannibalization, in particular.

Retesting and Rolling Out Your Findings
Retesting is an important part of the testing process. It is seldom sufficient to test a concept on a small, randomized sample and scale the findings to an entire population. Good test strategies incorporate methodologies for retesting concepts, particularly when results are marginally clear. Once a variable is retested and validated on a larger sample, rolling out can be surprising.

While rollout of a winning test often yields improved results - as you might expect from the test results - the scaled results (the response you get from mailing an offer to an entire population, for example) will undoubtedly be lower than the test results, given the distribution to a more robust sample. Don't be surprised by it; instead plan for it by forecasting modest decreases in your projections based on the results you saw from your retesting efforts. Being prepared is better than being surprised.  

Test, Test, Test
Regardless of the sophistication of your direct marketing business, you should be testing at every possible opportunity. If you've tested a concept in the past but the business has shifted, the brand has been repositioned, go back and retest those concepts. If you've run out of big things to test and established an ironclad control, work on testing the details, tightening the screws as it were to improve results all the more. In the science that is direct marketing, testing is the path to greater success.

Steve Trollinger is executive vice president of J. Schmid & Associates, Mission, Kan. You can reach him at stevet@jschmid.com.