Direct Selling : Testing Makes Perfect

Creative format testing - from customers to prospects

September 2008 By Steve Trollinger

multipage

You may have merchandised the self-mailer wrong; it may have mailed too early versus the catalog; your Web site may have been ill-equipped for the campaign and sent customers away before buying. In other words, a variety of factors may have muddied the waters with respect to conclusions you can draw. Those factors are referred to as confounding variables, which generally can't be controlled but should be considered in your conclusions.

So, what should you consider and attempt to minimize as you set up your format testing? The big three are: messages, offers and timing. If at all possible, the same copy (or elements of the same copy including tone, detail of information, etc.) should be used in each of your creative test pieces.

Second, each format must present the same promotional offers and the same channel ordering options in the same ways. If your URL is a prominent front cover element on one creative but is buried on the other, customers may not respond because they don't know where to go.

Finally, the timing of the campaigns must be consistent. For example, full-size catalogs mail at the Standard Flat rate and often take five to 10 days to arrive, while smaller direct mail pieces such as postcards, self-mailers and solo packages often intermingle with First Class mail and deliver in as few as three to five days. That difference in timing can play a role in response. Customer and Prospect Format Testing

When developing format tests for customers, bear a few things in mind. First, customers let you mail them a wide variety of formats without major negative implications. As a result, you need to understand what your testing goals are for these mailings.

Make sure, too, that you don't have any additional tests taking place at the time of the format test. Customers often are targets for longitudinal testing where companies test to understand the effects of multiple mailings and contact sequence on response and retention. In those cases, you must control for the first study or risk compromising the results of both testing efforts.

Prospects-who don't necessarily know your brand or understand your offer proposition-are typically more finicky about how you communicate with them. As a general rule, more is better for direct selling to prospects. Greater page counts typically produce greater response rates. However, if the answer was that easy, only one format would exist. This is why you test.

Measuring Results, Retests and Rolling Out
The "Rule of 100s" is a good place to start when evaluating format performance. The rule claims that if a segment produces at least 100 responses, there is a greater likelihood that the result would duplicate in the future. In evaluating test results, the rule suggests that if dollars per piece mailed for format A was $2.19 based on 137 orders and dollars per piece mailed was $1.65 for format B based on 105 orders, A will most likely outperform B by about 30 percent again.

When gross responses are low, the limits of the Rule of 100s get tested. For that reason, it's a good idea to retest to validate results. Retesting may become essential when one format works well with one segment, but a different format works best for another segment, and it's tough to draw hard and fast conclusions. When this happens, retests to similar segments controlling as many potential confounders as possible are necessary. Retests also are good if you've tested a format to a small group but the overall roll-out quantity is significantly larger. It's important to validate the result with a larger sample before putting all of your eggs in the winning format's envelope, so to speak.

Finally, when evaluating results of format tests, always consider the cost to roll out the campaign, not just the test cost. In testing, quantities often are controlled and lower than they would be in a roll-out situation. As a result, per piece costs often are high and can negatively impact the numbers of the test. Only by paying attention to roll-out costs can you see what the real effect is and confidently discover your new winning format.

Steve Trollinger is executive vice president of J. Schmid & Associates, Mission, Kan. You can reach him at stevet@jschmid.com.