Direct Selling: Testing Makes Perfect
When developing format tests for customers, bear a few things in mind. First, customers let you mail them a wide variety of formats without major negative implications. As a result, you need to understand what your testing goals are for these mailings.
Make sure, too, that you don't have any additional tests taking place at the time of the format test. Customers often are targets for longitudinal testing where companies test to understand the effects of multiple mailings and contact sequence on response and retention. In those cases, you must control for the first study or risk compromising the results of both testing efforts.
Prospects-who don't necessarily know your brand or understand your offer proposition-are typically more finicky about how you communicate with them. As a general rule, more is better for direct selling to prospects. Greater page counts typically produce greater response rates. However, if the answer was that easy, only one format would exist. This is why you test.
Measuring Results, Retests and Rolling Out
The "Rule of 100s" is a good place to start when evaluating format performance. The rule claims that if a segment produces at least 100 responses, there is a greater likelihood that the result would duplicate in the future. In evaluating test results, the rule suggests that if dollars per piece mailed for format A was $2.19 based on 137 orders and dollars per piece mailed was $1.65 for format B based on 105 orders, A will most likely outperform B by about 30 percent again.
When gross responses are low, the limits of the Rule of 100s get tested. For that reason, it's a good idea to retest to validate results. Retesting may become essential when one format works well with one segment, but a different format works best for another segment, and it's tough to draw hard and fast conclusions. When this happens, retests to similar segments controlling as many potential confounders as possible are necessary. Retests also are good if you've tested a format to a small group but the overall roll-out quantity is significantly larger. It's important to validate the result with a larger sample before putting all of your eggs in the winning format's envelope, so to speak.