Direct Selling: Testing Makes Perfect
By its very nature, format testing makes it difficult to strictly control every variable except the creative format alone. Often, testing a self-mailer versus a catalog, for example, requires a change to the overall message to accommodate less physical space.
As an example, assume you are testing a multipanel self-mailer versus a 16-page catalog to a segment of customers. Your goal is to determine which produces the most profitable sales or greatest ROI. Assume your self-mailer is merchandised with best-sellers, key branding messages and heavy emphasis on driving traffic to the Web, while the catalog is a typical piece that you might mail-multiple SKUs, a wider assortment, etc. When the results come in and the catalog wins, does that mean it's the better format? Not necessarily.
You may have merchandised the self-mailer wrong; it may have mailed too early versus the catalog; your Web site may have been ill-equipped for the campaign and sent customers away before buying. In other words, a variety of factors may have muddied the waters with respect to conclusions you can draw. Those factors are referred to as confounding variables, which generally can't be controlled but should be considered in your conclusions.
So, what should you consider and attempt to minimize as you set up your format testing? The big three are: messages, offers and timing. If at all possible, the same copy (or elements of the same copy including tone, detail of information, etc.) should be used in each of your creative test pieces.
Second, each format must present the same promotional offers and the same channel ordering options in the same ways. If your URL is a prominent front cover element on one creative but is buried on the other, customers may not respond because they don't know where to go.
Finally, the timing of the campaigns must be consistent. For example, full-size catalogs mail at the Standard Flat rate and often take five to 10 days to arrive, while smaller direct mail pieces such as postcards, self-mailers and solo packages often intermingle with First Class mail and deliver in as few as three to five days. That difference in timing can play a role in response. Customer and Prospect Format Testing