Strategy : Tested and Proven
The science of direct response testing
October 2008 By Steve Trollinger
|
But the term testing is used loosely. Some marketers contend putting two different pieces of creative in the mail at the same time and monitoring the resulting response equates to testing. And it does - if those marketers take the time to build a plan and execute it precisely to understand a specific answer to a specific question. The challenge is many direct marketers today stop short of building good, valid and reliable tests.
The Strategy of Testing
All good tests start at the same place: a good test strategy. Having a clearly defined outcome - a specific question to be answered - is the best place to start a test. For example, going beyond, "which version will work best?" to, "which version will produce the greatest lift per offer dollar?" is critical. You must know where you're going before you begin the trip, and your test should be established to answer the question you intended to ask, lest you sit with mountains of data at the end and nothing with which to make a decision.
Test the Big Things First, the Details Last
Because testing is such an expensive venture, you often get the most bang for your marketing buck by focusing early testing efforts on "the big things," those aspects of the program that have the greatest and most immediate impact.
* Creative. Creative tests are often among the most popular to conduct, typically in an effort to increase response rates. With catalogers, cover testing is by far the most regularly conducted because it's relatively inexpensive to test cover treatments, and the impact of getting it right can make a significant impact on performance, particularly response rates. Catalogers also test page count to determine the trade-off between response rates (typically higher with more pages) and costs (also typically higher with more pages).
In the solo package world, envelopes and envelope copy (both items make a difference in the piece getting opened, the obvious first step in and often greatest hurdle to response), letter versions, and offers/promotions are often the big things to test.
* Offers. Most marketers conduct offer tests to determine what promotion or combination of promotions will increase response rates or average order values. Reading results for offer tests, though, often falls short as marketers fail to account for the full cost of the test (i.e., the offer cost in terms of discounts or markdowns, as well as the actual cost of the marketing piece) and base decisions solely on the response rates or average order values the tests yield, without understanding the impact the promotion strategy has on the profit-and-loss statement.
Still, offer tests are important to understanding the drivers of response (particularly among prospects and inactive customers) and are the most cost-efficient way to build the customer file.
* Timing. Another big question direct marketers are constantly trying to understand surrounds timing and frequency of contact. For new customer acquisition, timing of mailings or ad placement, or more aggressive pay-per-click bidding, is critical to get the most customers for the smallest investment. For existing customers, you should know the value of the incremental mailing or replacing a mail piece with an e-campaign.
Should you mail earlier to beat the rush of pieces in the mail or mail later to be in the mail when customers are actually ordering? Should you mail in advance of major holidays and events (i.e., Thanksgiving and elections) or after? Should you mail deeper into your file during summer months or focus contacts on the cream of the crop or even a modified segmentation strategy? Should you replace a direct mail campaign with an e-mail campaign? Or three e-mail campaigns? These and other marketing questions need answers, based on your business and your customers, to help you reach your goals. There's no magic answer as to the optimal contact strategy. You have to test it to know.
These long-term tests, or longitudinal tests, require extended time frames to fully execute and understand. If you're testing a three-contact strategy versus a six-contact strategy, for example, you may need several months' worth of data collection to read the result. In order to execute that type of test, maintaining tight control over customers in each test group during the life of the testing period is critical. This means no reshuffling the deck with respect to these test cells-if they're selected, they're selected for the life of the test.
* Lists. In acquiring new customers, list testing, next to offer testing, is among the most fundamental must-haves in the marketing plan. By presenting a variety of prospect lists with an assortment of cover treatments and promotions, you can develop the optimal combination for bringing in new buyers.
As straightforward as list testing is, many marketers fall short on the interpretation, assuming a result will duplicate across all lists. Not so. It is common for a particular promotion to work well with one type of list but not another, depending on the composition of the outside file and the wants and expectations of the names on it. It's important to test your theories across all lists at various times of the year to assure the result you've produced is one that will replicate and can be trusted over time.
* Testing the Web. Homepage configuration, navigation, checkout pages - any element of the Web site that can be moved, repositioned, enhanced or versioned to influence site conversion - are candidates for testing. With the Web, the concept of testing the big things still applies, but the costs associated with those tests and the potential benefits of establishing better practices are tremendous.
Most hosting companies today have the ability to A/B split test content online in a manner that allows you to literally serve up a different version to every other customer or potential customer who comes to your site. Moreover, many of the testing capabilities allow you to establish tests for visits that come from specific activities, such as PPC programs or affiliate links. And as more customers go to the Web to place their orders, testing online and establishing controls as well as efficient practices are critical.
E-mail testing, too, is easy to do and can reap extraordinary rewards. Testing promotions, delivery time of day, delivery day of week, subject lines, total number of offers/clickthroughs, embedded navigation, rich media versus HTML versus text, content density, etc., all consist of relatively minor programming tweaks, which means, as tests, they're very easy to execute. But are they too easy?
A word of caution about testing online: When it's "so easy" to test so many different variables, it's also easy to get caught without a plan. Remember, testing starts with a plan and a question that needs answering. Just because you can test umpteen different variables in an e-mail campaign at once doesn't mean you should. Testing the elements that will produce the greatest effect first is still the best strategy, and where the Web is concerned, that means finding out what improves conversion to sale.
About Sample Sizes, Controls and Holdouts
A key aspect of testing is ensuring you've included enough records in a given sample to get a reliable (meaning it accurately measures what you were trying to measure) and valid (meaning the result is likely to duplicate over time) test. Mailing Offer A to 5,000 prospects and Offer B to 5,000 prospects to get 30 and 40 orders, respectively, won't tell you much. As a matter of fact, even though those figures equate to Offer B producing a 33 percent lift in response (.8 percent versus Offer A's .6 percent), the result is not statistically significant at these quantities.
As with sample sizes, maintaining control groups - those that get either the "established" winner version or get no promotion at all - is important to understand what would have happened if you'd simply done business as usual - no special offers, no revisions to the "base" approach. It's often not enough to know one offer outpulls another. You need to know what no offer at all would produce.
Holdout groups serve a similar role as controls, insomuch as they allow the marketer to observe what happens to those customers who get no test stimulus. This group may get no contacts at all ("what happens when we do nothing?") or may get the base contact strategy but no special or ancillary mailings. Simply put, these are records that are "held out" of a campaign in an effort to observe the effects of the campaign on customer behaviors, including migration and cannibalization, in particular.
Retesting and Rolling Out Your Findings
Retesting is an important part of the testing process. It is seldom sufficient to test a concept on a small, randomized sample and scale the findings to an entire population. Good test strategies incorporate methodologies for retesting concepts, particularly when results are marginally clear. Once a variable is retested and validated on a larger sample, rolling out can be surprising.
While rollout of a winning test often yields improved results - as you might expect from the test results - the scaled results (the response you get from mailing an offer to an entire population, for example) will undoubtedly be lower than the test results, given the distribution to a more robust sample. Don't be surprised by it; instead plan for it by forecasting modest decreases in your projections based on the results you saw from your retesting efforts. Being prepared is better than being surprised.
Test, Test, Test
Regardless of the sophistication of your direct marketing business, you should be testing at every possible opportunity. If you've tested a concept in the past but the business has shifted, the brand has been repositioned, go back and retest those concepts. If you've run out of big things to test and established an ironclad control, work on testing the details, tightening the screws as it were to improve results all the more. In the science that is direct marketing, testing is the path to greater success.
Steve Trollinger is executive vice president of J. Schmid & Associates, Mission, Kan. You can reach him at stevet@jschmid.com.
Companies Mentioned:
 |
|
The Business of Database Marketing
Cracking the QR Code
The Art & Science of Multichannel Fundraising