B-to-B Insights: Test Your Way to Success

How to quickly find your control

June 2006 By Russell Kern

multipage

Step One: Designing Your Test
The first step of your test program will determine the key success drivers: Which media channels will work for the offer? Which creative appeals will work for each list source, given a stated offer?

To illustrate this step, let’s consider a 25,000-piece program for a client I’ll call ACME. Here are the key assumptions driving the test program:

• ACME’s total market universe is estimated at 75,000 suspects. However, with limited information about campaign performance, the company has budgeted only a 25,000-piece test.

• Given a conflict between sales and marketing, we need to test a postcard and a letter package format.

• The sales manager thinks the sales leads generated so far from direct campaigns aren’t qualified. Therefore, we need to test three offers: product focus offer vs. a third-party educational offer vs. an educational offer with an incentive for immediate response. The sales manager has told the marketing leader, “I want to prove that our product is so good, it will sell itself.” So he’s demanded a test that promotes only the product. The marketing department knows better and would like to demonstrate the benefits of an early buy cycle educational approach. So it’s requested tests of a few soft offers involving an executive report prepared by an independent third party.

• Media research has shown that we need to test five list sources. Five thousand records are available for testing from each source. Four sources are controlled circulation subscriber files from trade publications in which ACME has advertised, and the other is a file compiled from D&B that ACME can use at half the cost per thousand of controlled circulation files.

• A 2 percent or greater response is required for any cell to be within the acceptable range for cost per inquiry.

Assuming all of the above, the testing grid would look like the Test Matrix shown below.

Although each of these 30 cells contains 833 names, it doesn’t mean the result from each cell is, by itself, statistically valid. For each cell in the grid to be statistically valid, the test would have to involve roughly 150,000 pieces, or twice ACME’s 75,000-piece mailing universe.

However, much can be deduced about market desire from the results of a single cell—when looked at in the aggregate. When conducting this type of multivariate testing, you are looking to identify strong trends and hot spots that will tell you what and what not to do next.

Let’s assume the test generated the response rates shown in the Response Rate Matrix (see below). In this example, it is safe to conclude the following:
• Nothing worked with List One.
• Package Y is a clear winner over Package X.
• Offer C worked best with Lists Two, Three and Four.
• Offer B worked best with List Four.

Step Two: Confirm Your Findings, Expand Your Testing
This is the point at which you conduct a confirmation test of the results from step one. This time, however, you must use only statistically valid sample sizes within test cells to confirm your results with confidence. Here’s a good rule to follow: You need 50 observations from a given cell, drawn from a sample size of 5,000 to read your results at a 95 percent confidence level.

Assuming we can find a new list that matches the profile of List Four and can replace List Five, with a response rate of 2 percent or higher and test cell counts of 5,000, the results from this confirmation test would be statistically projectable, as illustrated in the Check Test Cell Counts Matrix (see below).

Now you see how this multivariate testing methodology can identify a winning control package with just 65,000 names, as opposed to the 150,000 that would have been required for a test of this magnitude using traditional methodology. This could translate into a savings of $25,000, or even $50,000—money that now can be redeployed for a third campaign.

A Final Word
When developing your testing, always look for ways to do it inexpensively. Use laser imaging or black-plate changes to test splits. Once you have identified your control packages, use the 80/20 rule to keep pushing your campaign performance. Run 80 percent of your campaign using your control message, offer and list strategy, and then invest the other 20 percent in testing big differences that really can move the needle.

Russell Kern is president of The Kern Organization, a fully integrated offline and online direct marketing agency in Woodland Hills, Calif. He can be reached at (818) 703-8775 or via e-mail at russell@kernorg.com.

• MORE ON
• B-TO-B