Small Test Samples: The Key to Unlocking Big Results?
WARNING: Article contents will make statisticians very angry!
By Spyro Kourtis
I often run into companies that can't afford to test because they don't have the budget, the mail quantity or the time.
If you're a marketer, and it's your job to actually sell things, then listen up: The statisticians have it wrong when it comes to finding what works in the market.
I'm not saying that statistics is an invalid science. It's indispensable to what you do. But when you're halfway through the quarter, you have an aggressive sales number to hit and your VP is screaming at you to find what gets killer response, the last thing anyone wants to hear is "that sample's not large enough to be statistically valid."
You need what works, and you need it now.
Through 18 years of testing, and then testing and testing some more, I've found that statistically valid numbers are important and can be established once the market has spoken. But the practical use of testing (and much of the power of direct marketing) comes from direction the market can give you. And directional learning is better than statistically valid data if it means you get actionable intelligence on more variables. In those terms, the market can give you some great advice.
Let the Market Shout at You
When the market speaks, the saying goes, it shouts. You can test things like paper stock, color and other minutiae of your packages, but first, please, let the market speak on the larger things that matter, such as your offer, list and creative.
At The Hacker Group, we are able to test a virtually limitless number of variations through our proprietary PowerTest? methodology. It's the way we are able to listen to what the market is indicating it wants without having to sweat the academic question of statistical validity. The PowerTest has been giving us successful control packages for clients for years.
At the start of a new program, you need to find a control fast. A PowerTest allows us to break the elements being tested into a matrix of test cells so you can test and measure the largest number of variables possible.
Here's a hypothetical example of how we might use the PowerTest methodology to find a preliminary control in a simple, 30,000-piece mailing. For this example, let's say we're testing the following elements:
* Two package formats (Package X, Package Y);
* Three offers (Offer A, Offer B, Offer C); and
* Five lists of 6,000 each.
(Remember the 40-40-20 rule. A direct mail package's success relies 40 percent on the list, 40 percent on the offer and 20 percent on your creative. These are the major ways the market shouts at you.)
Using this process, most of our clients find a winneror several winnersin the first test.
You'll notice that the number of mail pieces in each cell is relatively low. A statistician probably would scoff at the results of a 1,000-piece sample. But we're not looking for classic statistical validity here. All we want is a direction. The market can give you an idea of what direction to take, and you can execute on the winning cells in what my firm calls a CheckTest? to confirm your PowerTest indicators. Then you can roll out on the winners quickly and furiously, refining your winning packages as you go. The important thing is that you're getting to winning lists, offers and creative executions as fast as possible.
In direct marketing, don't ever forget that the recipient makes the rules. As a marketer, it's your job to discover those rules and use them to win. This kind of testing is the way to listen to the market and to give it more of what it wants. Think of it this way: Unless you're setting aside 20 percent of your budget for testing both new and ongoing programs, you're not testing enough. It's an investment that will pay off.
You Think These Samples Are Small ...
Consider this: Many advertising agencies, marketers and research firms work with focus groups. Let's take a look at focus groups for a minute.
Imagine it. There are a dozen or so people around a table in a room with a large mirror on one side. They are all there together, told whom they're supposed to represent and what kinds of marketing they are going to be looking at. Once everyone has been instructed, the group is presented with marketing materials and asked, as representatives of their demographic or market segments, how they would respond.
That "would" is a mighty big word in this scenario. A great deal can go absolutely haywire with "would." Let's look at just a couple of examples of how focus groups don't represent what really happens in the market.
If your focus group has one charismatic member, or a member with a big mouth and a big opinion, he or she could sway the rest of the group and give you badly skewed results. Orand I've seen this happenthe very topic of discussion could be something people want to be seen as responding to in one way, but get them alone in their homes (that's where the real market is) and they do the exact opposite.
The beauty of direct mail is that it's a one-to-one communication between the seller and the buyer in the privacy of the buyer's home or office. It's a private moment happening among, say, 25,000 people at the same time.
A dozen people acting in a group will behave very differently than a single, well-targeted person reading a personal letter with an offer made especially for him or herin other words, a letter that uses the 40-40-20 rule well.
A Little More on 40-40-20
When you receive market information from testing, it's important that you are able to respond quickly and efficiently with your offer, list and creative. With such small samples, you need to be ready to turn on a dime when you receive new intelligence. A PowerTest is a great direct mail tool, but you need to have precise control over the working parts. They are:
*Offers. The right offer gives your target audience a reason to respond. Determining the right offer requires careful consideration of your audience and what excites it. Start with these basic offer categories and brainstorm to get inside the head of your recipient:
Related or hard offers such as discounts or free shipping. Related or soft offers like a free trial of your product or service or a white paper.
Non-related offers are the tchotchkes for which direct mail is famous. You'd be surprised how effective they can be.
Sell-through offers and sweepstakes are definitely worth testing as well.
Remember: Offers are an emotional driver. The more passion you can put into your offer, the better.
*List. Don't treat your lists as an inventory item. Even though there are thousands of names there, each one represents a personal conversation between you and the recipient. Granted, that conversation may be held with your mail dangling over the trash, but you've got that personal audience with each and every prospect or customer. And a PowerTest gives you the opportunity to try a virtually unlimited number of approaches to that communication.
*Creative Copy Platforms. It's important to test copy elements that address both sides of the human psychethe emotions and the intellect.
*Emotional copy: Ask yourself, what emotions will drive the decision? Test different approaches based on fear, greed, anger, exclusivity and salvation to reveal which emotion works best. Use a letter to be more personal and emotional.
*Intellectual Appeals: Test different benefit statements to create a variety of logical reasons to respond. Try testing a cost benefit against a productivity benefit with two different copy platforms, or test different leads based on different benefits. Use a brochure to present even more rational information.
Let's face it, as much as we may know about statistics, and however important they are to direct marketing, we're not in academia. We're here to sell things. As business people we need immediately actionable information from our tests. We're not going to have to defend marketing test results to a Ph.D. panel, but we will have to report to VPs, boards of directors and stockholders that our mail programs pay off in a timely manner.
Besides, when was the last time you saw a statistician actually sell anything?
Spyro Kourtis is president of The Hacker Group, a subsidiary of Foote Cone Belding Worldwide, located in Bellevue, Wash. The Hacker Group creates performance-based direct marketing programs for a number of Fortune 500 clients. Kourtis speaks nationally on direct marketing issues and is publisher of "High Performance Direct." He can be reached at email@example.com or (425) 454-8556.