Put Your Tests to the Test
How many times has your creative team laid odds on who could pick the winning package among your test panels, only to find that everyone had bet on a loser?
The reason why testing is an art and a science is that predicting human
behavior can be tricky. Not only is it difficult to determine what different target audiences might find interesting and of value, but it also is challenging to take your own bias out of the equation when selecting creative and offer positioning you think will be successful.
Results from prior testing experiences give you insight on what approaches haveand have notresonated with audiences. But analyzing historical research is like "looking into a rearview mirror," says Sandra Blum, proprietor of Blum Direct, a direct marketing consultancy and creative firm in Fairfield, Conn. A good complement to historical research and your intuition, she explains, is pretesting, which better allows you to get "a sense of the future."
Test Elements That Tell the Future
Pretesting is the testing of your ideas to determine which show the most promise for further exploration. While you can use this technique to zero in on any number of variables, Grant Johnson, president and CEO of Johnson Direct, a full-service direct marketing agency based in Brookfield, Wisc., explains that pretesting is best suited to helping you "look at big things and better determine if they will work."
Offer testing is a major part of direct mail success, but messaging often can be the more difficult aspect to pin down. Pretesting helps you make sure the message you are presenting is cogent and thatfrom the recipients' perspectiveit's the one you intended it to be. Most importantly, Johnson says, you can learn if your positioning is different enough to separate you from your competitors and not simply another ho-hum, "me too" message.
Blum finds that even if continual pretesting doesn't produce any major breakthroughs, it often results in short-term ideas that can be leveraged to spruce up what you're already mailing until you develop enough ammo to strike out in a whole new direction.
Since pretesting allows you to take a peek at a range of possibilities, it also can turn you on to an offer or creative approach that you never would have considered previously, says Johnson.
Just as important as finding out what could work is learning what doesn't seem to hold water, Blum adds. You could waste a ton of money on full test panels that, according to pretesting results, were destined to go nowhere.
The Online Environment
Just about any channel can be used by marketers to try out their concepts. For example, Johnson has conducted such research via focus groups, telephone research, Internet surveys and research, direct mail test panels, and e-mail.
Focus groups can require a little more of an investment than some direct marketers are prepared to make. For this reason, along with its benefits of speedy results and versatility, e-mail is highly prized as a pretesting vehicle.
The e-mail channel is great for determining if variable messaging could create a significant lift in response, says Blum. She pretested such an idea within a client's e-mail newsletter that was informational in nature, but offered her the option to embed a promotional message with variable content. (To simply have developed a promotional e-mail with variable content might not have pulled strong open or read rates, leading to problems in analyzing results across the test panels.)
What would have taken quite a bit of time to set up, mail and read via
direct mail was handled much faster via e-mail, Blum explains.
In addition to variable content, direct marketers have found that HTML e-mail allows them to pretest ideas for outer envelopes with multiple subject lines and headlines, as well as graphics above the fold. With an e-mail audience large enough for segmentation, you can get a quick read on what works, says Blum. Different offer presentations also perform well in e-mail tests.
But the e-mail environment is not a sure thing. First of all, says Blum, you should not use e-mail as a pretesting environment if you don't already have a solid history of selling your product or services in this channel. Your customer base must be comfortable with this realm, she states. Also, you need to manage issues such as deliverability (what if a large section of one panel never reached recipients?) and tracking (what e-mail link(s) were clicked on and what did recipients do when they reached your landing pages?). Without this information, your pretests aren't much help.
E-mail isn't the only online pretesting tool available to direct marketers. Banner ads, interactive surveys and plain old Web site versioning offer further insight into offers, creative, products and new markets worth closer examination.
Blum makes it a habit to ask her clients for the results from any of their online testing or surveying efforts. For example, National Geographic magazine involves site visitors by soliciting their vote on which of three possible features listed will make the cover story for the next month's issue. While the top covers selected by Web site visitors don't immediately translate into surefire headlines or creative platforms for a direct mail campaign, they could give Blum more feedback on what types of editorial content intrigue readerscurrent and potentialthe most. Depending on how similar your online and offline customers are, your offer and product testing could be more insightful.
The Pretest in Print
What better place to experiment with a few direct mail pretest ideas than in a few direct mail packages? Blum has culled meaningful results from the insertion of buckslips into control acquisition packages to assess the effectiveness of various premiums and additional creative concepts. "Since you're getting results from the same stream of response, it can tell you which direction to take when the current mailing wears out," she explains.
Working with a publishing client, Blum also has pretested ideas via buckslips in renewal efforts, as well as via bind-ins and blow-ins. This direct marketer had noticed a jump in newsstand sales, and decided to conduct a full-scale pretesting programincluding cover wraps and Web adsto learn three things:
1. Was there a new market (or markets) for its publication?
2. Could it attract more of these people to subscribe?
3. Would this market pay a higher price?
Pretesting helped the publisher try a few aggressive ideas that did not prove successful but could have opened up a huge circulation opportunity for the company. And this is the point of pretesting, notes Blum. You're supposed to take a few chances with this technique to see if you can come up with a home run. Even though the direct marketer went down swinging on a few pretests, it still scored some solid base hits on others.
Be aware that using bind-ins and blow-ins requires patience. Newsstand sales don't work the same as a direct mail campaign drop; you have to wait for the cards to trickle in and then for the full report from the publisher's agency. All told, Blum's client invested more than a year on its pretesting program.
You Can Still Use the Phone!
The telephone is more than a tool for order taking and customer service. Market research, especially to your current customers, is a great way to leverage the power of the phone. You can choose to simply gather information or conduct test sales pitches. It's what Blum calls "dirty research," since it's not likely to be statistically valid, but you can use it to keep generating ideas and checking to see if your hunches from all your pretest channels are building to anything solid.
For example, Johnson Direct conducted some pretesting via phone and Internet surveys for a B-to-B client that sells special service vehicles that cost, on average, $300,000 each.
"The pretest helped us refine our offer and influenced (changed) what publications we ran in. It also demonstrated that the decision maker was more likely to visit a microsite for more detailed information. The client was convinced the audience would call the inbound 800 number for more
details," says Johnson.
In testing the space ads, Johnson Direct focused the messaging on the client's strengths that had been identified with the market in the surveys,
reinforcing the firm's position in the prospects' minds. Johnson reports that the space ad campaign has increased the client's leads from nine per month to more than 70 per month.
And as any good pretesting and testing can do, this insight will be used by Johnson Direct to create a more effective direct mail lead generation program in 2005.
Making Sense of Your Results
As with any testing venture, you need to have a strong grasp on the quantitative and qualitative aspects of your pretest results. Make sure you understand the audience composition and any inherent biases that could exist, says Blum, especially across channels. Certainly, a self-selected audience is circumspect, because you have very little control over the make-up of such a response group.
For example, Michael Weil, senior product manager at book and newsletter publisher Boardroom Inc., told Inside Direct Mail last year that his firm is skeptical that e-mail pretests can predict successful direct mail tests. To gather additional support, Weil runs direct mail tests alongside e-mail tests to identify channel conflict.
Another pretesting blind spot that occurs in self-reported experiments is cognitive dissonance, or the difference between what people say they will do and what action they really take. Cognitive dissonance is the reason why Blum doesn't believe in using focus groups to evaluate specific package creative. However, she says, it is possible to get good directional information on how people feel about a product, their life and the product's place in their life.
Johnson agrees with this assessment. "Most pretests are not real world, and thus [are] more controlled than they should or can be. With direct mail that's tested and then rolled out, you learn from real world experience. The benefit of the pretest is to determine what big things will make an impact with the right audience."
Johnson offers the following example of a focus group's effect on a direct mail test for a Blue Cross and Blue Shield division that sells individual health insurance policies.
"We relied on traditional focus groups to determine what creative would and would not work in the direct mail channel. We mailed the creative that won in the focus groups with the creative that we thought would pull in more leads. The results proved that you need to be very cautious with pretests. The 'sure' winner in the focus groups pulled in half the leads of the 'uglier' package that the focus group said they would not respond to."