What Are Some Ways to Test Lists So You Can Be More Confident
The 2005 Lists Web Community Marketer's Idea Exchange.
"The obvious answer is to make sure you're testing enough to have statistically reliable responses and make sure each list has a key code (response mechanism) in order to read and evaluate results. Research as much as possible—the better educated you are, the easier it will be to determine which lists to include in your test matrix. Try to test in your 'good' season, although, out-of-season tests can be adjusted. Try for vertical market tests first time in the mail."
—Cheryl Bagdan, senior account executive, exhibit & promotions director, Leon Henry Inc.
"Use a control segment as your benchmark for performance. You need to be able to measure success. When testing two different lists, make sure all elements of the mailing are identical so as not to skew results. This includes quantity, selects, mail piece, response vehicle, call to action and mail date. It should be an 'apples-to-apples' test. Use A/B splits and random nth selects to properly disperse names within file segments. You will ensure that all geographic areas and other elements of the file are weighted equally when evaluating results.
Use statistical models to narrow down your segments to only those with the highest probability to respond. You'll be confident that you're mailing the best possible names from that file, whether it works or not.
From a broker perspective—you need to have the knowledge and understanding of your client, the specific market, the product/offer, and their competition to make the most valuable recommendations on testing lists.
Selections are key. If a test doesn't perform to expectations, a retest, using different selections can sometimes make all the difference."
—List Management and List Brokerage Sales Groups, List Services Corp.
"As a general rule, when testing use the most targeted segment you can for the initial test and broaden out from there on subsequent retests. I'd recommend testing a hotline, definitely paid, and absolutely look for direct
mail sold. If the file isn't new to the market, always look for relevant continuation usage to support your rationale.
I've based these statements on the fact that the offer is a direct mail-sold catalog or subscription. If this is not the case, then rules/segments may change slightly dependent on your individual offer. Fundraisers may want to segment out $5+ only donors, for example.
A competent broker should lay out all his reasoning for testing for you and be able to back it up with usage, counts, etc."
—Walter Perkowski, vice president, RMI Direct Marketing
"When testing a list, make sure you know where your list broker is getting its lists from! It should always be a reputable company that has relationships with the professionals on its lists. A controlled-circulation list, that offers rich demographic information will also allow you to know who you are targeting and thus you will get a more accurate reading on the evaluation."
—John Gennaro, account manager, DM2
"To be more confident in test results, we always suggest a testing plan that gives you feedback on different segments, frequency, timing and creatives. Do not rely on a one-shot effort to gauge your overall results."
—Kym Vance, vice president, marketing and sales, Datagence
"When using a large direct response file, be sure to tighten the select criteria as much as you can. Rather than selecting recency and donates to "type" of cause, add gender, age 55-plus and any other selects that may be available that make sense. For the test, you want to see that the list performs, and you can always go back and retest with looser select criteria to increase the available universe. If the broader select criteria do not perform—you either will have to put more money towards re-testing or not retest and possibly pass over a potentially winning file."
—Jennifer Honadel, account director, not-for-profit sector, ICOM
"Learn all you can about a list you want to test. Ask for mailing pieces or samples of recent ads that generated the list's names. Find out where the ads appeared, so you get the "flavor" of the list."
—C. Rose Harper, first woman to serve as chairperson of the DMA,
DMA Hall of Fame inductee, former president of The Kleid Co.
"List owners frequently load tests with hotline or multibuyers. The result is a falsely high response on the test, which brings you running back for more names — only to suffer death and destruction on the rollout. What can you do? Forecast lower results on your rollout than your test. Some direct marketers say never step up more than five times (e.g., if you test 5,000 names, don't go back for more than 25,000). I disagree. Know your lists, know your broker, know your list owner and be guided by your experience."
—Paul Goldberg, direct marketing consultant
"Always order an odd number of test names. Most list rentals require a 5,000 or 10,000 minimum. A lot of people will want more than 5,000 names as their test quantity. Order 5,200 or 5,500, and change the number each time you do it. Here's why: There are some list managers and owners who don't run a 5,000-test panel for every order of that type that comes in. They take their subscriber file of 200,000 and, once a quarter, they'll knock out 5,000 names to be used to fulfill tests. So everyone who orders tests in that three-month period gets sent the same 5,000-name list!
You don't want to be part of that because it's being mailed much more than everything else is. You want your own list.
An even worse potential problem with orders of 5,000 can occur when you order something other than standard active subscribers. Let's say you want only people who have renewed or people with a particular job title. If you order a round, even number, they may still give you some names in the plain old select. If they don't do that, they might confuse your order and give you someone else's 5,000-name selection. But if you order an odd number of names, it's much more likely you'll get what you asked for."
—Don Nicholas, president, Digital Media Advisors