Brian Kurtz on DM Testing in Hard Times
EB: Are big tests or small tests better during downtimes?
BK: To test something that’s too small doesn’t give you the ability for a breakthrough or at least a 20 percent or 30 percent lift, which is kind of a rule of thumb we use. If we don’t think the test can produce [that kind of] lift, it’s probably not worth doing. Whereas, in years past, if you got a 5, 10, 15 percent lift, it was pretty good. But under these very difficult circumstances in terms of high postage costs, high paper costs, etc., to plan a test based on that is a mistake.
EB: What’s too small a test?
BK: Changing the color of paper, the kind of stock, a border. You’ll see over the years that those things give you a lift, but those are the types of things I don’t have patience for anymore—those are tiny tests. I will say, however, that little things still mean a lot, such as anywhere near the order device or order card.
EB: Is it a good time to violate some of the rules, such as the rule that states you should test only one variable at a time?
BK: That single variable is still the rule of thumb; it’s what makes direct mail so fantastic as a testing medium. We’ll make a list of all these little things that we’d like to do to the package: change this to red, move this to page 4 from page 2, etc. So we’ve violated that rule a little bit over the years and created what we call the “killer package.” It’s how we look for that 30 percent lift. It [also] appeases a lot of people internally to get some things that they’d love to see in the package changed.