Testing, The Dirty Dozen (1,848 words)
The company faced two problems: 1) no one would ever know whether the results could have been improved, and 2) if the control went suddenly from moderate decline to rapid death—not an uncommon occurrence—the company would be at a major disadvantage with no alternative at the ready.
Some experts recommend you allocate roughly 10 percent of your direct marketing budget to testing. Don't expect miracles, though. Over the long haul, most tests will not beat the control. The key is to aggressively test your best element ideas, especially when you are going against a well-established control.
It's also important to have a mix of testing strategies. Element tests— color, pricing, copy, etc.—usually do not produce the radically different results that breakthrough tests, such as an entirely new acquisitions direct mail piece or a completely fresh telemarketing script, will. But, because element tests typically cost less, you can run them more often.
Mistake #3: Non-clinical Testing
The transgressions here are numerous. Not taking random samples, testing too many elements, testing too large or small a quantity, and not seeding lists are all examples of abandoning the discipline of a true test.
I once encountered a large mortgage company that prided itself on having arrived at a creative control through multiple testing. This was a top-five lender with ample resources. Through a series of questions, I found that in determining how to set up their test-vs.-control cells, they simply "split the country in half!" People living on the eastern half of the country received the control and those on the western half got the new creative. Ouch! This set up all sorts of behavioral, demographic and lifestyle biases that basically invalidate the testing. Sad but true.
Mistake #4: No Control
With an element test, you'll know the impact of the element you've tested. With a breakthrough test, you will not know what specific element(s) drove the difference in results. Nor do you care—you're testing entire packages.