Do Your Tests Pass Inspection?
Not if you’re guilty of any of these common testing mistakes, according to Marketing Managers Tammy Nelson and Christine Andrews. In a session at the DMA’s annual conference in October, Nelson and Andrews drew from their own experiences with American Express Property Casualty companies to expound on some dangerous testing pitfalls—and how to avoid them.
• A lack of formal preplanning and number crunching. Testing needs to be more than throwing some ideas into the mailstream to see what sticks. Achieve statistically sound, measurable results by determining what variables you are testing for, what you hope to learn, and what ROI will indicate a successful test—before you begin.
• Too many/not enough variables in your test package. While you need to have a control for every variable you test so you can be certain which factors resulted in a positive change, testing only one variable at a time can hold your mailing plan hostage. Instead, build a full factorial test structure that allows you to test each change, while controlling all variables.
• Only trying little tweaks. Every change should be tested, but small changes will only ever yield small gains. Set aside a portion of your testing budget for more significant tests in which you try new concepts against your control.
• Testing in a vacuum. Consider external factors that affect response, such as seasons and current events.
• Abbreviated evaluations. Evaluating packages based solely on response will give you only a piece of the picture; measure results based on other factors as well, such as sales, cost per sale and customer lifetime value, for a better understanding of the long-term implications of the package.