Six Little Secrets of Test Design and Analysis
Secret 3: Only make one change at a time to your test panels. If you make multiple changes to your test packages, you will not be getting the most out of your testing program.
For example, if your test package loses against your control, it may very well be a result of only one of the changes you made to the package and not all. You will never know which changes were working and which were not. As a result, you may overlook potentially winning elements. This simply is not smart testing.
Secret 4: No direct marketer should ever consider evaluating its test results with a confidence level lower than 85 percent to 90 percent. To do so assumes way to much risk. And, fishing for a confidence level that yields a significant result should never be practiced.
The proper rules that any good direct marketer should follow regarding significance are as follows:
Begin by assessing your test with 95 percent confidence. If it’s significant at 95 percent, see if it also is significant at 99 percent. If it also is significant at 99 percent, you definitely have a winner, and rollout should be a no brainer. If it’s not significant at 99 percent, but is at 95 percent, a partial to full rollout at a minimum should be considered. If it’s not significant at 95 percent, take a look and see if it is significant at 90 percent. If it is significant at this lower level, a retest or partial rollout certainly is in the cards. If it is not significant at this lower level, you definitely have a loser and should not consider the test any further.
Secret 5: A full factorial test design usually is never warranted. The only time it is warranted is if you truly believe all the elements you are testing will interact with one another. Most elements simply do not interact with one another.