Strategy : Tested and Proven

As straightforward as list testing is, many marketers fall short on the interpretation, assuming a result will duplicate across all lists. Not so. It is common for a particular promotion to work well with one type of list but not another, depending on the composition of the outside file and the wants and expectations of the names on it. It's important to test your theories across all lists at various times of the year to assure the result you've produced is one that will replicate and can be trusted over time.

* Testing the Web. Homepage configuration, navigation, checkout pages - any element of the Web site that can be moved, repositioned, enhanced or versioned to influence site conversion - are candidates for testing. With the Web, the concept of testing the big things still applies, but the costs associated with those tests and the potential benefits of establishing better practices are tremendous.

Most hosting companies today have the ability to A/B split test content online in a manner that allows you to literally serve up a different version to every other customer or potential customer who comes to your site. Moreover, many of the testing capabilities allow you to establish tests for visits that come from specific activities, such as PPC programs or affiliate links. And as more customers go to the Web to place their orders, testing online and establishing controls as well as efficient practices are critical.

E-mail testing, too, is easy to do and can reap extraordinary rewards. Testing promotions, delivery time of day, delivery day of week, subject lines, total number of offers/clickthroughs, embedded navigation, rich media versus HTML versus text, content density, etc., all consist of relatively minor programming tweaks, which means, as tests, they're very easy to execute. But are they too easy?

A word of caution about testing online: When it's "so easy" to test so many different variables, it's also easy to get caught without a plan. Remember, testing starts with a plan and a question that needs answering. Just because you can test umpteen different variables in an e-mail campaign at once doesn't mean you should. Testing the elements that will produce the greatest effect first is still the best strategy, and where the Web is concerned, that means finding out what improves conversion to sale.

About Sample Sizes, Controls and Holdouts
A key aspect of testing is ensuring you've included enough records in a given sample to get a reliable (meaning it accurately measures what you were trying to measure) and valid (meaning the result is likely to duplicate over time) test. Mailing Offer A to 5,000 prospects and Offer B to 5,000 prospects to get 30 and 40 orders, respectively, won't tell you much. As a matter of fact, even though those figures equate to Offer B producing a 33 percent lift in response (.8 percent versus Offer A's .6 percent), the result is not statistically significant at these quantities.