List Testing - Parameters & Strategies
Schwartz's comment harkens back to the late Dick Benson's rule of thumb on testing lists: Look for 100 orders to feel that a result is statistically valid. Those 100 orders might come from 2-percent response to a 5,000-name test or a 1-percent response on 10,000 names.
On very high-end products, you might allow a bit more leeway. "On a $2,500 price-point item, you cannot expect to pull a 2-percent response," Schwartz notes. "In that case, 60 replies may be all you need to continue."
Instead of thinking of testing a list of 5,000 or 10,000 names, think of testing cells. Kroll, who also heads Media Management Group and Compiled Solutions, all in Plainsboro, NJ, says, "You need two or three or five test cells to get valid results."
The concept of a grid, or matrix, test is that mailers test several lists, creative approaches and offers at once. The benefit, in addition to the head-to-head comparisons, is that enough names will be used from a given list to meet the minimums—but they'll be split into various test cells.
"Say you have 10 lists to test, two or three creative approaches and two or three offers. You split those tests into cells. List A with 6,000 names could be split into three test cells of 2,000 each," Schwartz says, noting, "this is the way to test effectively."
The key to running a successful grid test is tracking. "The mailer must understand, use and track the key codes to follow the results," Schwartz explains.
Beware that no mailer gets 100-percent capture rate on key codes. Even when you instruct the telemarketing staff to ask for the codes, customers may not be able to find the code on the mailing piece, or perhaps they borrowed the catalog from a friend. Of course, a higher percentage key code capture rate means a greater level of confidence in the test results.