DH to Langsdorf:
Sorry to belabor this thing, but if the winning cell has 60 and the other cell has 59, total orders are 119. The square root of 119 is what? I cannot do square root any more, but it seems to me that the equation changes. Or am I losing my mind? Cheers.
Langsdorf to DH:
Only people with calculators can do square roots. Sqrt 119 is 10.9. Rule you quoted says: “For a test cell to be a clear winner, the number of responses must be more than double the square root of total orders.” —Axel Andersson
Double 10.9 is 22. 60 is more than 22. Therefore, by the rule as written, the test is “a clear winner.”
But it isn’t. My guess is the rule should be: “For a test cell to be a clear winner, the increase in the number of responses must be more than double the square root of total orders.” The results would have to be 71 to 48. That would be a clear winner.
In the 48 to 71 example, the Z test (assuming sample size of 2000 for each test) would be 2.14, which gives a one tail confidence level of 96.8%. That is a clear winner.
And I think either of these rules assumes that the mailing size for both samples was the same. If you mailed 5,000 to get 71 and 1,000 to get 48, the 48 is the winner.
In short, the rule is too short.
A potential subject for a newsletter article is how people make decisions when the difference is not that dramatic. At 71 to 48, I wouldn’t ordinarily even be doing the math. I’d probably be spending time to be sure there wasn’t an error in the data.
To go back to your article, if Zales had done the test you proposed, I seriously doubt that the difference in sales would have been 71 to 48, or an almost 50% increase. In fact, in 2005, sales were up slightly, but not what they expected—and had led Wall Street and the board to expect. From their press release at the time: