Never too Small to Test
June 2006
Just because your database isn’t in the millions and your budget isn’t the size of Dell’s, that doesn’t mean you can’t test. But if your housefile is only 25,000 names and your usual mailings are no more than 35,000 pieces, how do you get statistically reliable data from your tests? The answer lies in repetition.
Whatever testing rules you are following, the statistical hurdle of 50 or 100 orders can be daunting for a small business. Overcome that problem by testing the same concept repeatedly. For example, if you are curious about whether installment payments will benefit your product but are unsure how much of a lift you’ll get, test with and without installment offers in multiple mailings, inserts, ads, Web pages, etc. Then look back and gather your data to determine trends to make your decision. If four of the five times you test installment payments they lift response by 10 percent or more, and the fifth time it’s a wash, your decision is a no-brainer—roll out installment payments wherever you can. If on the other hand, installments win three out of five tests by 5 percent or more, and you lose 5 percent or more on two of the tests, you need to dig deeper and see if there are further tests you can try to understand when, for which audiences, and for which products and offers, installment payments make sense.
—Shari Altman
Whatever testing rules you are following, the statistical hurdle of 50 or 100 orders can be daunting for a small business. Overcome that problem by testing the same concept repeatedly. For example, if you are curious about whether installment payments will benefit your product but are unsure how much of a lift you’ll get, test with and without installment offers in multiple mailings, inserts, ads, Web pages, etc. Then look back and gather your data to determine trends to make your decision. If four of the five times you test installment payments they lift response by 10 percent or more, and the fifth time it’s a wash, your decision is a no-brainer—roll out installment payments wherever you can. If on the other hand, installments win three out of five tests by 5 percent or more, and you lose 5 percent or more on two of the tests, you need to dig deeper and see if there are further tests you can try to understand when, for which audiences, and for which products and offers, installment payments make sense.
—Shari Altman




Cracking the QR Code
The Art & Science of Multichannel Fundraising
Hitting the Email Inbox
The Ultimate Guide to Email Marketing
Social Media Success