10 A/B Test Preparation Tips
Minutes of testing could save marketers’ hides for months, if not longer. Testing is touted but sounds difficult to many, so marketers may not do it as often as would be ideal. However, having a firm foundation may make marketers feel more comfortable about taking the plunge.
Lisa Qian instructs viewers of the O’Reilly Media video to take the following 10 steps to prepare for A/B testing:
- Define the Goal and Form a Hypothesis. What do marketers want to get out of the test? What answers do they think they will find?
- ID a Control. This is where the A/B comes in, as the test sample will run against this control. Is it a call-to-action button? A home page?
- Pick KPIs. Will the test cover revenue? Site traffic? Engagement? What aspect? What will indicate success?
- Choose What Data to Collect.
- Ensure the Data Collection Works. Qian says logging the data is essential.
- What Size Should the Measurements Be? Will the test drill down to 1,000 emails or 5,000? This gets into what error rate is tolerable.
- Pick a Sample Size. While Qian doesn’t tell marketers what size the samples should be, Simms Jenkins told me that if marketers have 1 million people in a database, they’ll want that sample size to be “probably 10,000 to 20,000.”
- Run a Test Before the Test. This helps marketers decide how much data to collect and how long to run the full test.
- Stick to the Original Test Length. Don’t stop the test early simply because results come in, “because that could happen just by chance,” Qian says.
- Before Deploying the Test, Run a Dummy Test. Qian calls this an A/A test, or control vs. control, for first-time testers. That helps those marketers find problems in the system before running the real test.
What are some other A/B testing tips marketers should use?
Please respond in the comments section below.