E-commerce Link: Commit to True Online Testing
In virtually all such measures, Ryan is the better candidate. If you were choosing a player for your team, you’d certainly pick Ryan.
But think on that a moment. The reason you feel confident signing Ryan stems from your familiarity with the metric and fitness function that are implicitly applied when you speak of baseball. Your decision might be quite different if you want to pick a donut quality assurance taster. Suddenly, Simpson is back in the running.
Even then, your confidence may be based on your understanding that “tastes better” is the donut metric and that Simpson is an acknowledged expert in donut consumption. But what is the fitness function? That is, what does it mean to “taste better”? Are you solely relying on Simpson’s reputation as an expert? His expertise is based on consumption quantity, so perhaps you suspect he enjoys all donuts equally. In other words, it’s quite possible you don’t have any knowledge at all of what you might call the “donut taste” fitness function.
Interestingly, marketers are asked to make more important decisions with less information and an undetermined fitness function.
More formally, the aforementioned process is known as A/B testing, and it has three steps:
1. Identify a metric. What will be contrasted?
2. Agree on a fitness function that describes that metric. How will we measure and contrast the differences?
3. Optimize by tweaking the system based on comparison of exactly two tested solutions, which differ in only one respect of how they meet the fitness function.
A typical problem for A/B testing might be: “Do more people buy when I use a red buy-it-now button or when I use a blue buy-it-now button?” The data for this test will come from your Web analytics. Once you know which candidate converts better, you use the winner and discard the other. You might then test again, comparing the winner to some other variant you have in mind.