Cover Story: Finding 'Definitive Insights'
3. Completion: Test Lab carries out the test and determines when there's been enough response to declare a winning option with confidence. SAP's rule is to requires about 1,000 conversions per test and an overall confidence level of 95 percent.
4. Definitive Insights: Determine whether or not the test has yielded the kind of result that can be applied to all of SAP's markets. These are shared with the entire marketing department through quarterly publications.
"What we look for [in good tests] are areas that are close to our key conversion points," says Sapirstein. "For a lead generation site like SAP.com, those are typically registration form pages or content areas that lead to those forms, so those are high-impact areas that we'll focus on."
Scalability is another key requirement for SAP's testing. "We like to only test things that are easily applicable in a large-scale way," says Sapirstein. "We have several dozen country websites across the globe, so we will not do any test unless we decide the insights we'd gain from it are easily syndicatable across the globe."
For the SAP digital marketing team, the Test Lab has made life simpler and perhaps even more enlightened.
"All the shades-of-blue subjective discussions you can get into about 'We really like that version versus that version, it just feels better,'" are eliminated by A/B testing, says Burns. "Gut feel is gone. At this moment in time with real-time data, you're just [deciding based on], 'Our customers have voted with their mouths, and they're saying A versus B, and now we're going to deploy that worldwide.'"
Becher sees the same benefits. "The testing lab has taken the guesswork out of our digital marketing. It's a perfect example of how the science of marketing informs the art and results in greater efficiency and impact."