Open Enrollment | Subscribe to Target Marketing HERE
Follow us on

Cover Story : Finding 'Definitive Insights'

SAP's Test Lab is remaking its website one A/B test at a time to lift conversions, slash waste and find what customers want

January 2013 By Thorin McGee
Get the Flash Player to see this rotator.
"It was 24 months ago that Facebook was the social darling and Google was search marketing, right? Google was the quintessential paid media and Facebook was the quintessential earned media," says Shawn Burns, worldwide head of digital marketing for SAP. "Now, Facebook's all paid and Google's social. How do you make media investment decisions in that environment if you're not doing high-end analytics to contribute to attribution modeling?"

For Burns, the realization that it was impossible to understand what was happening to the marketing spend of this digital, data-driven software company was a problem that required a new approach to making marketing decisions about His team now takes its cues from the clients and prospects it's trying to reach.

Finding reliable, real-time insights into what they want is where SAP's dedicated Test Lab comes in. Created and run with help from the marketing intelligence arm of Richardson, Texas-based KBM Group, Test Lab uses real-time A/B testing to optimize conversions on and find incremental lift in all of SAP's demand generation and nurturing programs. The key performance indicator for SAP's Test Lab director Jake Sapirstein is "definitive insights," which Burns describes as results so solid and repeatable the company never has to test them again.

So far, SAP CMO Jonathan Becher credits the Test Lab with an average of "27 percent lift in incremental sales leads from digital" and "digital marketing budget savings of 20 percent" by avoiding less effective online techniques.

"We've moved away from instinct and now depend on quantifiable insights," says Becher, and "the impact is tremendous."

S for Social
According to its fact sheet, SAP is the largest global provider of enterprise software and services by revenue, with revenues of more than $18.7 billion. Headquartered in Waldorf, Germany, it has offices in 130 countries and 197,000 customers worldwide. Becher joined the company as CMO in August 2011 and spearheaded a shift toward becoming a "social enterprise."

"We've made big bet on social at SAP," says Becher. "We infuse social in everything we do, from how we decide what solutions to build, to how we market/sell them, to how we support customers after sale. ... I'd go so far as saying the 'S' in SAP now stands for social."

That has tremendous impact on Burns and the digital marketing team, which is responsible for the corporate website—visited by just about every customer or prospect SAP has—and all demand generation activities.

"We've made this massive change through new marketing leadership, which has put a very high profile on digital overall," says Burns. "Everything you're trying to do is open instead of closed. So instead of very traditional one-way messages and locked-down heavy marketing spin," SAP has embraced "open, transparent, normal human being language."

That meant a complete rebuild of Not just the platform and HTML, according to Burns, but the content and messaging of the entire site.

If you've been to an airport, you know SAP invests in space advertising through things like airport signage and highly visible sponsorships—the Tribeca Film Festival; the SAP Open tennis tournament; and the SAP Arena in Mannheim, Germany, to name a few. For anyone who engages with those campaigns, the next step would be to visit a page on The same is true for anyone who's inspired to visit SAP through word of mouth, reputation, a case study or other campaigns.

All of those prospects end up in Burns' marketing pipeline. SAP expects that 100 percent of its leads and existing customers will pass through it.

"A corporate website serves the needs of many masters," explains Burns. "Whether you're a customer looking for a solution, an interested analyst looking for an investor relations report, we serve all those audiences." However, he continues, visitors to "are generally new to [at least] a portion of our solutions ... So we gear it toward the prospect experience ... [with an] awareness and demand-generation approach."

Any website rebuild is a challenge, but for SAP the biggest change was that "for the first time, we built a website with our customers," says Burns. "As opposed to what we thought they should be interested in, we were building it with what we absolutely know is the information they're looking for."

This testing pays off in sections like the master navigation of the site, which has nine elements: solutions, industries, lines of business, services and support, customer testimonials, training and education, about SAP, partners and events. Because of the site testing, Burns says with confidence that those options cover what visitors are looking for about 95 percent of the time.

The text on the site also was completely retooled to focus on visitors from search.

"Customers today are primarily searching. When they're out there looking to get educated, they're just going to Google, they're going to Bing," explains Burns. "If you're speaking in a heavy-handed, branded marketing tone of voice, you don't show up in those searches. ... The way my customers speak, the way they want to be engaged, I have to play that back to them."

Slavi Samardzija, vice president of marketing intelligence at KBM Group, has worked closely with Burns on Test Lab and recalls one example of this from testing paid search keywords for the SAP Business ByDesign package: Visitors were coming to the landing page through a set of about 100 paid keywords. But, as Samardzija and the rest of the team looked at those keywords, they noticed themes—some were about accountability, others showed the searcher was seeking flexibility, etc.

"We grouped the keywords into different themes and we changed the tagline on the landing page to align it with incoming keywords," says Samardzija. So a search looking for keywords concerned with flexibility landed on a page with a tagline like, "SAP offers optimal flexibility." Accountability keywords landed on a page that essentially said "SAP offers optimum accountability."

Samardzija says aligning those taglines with the search terms more than doubled conversions on those pages.

Beyond tone and word choice, SAP shifted the message it was trying to convey through the website, as well. Visitors "don't need to know that we have millions of users," says Burns. Instead in December, for example, the four tabs on SAP's homepage offered session replays from the SAPPHIRE NOW + SAP TechEd user conferences, a blog post about online communities, tips for using SAP products, and a free trial offer—all content that's aimed at what visitors most likely want to know.

Conversion is the website performance metric that most aligns with success for SAP. "We have looked at time on site, we have looked at a basket of engagement metrics, and the way the world's working right now, a lot of what used to be relevant historically has evaporated," says Burns. "All we worry about is that they take the next step and they accomplish their tasks."

The conversion in question could be anything users come to the site to accomplish. "That could be a developer blogging about something in the community. That cold be a prospect simply asking us to have a chat. That could be an investor asking us for the latest copy of the annual report," says Burns. "Whatever that next step is differs by audience, but the goal of the site has got to be to convert them to that next step."

Figuring out the best way to do that—to optimize the homepage so visitors can get to the things they're looking for, to design landing pages so visitors follow through on them, to phrase offers, etc.—is why Test Lab was created.

A for Analytics
Finding the data to quantify that was a challenge using just website analytics. "You're about as likely to make the news by reading the newspaper as you are to make change by reading a Web analytics report," says Burns. Alone, that data is "not actionable insight."

SAP brought in Adobe's Test&Target (previously Omniture Test&Target) two years ago, an A/B testing suite for optimizing website content and landing pages. The ability to A/B test the website "changed everything" about how SAP approached its website and digital marketing, according to Burns. "The moment you're able to render two different versions of your website simultaneously and just let your customers select with their mouses what they prefer, you're literally enacting change."

Going back to those nine items in SAP's main navigation that Burns "knows" 95 percent of visitors are looking for, he attributes that to testing.

"We got 25 ideas [from customer focus groups] about what needs to be on the homepage. It's way too much, it's not going to meet the [standard] for simplicity. So how do we take it to the next level? ... Mock it up. Let's actually put all 25 on [the navigation], randomized. Let's roll it out to a series of customers. So now, instead of talking to focus groups 20 and 30 people at a time, you're doing thousands of tests in a live environment. And all of a sudden nine magic things rise to the surface, and quantitatively you're ready to go."

Even with all of the testing capabilities those tools enabled, Burns says, "what we realized in our first year with Test&Target was all we had was a platform. And a platform, when you ask an entire company to just spend 10 percent of their time [on it] ... it's nobody's job, it just doesn't happen."

In fact, it wasn't being used at all. Burns could see from the log-in statistics that SAP's trained users just weren't logging in and using the tool. All he had done during that first year with Test&Target, says Burns, was "put one leg of the stool in place, which is the platform. What about the people and what about the process?"

"Once you get into something like Test&Target that requires somebody to continually use the software and develop expertise," explains Samardzija, "enterprises find themselves in this environment where they've spent a significant amount of money on technology and then trained and are trying to create these power users, but the technology's really not being utilized. And if it's being utilized, you get these kind of entrepreneurial pockets and folks that are able to pick that up, but they're doing tests for the sake of testing that don't really necessarily line up globally with some overall vision or marketing objectives"

"I started with a platform," Burns recalls from the experience. Now he had to "get people and get a process so that the people and the platform are used to derive the most value for the company."

P for People ...
The Test Lab was created as a way for SAP to put the people and the process behind Test&Target to make the overall testing program a stable, three-legged stool. It's one department headed up by Sapirstein with support from KBM Group.

The lab takes testing proposals from throughout the company, prioritizes and vets them, and runs approximately 100 statistically rigorous tests a year across SAP's multinational markets to derive definitive insights that are then evangelized throughout the company.

"'Definitive' is a big word for us because it means we won't test it again," says Burns. "If everybody [in the marketing department] is doing their own tests, what I can guarantee you is in the next 24 months, we will test the same button 26 different times and nobody will leverage what the other guy did." With Test Lab hunting for definitive insights, it can say to those users, "We did that six months ago, we already know this is how you have to lead that approach."

As mentioned, digital marketing's focus is on conversions, and that's primarily what the Test Lab examines. The lab runs A/B tests on any part of SAP's digital presence to determine how to optimize that page for conversions.

"Simply put—testing improves results exponentially," says Becher. "For example, most people think that design and layout is a matter of talent and taste," but SAP has found A/B testing design and layout choices can increase conversion by as much as 97 percent. Doing the same with content and offers has led to as much as 780 percent lift. In either case, Becher says, "We've moved away from instinct and now depend on quantifiable insights—the impact is tremendous."

"I had to have somebody who it's not 10 percent of their job, it is their job," in order to make the most of Test&Target, says Burns. A little more than a year ago, "we recruited some new talent into SAP who were doing this real whiz-bang agency-side [work] leading test labs for other companies ... so that second leg of that stool was bolted into place."

While Sapirstein is the main person inside the company, Samardzija and KBM Group helped Burns conceive the Test Lab and sell it to internal stakeholders, and they continue to support the lab with statistics and analytical muscle.

"KBM Group's first project around Test Lab was really a change management piece where we did this road map and executive interviews and stakeholder interviews," says Samardzija. "Working with Shawn's team, we gained buy-in from the broader SAP organization. ... There's so many different entities that are all working together that it was critical for us to get the buy-in for the vision that Shawn and his team had."

That kind of buy-in and executive support was key to not only getting the program off the ground, but putting it in position to succeed.

"You'll hear this in discussions in the testing world frequently," says Sapirstein: "Vital to the success of a testing program is having that executive sponsor who really gets a data-driven culture and the importance of it. So that's the key, to have someone like Shawn [Burns] who's not only supportive of it, but clearly communicates the value and the benefits to others."

"They're a large enterprise and you have the complexities of a large enterprise," explains Samardzija. "We are working with Shawn's team, and I think they're quite entrepreneurial and quite eager to test new ideas and to try new things. And that's why something like Test Lab, which came from a conversation we had, is now reality."

... And P for Process
Once Test Lab began generating results, SAP had no shortage of testers asking for it to analyze aspects of their marketing.

"We're a 65,000-person operation," says Burns. "It doesn't matter how many people you put in the Test Lab—and the talent we brought in was one dedicated Test Lab director [Sapirstein] who's working with agency partners to make this happen—he becomes that one lone soldier bobbing in the ocean. Sort of like that poor guy who raises his hand and volunteers, and 65,000 people tackle him."

SAP needed a process to make sure the testing resources were put to the best possible use toward its business goals, and that's a place where Samardzija and KBM Group were able to help with what Burns call a "high-level road map."

That road map outlines four steps that have been essential to running the Test Lab:

1. Submission: Users from throughout SAP offer submissions for what should be tested.

2. Vetting: The Test Lab determines which tests make sense to run based on how well they align with SAP's business goals and feasibility.

3. Completion: Test Lab carries out the test and determines when there's been enough response to declare a winning option with confidence. SAP's rule is to requires about 1,000 conversions per test and an overall confidence level of 95 percent.

4. Definitive Insights: Determine whether or not the test has yielded the kind of result that can be applied to all of SAP's markets. These are shared with the entire marketing department through quarterly publications.

"What we look for [in good tests] are areas that are close to our key conversion points," says Sapirstein. "For a lead generation site like, those are typically registration form pages or content areas that lead to those forms, so those are high-impact areas that we'll focus on."

Scalability is another key requirement for SAP's testing. "We like to only test things that are easily applicable in a large-scale way," says Sapirstein. "We have several dozen country websites across the globe, so we will not do any test unless we decide the insights we'd gain from it are easily syndicatable across the globe."

For the SAP digital marketing team, the Test Lab has made life simpler and perhaps even more enlightened.

"All the shades-of-blue subjective discussions you can get into about 'We really like that version versus that version, it just feels better,'" are eliminated by A/B testing, says Burns. "Gut feel is gone. At this moment in time with real-time data, you're just [deciding based on], 'Our customers have voted with their mouths, and they're saying A versus B, and now we're going to deploy that worldwide.'"

Becher sees the same benefits. "The testing lab has taken the guesswork out of our digital marketing. It's a perfect example of how the science of marketing informs the art and results in greater efficiency and impact."


Companies Mentioned:


Click here to leave a comment...
Comment *
Most Recent Comments: