3 Unexpected Wins That Emerged From A/B Testing
Version A of Anthem's mobile landing page for its PPC campaign has plenty of calls to action, but no images.
Version B of Anthem's mobile landing page included an image, longer copy and a single, clear call to action.
Duolingo completed an A/B test to improve its Spanish-to-English language courses, increasing the time spent using the app by each learner.
Working longer hours. Spending more on marketing. Offering the best quality products to consumers. Keeping customer service as priority number one. These are but a handful of the innumerable steps every entrepreneur takes to grow their business and take it to the next level. With the world moving online, a key contributor to the success of a business today is the quality of its web presence.
With the plethora of web design agencies dotting the landscape combined with DIY website platforms, churning out a "good" website is a given. Why, even if you're selling something as traditional as apparel, you could go digital with a few WordPress-style edits to a readymade theme using an e-commerce enabler such as Spaces.
Users are used to "good" by now. They've come to expect it, even. Wowing users, making them marvel at the ease of using your site are the next steps in an increasingly competitive and sophisticated playing field. And how do you achieve that level of smoothness? A/B testing of course.
By choosing specific website features and testing out various ways users interact with these features; website owners can refine the user experience into a finely honed, data driven masterpiece. The most critical thing you need to perform a successful A/B test is an open mind. Leave those preconceived notions behind and you'll discover tiny nuances about your website that you never even imagined existed. Here are three live cases of A/B tests with a twist. Who knows, your next test may just find inspiration right here!
Anthem — Do Images Make a Difference to Conversions?
It's a familiar piece of advice that we see floating around marketing blogs all over the internet. "Use relevant images" or "Avoid boring text or stock images" or even "The right images can improve conversions manifold." While I'm not going to debate the veracity of each of these statements, the fact remains that sometimes A/B tests go against deeply held beliefs, however popular and hip they may be.
WhichTestWon recently wrote about health insurer Anthem and its surprising experience with mobile landing pages. Anthem created two mobile landing pages for a PPC campaign — Version A without any images and Version B with an image of a smiling woman.
The page with the image had a single, gentle call to action and more descriptive copy. The text heavy version on the other hand had three different calls to action, no images to soften the message and very little descriptive copy. The expectation was that the version with the image would perform better.
However, contrary to expectations, the all-text version outperformed the version with the image hands down. Leads increased by 166% at a 95% confidence level. The platform in question made all the difference. On a mobile device, users found the multiple action choices in Version A more convenient than the single call to action in Version B. All that copy in Version B that was meant to convince users was wasted on the small screens of mobile devices. Calling customer care as suggested in Version A, was probably easier to do on a mobile phone than read reams of tiny text for clarifications about Anthem's health plans.
Obama's Presidential Campaign — Million Dollar Email Subject Lines
Email is often considered an irritant, distracting employees from their daily to-do lists with its endless back and forth. While the option of optimizing email for task scheduling and collaboration does exist, the platform suffers a terrible reputation thanks to piles of one-line conversations, irretrievably of relevant data from threads, conversations diverting totally from their initial subject lines, and spammy marketers clogging inboxes across the board.
With this rather humble background, email marketing stood out as an unlikely, yet, key contributor to President Obama's re-election bid in 2012. Toby Fallsgraff, the email marketing director of the Obama campaign, explained in an interesting video how A/B testing resulted in millions of dollars in donations purely via email.
The campaign managers tested multiple drafts of the same email to single out the best performing copy. Subject lines were tested extensively to zero in on the money shot.
While formal sounding subject lines showed average results, surprisingly, a subject line that contained mild profanity — such as "Hell yeah, I Like Obamacare" — was a distinct winner. Subject lines with just the first name of the recipient and nothing else, did well also. However, the home run among the subject line tests was hit by the famous "Hey" series of emails. The casual tone of the subject line combined with the President's name in the Sender column, gave this version a power-packed punch that helped it beat all others.
In total, Obama's email marketing campaign alone netted him $690 million dollars in campaign funding. Not shabby at all.
Duolingo — Learning Languages by Trial and Error
Duolingo is probably one of the most popular language learning apps available today. Besides offering free language lessons, one of the key attractions of the app is its extensive gamification of the process of teaching new languages.
Most apps and games embrace A/B testing to improve their features and usability, with the eventual aim of making more money off of their users. This is where Duolingo differs from other apps and websites. Unlike most other brands, Duolingo does not use A/B testing to sell more language lessons or to upgrade users to paid plans etc.
Instead, the company focuses its A/B testing capabilities on two primary goals that raise user experience to the next level. These are
- Increase time spent by each learner on the app
- Improve and accelerate language learning outcomes for learners
Luis Von Ahn, founder of Duolingo calls it "teaching languages based on proven learning patterns."
A/B testing has offered Duolingo insights in language learning that it may never have known otherwise. For example, the Spanish language does not have a gender neutral pronoun "it." So when native Spanish speakers learn languages like English which use the word "it" extensively, early learners stumbled and eventually quit the course in frustration.
A/B tests revealed that moving lessons involving the word "it" to a more advanced level in the Spanish-to-English language courses made sure that native Spanish speakers encountered this obstacle at a much later stage of their English learning journey, and were more confident in navigating around it, thanks to a stronger foundation.
Duolingo redesigned the English course for Spanish speakers on a permanent basis, leading to far better learning outcomes overall.
William Edwards Deming, renowned statistician and author, famously quipped "In God we trust; everyone else must bring data." When it comes to your website and maximizing its performance, you'd do well to follow Mr. Deming's advice. After all, results matter more than how "cool" your site is perceived to be.
Rohan Ayyar is a project manager at E2M, a digital marketing agency offering creative content strategy, analytics and conversion optimization services for startups. Rohan is an avid blogger, with posts featured on MarketingProfs, Social Media Today and Entrepreneur, among other places. Follow him on Twitter at @searchrook, and reach him via email at firstname.lastname@example.org.