Q&A: Match.com's Jim McDonald on How Multivariate Testing and Landing Optimization Work Together
Many savvy online marketers understand that multivariate testing can affect the success of any online acquisition campaign — and simple changes to landing pages can increase online sales.
Last week, eM+C discussed this trend with Jim McDonald, senior online marketing manager of search and CRM at the Dallas-based online dating service Match.com. McDonald discussed how his company has created a harmonious relationship between its paid search and affiliate marketing programs.
eM+C: Match.com uses multivariate testing in conjunction with its landing page optimization programs. Why?
Jim McDonald: Match.com is a very direct response-driven business. We do a good job at marketing our service and getting potential customers to visit our "store." We want to acquire customers as efficiently as possible. Like a smart merchant, we wouldn’t want to spend money to get people into our store and then lose those opportunities once they arrive because of a suboptimal landing page experience.
While A/B testing is part of our optimization strategy, we often find it laborious and time consuming when trying to iterate on nuanced changes. Multivariate testing provides the insight, flexibility and speed we need to maximize landing page efficiency.
eM+C: How is multivariate testing different from A/B testing?
JM: My answer will sound somewhat geeky, so bear with me. A/B testing technically tests different values for one variable. Usually this one variable is an entire web page, such as a homepage or registration page, where you would test one design against another, which are the values. Multivariate testing, as the name might suggest, tests multiple values for multiple variables.
Think of the variables as components or puzzle pieces on a page. You may have four variables, for example, such as an image (variable 1), headline (variable 2), body copy (variable 3) and “submit form” button (variable 4). You may also have three values of each variable that you want to test. Values are the different iterations of each variable — think of values as the different colors of a particular piece of a puzzle.
For example, value A for variable 1 (image) might be a blonde woman, value B a brunette and value C a couple. You'd have more than one value for each of the other variables, too. All of these components are automatically “mixed up” in a statistical stew to create a new page that's served to a particular visitor to that page. One visitor may see a page with a blonde woman, while another visitor may see the same page but with a couple on it instead.
Although there are other benefits, the true beauty of multivariate testing, in my opinion, is twofold. First, you have tremendous speed in testing. In the example above, if you wanted to isolate a particular value and variable in controlled A/B tests to ensure you knew which page combination performed best, you'd have 81 permutations — three (values) to the power of four (variables). Instead of running all those tests, you can run just one multivariate test and get the same insight. Additionally, you can often reach statistical significance, where you know you can trust the results within a certain degree of confidence, much quicker than with standard A/B tests.
The second beauty of multivariate testing is direction. A properly executed multivariate test will not simply tell you which combination did better, but will provide you with clues as to which button or image or headline contributed to the better performance. That’s huge because you can now easily iterate in the right direction. It’s almost like having a cheat sheet in college.
eM+C: Is it more difficult to implement than A/B testing?
JM: Simply put, no. The hardest part is determining which components will be tested. There's software available that does all the work for you and provides reporting that even a marketer like me can understand. Google Website Optimizer is a free version offered by Google. I use a more robust service provided by Optimost.
eM+C: Can you offer an example of how the technique has helped drive results?
JM: We actually use both multivariate and A/B testing in tandem. Here’s how it works: We have one stream of testing we call the “Lewis and Clark” stream. This is where we constantly push the boundaries of our landing page design using A/B testing. We'll test landing page designs that originated in different industries or verticals, for example. We try to look for the next big thing. This stream adheres to the old adage, “If you want to climb the highest mountain, you don’t start off in the Rockies.”
Our other stream is called the “Thomas Edison” stream. This is where we utilize multivariate testing. We take our best-performing landing page and constantly iterate on its design, layout, components, placement, color, etc., to eke out performance gains. If we get a new “champion” from the Lewis and Clark stream, we place it as the control in this stream and start the improvements.
eM+C: Any best practices around multivariate testing/landing page programs?
JM: The best advice I can give is to start testing. It’s not as scary as the name might imply, and the results will leave you wondering why you didn’t start using multivariate testing sooner.
To learn more about this topic, attend McDonald's session at the Search Engine Strategies 2009 Conference & Expo at the McEnery Convention Center in San Jose, Calif., Aug. 10-14. McDonald's session, "Real World Multivariate Testing," will take place on Aug. 12.