One example of a focused test is [a change we made to] our e-mail form, which is filled out by users who want to get information from a company listed on our site. We found that by changing this form, we can increase the number of e-mails sent. What worked best was enabling the user to type their message first, before their contact info (as opposed to entering the contact info first). The thinking behind this suggests that users are more likely to commit to sending the e-mail once they have entered their comments upfront; requiring contact information up front interferes with this commitment.
TM: What has been the biggest lesson you’ve learned about leveraging the power of multivariable testing?
BM: Since so many things can be tested with a system-generated approach, there is a temptation to do too many tests. Tests need to be properly prioritized, and there needs to be a mix of both multivariate testing and a more focused, deeper approach on specific issues. We’ve found the best approach is to use tests in succession so that focused testing is used to shape the user experience, and the multivariate testing is subsequently used to refine the experience.
By using the testing methodologies together, we can get more out of each test than we could if we used them in isolation. Testing is about discovering improvements that will hopefully have a positive impact on a user experience. It’s important to keep the goal of each test top of mind. Ultimately, it impacts our bottom line when a user has a good experience.