Database : Model Behavior

How to tell a good analytics tool from a bad one

September 2008 By Li Zhou

multipage

4. Delayed impact? Not all results are instantaneous. Many campaigns can have a lag effect of weeks or months to see results. Some can trickle in for many months, even years. That means results from prior campaigns can get confused with results from a current campaign. Many solutions address this as a data-gathering issue, but that is rarely the best option. A good model accounts for delayed results by, once again, using at least two years of accurate data to correlate each campaign stimulation with results over time.

5. Spillover responses? Just because a call to action instructs consumers to go to a particular Web page doesn't mean they'll go there. They may go to another related Web page. Or they may find the number to customer service and call to place a new order. Consumers are wily, and many are very good at dodging the usual tracking mechanisms. The only realistic way to account for this is to measure and develop response models for each directly tracked campaign, then measure all results across all channels from the total marketing spend and compare-that's when you can detect and account for spillover responses.

6. Cross-media influence? Messages in one medium can affect responses to messages in other media channels. That's especially important for companies that market heavily across dozens of media. Accounting for these crossover effects (also known as "media synergy") can become highly complex and require very sophisticated and customized algorithms. Media mix models in particular must be able to correctly assess these effects to give accurate predictive results critical to budget decisions.

7. Data series that influence other data series? This is a sophisticated question that only the most advanced analytic models address. But the concept is not difficult to understand. Many models treat different series of data as either independent or having a static correlation, meaning if one changes at a certain time, the other changes predictably at that same time. But that's almost never the complete picture.

Far more accurate and revealing is looking at both cross-correlation and autocorrelation, which compares results to past (usually fairly recent) data series or with recent results. If you're analyzing total sales, for instance, cross-correlation determines how the recent history of various media activity affects current sales, while autocorrelation determines the relationship with prior sales.

The complex relationship of a series of data to other series of data is the most important consideration in building an accurate predictive model.

And Your Final Question
The acid test for any modeler is predictive accuracy in real time. If your model predicted that 25,000 consumers would respond to your call center this week, how many actually responded?

We counsel our clients to strive for the highest possible predictive accuracy so they can make, for example, realistic staffing decisions for their call centers. The larger and more complex your marketing program, the more important additional accuracy becomes to your ROI.

A good indicator of the ultimate quality of analytic models is the quality of the modelers themselves. Ask about the education and experience of the analysts who build the models. That should include advanced degrees in mathematics and statistics, as well as a significant number of years of experience in decision support modeling. Ask about the types of clients they've built and run models for. They should be clients as big and as sophisticated as your company.

What's Your Payoff?
Analytics is a frightening topic to many in marketing. Why is that? It can't be because it's complex. Marketing, especially in this age of splintered media, is equivalent to rocket science (where brand and product is the payload). But increasingly, analytics is the telemetry (literally "remote measurement") of marketing, the feedback data that allows us to change course when necessary. At its core is statistical mathematics that are daunting to all but the most highly trained.

But the potential payoff is enormous. The seven questions discussed here build up to what might be called integrated marketing decision support, with predictive and optimization capabilities that are just now infiltrating marketing management.

Imagine the future: A company needs X sales. Using advanced analytics, we can determine the exact marketing budget - broken into specific media budgets from direct to mass with precise flight plans - that will yield those results. We don't get there from data and number crunching alone. It still takes people. All the experience a marketer has is still required to interpret the analytical results.

So how can you tell which companies are using the best analytics models? It's not always readily apparent from an outside perspective. A good indication is the degree to which a company's marketing efforts are integrated across multiple media. The more you see coordination, the more you can bet that it's learned how to tell a good analytics model from a bad one.

Li Zhou, Ph.D., is vice president of research and modeling for Javelin Direct. After earning bachelor's and master's degrees in mathematics, Zhou completed his Ph.D. in management science. He has 15 years of professional experience, first in demand forecasts and optimizations for airlines, followed by media-mix modeling for marketing. He can be reached at (972) 443-7000. 

• MORE ON
• DATABASE & CRM