Database : Model Behavior
How to tell a good analytics tool from a bad one
September 2008 By Li Zhou
|
It's more than an idle question. The difference could mean meeting your goals or not, or having to go back to beg for more marketing dollars before the year is out. It could mean spending way too much on a media channel that's beginning to suffer fatigue and not nearly enough on a fresh media channel that could rocket your sales. It also could mean you over- or understaffed call centers because you weren't able to accurately predict customers' responses to your calls to action.
The marketing industry is beginning to learn the value of analytic models, but the quality of models out in the market range from so poor you may as well just guess to nearly spot-on perfect. What you need is a set of questions to ask, a few basic rules against which to judge the value of a model.
The Seven Questions
These are the essential questions to ask providers of marketing models to determine if the model is a good fit for your company. They drill directly into the most important elements of reliable, verifiable and predictive models-essentially those elements that are most useful to you.
How does your model account for:
1. New media for which you have no historical data? When a product or service has been marketed over certain media for a period of time, and you want to explore a medium you've never used, you're in new analytical territory. Even the best models must import assumptions and data from other sources to account for behavior in new media. Truly sophisticated modelers have become quite good at transferring this knowledge with a high degree of predictive accuracy.
2. Insufficient historical data? To have a rock-solid foundation, models should be based on a couple years of high-quality historical data. Many analytics groups use data that covers only a few months. You should always be wary of any models that call for only a small amount of historical data.
3. Diminishing returns due to media fatigue? All media tend to have diminishing returns as you use them over time, especially if you continue to raise the ante and invest more into them. The return on your first million spent is almost always larger than on your last million, all other things being equal. Defining the decay curve takes historical data that shows response (usually sales) versus marketing spend for each media. What you invariably detect is a curve that shows a startup investment to get to a high payoff zone, then a slow decline as returns diminish. The only way to realistically account for this is to develop models based on accurate data over at least two years. Models that do not account for this will miss their predictive marks.
4. Delayed impact? Not all results are instantaneous. Many campaigns can have a lag effect of weeks or months to see results. Some can trickle in for many months, even years. That means results from prior campaigns can get confused with results from a current campaign. Many solutions address this as a data-gathering issue, but that is rarely the best option. A good model accounts for delayed results by, once again, using at least two years of accurate data to correlate each campaign stimulation with results over time.
5. Spillover responses? Just because a call to action instructs consumers to go to a particular Web page doesn't mean they'll go there. They may go to another related Web page. Or they may find the number to customer service and call to place a new order. Consumers are wily, and many are very good at dodging the usual tracking mechanisms. The only realistic way to account for this is to measure and develop response models for each directly tracked campaign, then measure all results across all channels from the total marketing spend and compare-that's when you can detect and account for spillover responses.
6. Cross-media influence? Messages in one medium can affect responses to messages in other media channels. That's especially important for companies that market heavily across dozens of media. Accounting for these crossover effects (also known as "media synergy") can become highly complex and require very sophisticated and customized algorithms. Media mix models in particular must be able to correctly assess these effects to give accurate predictive results critical to budget decisions.
7. Data series that influence other data series? This is a sophisticated question that only the most advanced analytic models address. But the concept is not difficult to understand. Many models treat different series of data as either independent or having a static correlation, meaning if one changes at a certain time, the other changes predictably at that same time. But that's almost never the complete picture.
Far more accurate and revealing is looking at both cross-correlation and autocorrelation, which compares results to past (usually fairly recent) data series or with recent results. If you're analyzing total sales, for instance, cross-correlation determines how the recent history of various media activity affects current sales, while autocorrelation determines the relationship with prior sales.
The complex relationship of a series of data to other series of data is the most important consideration in building an accurate predictive model.
And Your Final Question
The acid test for any modeler is predictive accuracy in real time. If your model predicted that 25,000 consumers would respond to your call center this week, how many actually responded?
We counsel our clients to strive for the highest possible predictive accuracy so they can make, for example, realistic staffing decisions for their call centers. The larger and more complex your marketing program, the more important additional accuracy becomes to your ROI.
A good indicator of the ultimate quality of analytic models is the quality of the modelers themselves. Ask about the education and experience of the analysts who build the models. That should include advanced degrees in mathematics and statistics, as well as a significant number of years of experience in decision support modeling. Ask about the types of clients they've built and run models for. They should be clients as big and as sophisticated as your company.
What's Your Payoff?
Analytics is a frightening topic to many in marketing. Why is that? It can't be because it's complex. Marketing, especially in this age of splintered media, is equivalent to rocket science (where brand and product is the payload). But increasingly, analytics is the telemetry (literally "remote measurement") of marketing, the feedback data that allows us to change course when necessary. At its core is statistical mathematics that are daunting to all but the most highly trained.
But the potential payoff is enormous. The seven questions discussed here build up to what might be called integrated marketing decision support, with predictive and optimization capabilities that are just now infiltrating marketing management.
Imagine the future: A company needs X sales. Using advanced analytics, we can determine the exact marketing budget - broken into specific media budgets from direct to mass with precise flight plans - that will yield those results. We don't get there from data and number crunching alone. It still takes people. All the experience a marketer has is still required to interpret the analytical results.
So how can you tell which companies are using the best analytics models? It's not always readily apparent from an outside perspective. A good indication is the degree to which a company's marketing efforts are integrated across multiple media. The more you see coordination, the more you can bet that it's learned how to tell a good analytics model from a bad one.
Li Zhou, Ph.D., is vice president of research and modeling for Javelin Direct. After earning bachelor's and master's degrees in mathematics, Zhou completed his Ph.D. in management science. He has 15 years of professional experience, first in demand forecasts and optimizations for airlines, followed by media-mix modeling for marketing. He can be reached at (972) 443-7000.
Companies Mentioned:
 |
|
Social Media ROI
Email Marketing that Works (2nd Edition)