Not all data are created equal. There are one-dimensional demographic and firmographic data, then there are more colorful behavioral data. The former is about how the targets look, and the latter is more about what they do, like what they click, browse, purchase and say. On top of these, if we are lucky, we may have access to attitudinal data, which are about what the target is thinking about. If we get to have all three types of data about the customers and prospects, prediction business will definitely get to the next level (refer to “Big Data Must Get Smaller"). But the reality is that it is very difficult to know everything about anyone, and that is why analytics is really about making the best of what we know. Predictive modeling is useful not only because it predicts the future, but also fills gaps in data. And even in the age of abundant data, there are many holes, as we will never have a complete set of information (refer to “Why Model?”).
Among these data types, some are more useful for prediction than others. Behavioral data definitely possess more predictive power than simple demographic data for sure. But alas, they are harder to come by. It could be that the target is new to the environment, so she may not have left much data behind at all. May be she just looked around and didn’t buy anything yet. Or she is very privacy-conscious and diligent about erasing her behavioral trails on the net or otherwise. Maybe she explicitly opted out of being traced at all, giving up much of the convenience factors of being known by the merchants. Then the data coverage comes into the equation, and that is why analysts rely on demographic and geo-demographic data for their readily available nature. Much of such data can easily be purchased and appended on a household or individual level, at least in the U.S. If we get to have some hint of identity of the target, there are ways to merge disparate data sets together.
What if we don’t get to know who are leaving data trails? Again, it could be about the privacy concerns of the target, or the manner by which the data are collected. Some data collectors avoid personally identifiable information, such as name, address or email, as they do not want to be seen as the Big Brother. Even if collectors get to have access to such PII, they do not share it with outsiders, to maintain dominance and to avoid the data privacy issue altogether. And there are many instances where that “who” part is completely out of reach. Movement data would be an example of that.
Weaving multiple types of data together is often the main source of trouble when it comes to predictive analytics. I have been talking about the importance of a 360-degree view of a customer for proper personalization and attribution, but the main show-stopper there is often the inability to merge data sources with confidence, not the lack of technology or statistical skills. That would be the horizontal challenge when dealing with multiple types of data.
Then there is the time factor. Like living organisms, data get old and wither away, too. Let’s call it the "data atrophy" challenge. Data players must be mindful about it, as outdated information is often worse than not having any at all for the decision-making or prediction business.
Now, not all data types deteriorate at the same rate. The shelf-life of demographic data are far longer than that of behavioral data. For example, people’s income levels or housing size do not change overnight, while usefulness of what we call “hotline” data evaporates much faster. If you get to know that someone is searching for a new car, how long will he be in the market? What if it is about a ticket or pay-per-view purchase for tonight’s ball game? Data that is extremely valuable this minute could be totally irrelevant within the next hour.
However, is that really true? Do seemingly expired data really not have any value for future prediction? Yes, “real-time” data is one the most over-used terms in marketing, and yes, if marketers get to know someone is looking for some product that they sell, they need to react to such information right at that moment. In fact, most personalization efforts and related technologies are about such reactionary endeavors. In such cases, there is not much need for modeling or statistical work. If you know someone is looking for a mid-size luxury car, lead him to it now. If the customer is following a certain path on the journey map, usher her accordingly. But the “hot” data becomes history eventually, and historical data definitely has its place in the prediction business.
I learned from experience that the best predictor of future behavior is past behavior. People do not change that drastically. Most customers and prospects can be defined by what they have been doing. Those who shopped for luxury cars in the past will do that again sometime in the future. Folks who have been regularly buying golf equipment will do that again. Movie fans who prefer drama over action films won’t change their taste overnight. And statistical modeling will not only fill in the data gaps as I explained earlier, but also keep data that are getting colder lukewarm, at least for some time. There is a tremendous amount of value in elongating the usefulness of data, as so-called “hot” data only describes a small fraction of the target, only for those who left data trails in real-time. If marketers want to make a sizable improvement in their personalization efforts, following such a small target is simply not enough. They must expand the universe to the maximum.
Nevertheless, such modeling work – to fill in the data gaps or keep the data warm – becomes effective, only if we treat fresh and not-so-fresh information properly. That is why a simple time stamp must be converted into measurements of intervals between events. Instead of simply calling someone a golfer because his number of purchases in the golf equipment category to-date are higher than average, data players must add time elements to the data menu. Because he may not be a golfer any more. How many weeks have elapsed since the last purchase in the golf category? What are the average number of days between transactions there? What are the average number of weeks between new product releases and actual purchase of the item? In other words, does he buy the newest items at a premium price, or does he wait until the price comes down a bit? Or, is he a bargain seeker who only buys older models? How about such measurement by channel? If he doesn’t show up at stores any more, maybe he turned into an online buyer for consumable items, such as golf balls? None of these are what we would call hotline data, but they will be useful for targeting and personalization.
Yes, if the buyer is specifically looking for a new driver of certain brand, by any means, usher him there and close the sale. But let’s get ready for the next time he shows up (or for proactive promotion of other related items) by collecting, maintaining and transforming historical data, as well. Once these predictors (i.e., descriptive variables in models) are in line, statistical modeling can assign “scores” for specific behaviors that marketers care for. Along the line of this example, such model scores can be assigned for “cutting-edge buyers,” “bargain seekers,” “online buyers of repeat items,” “infrequent high-value customers,” “frequent small item buyers,” etc. No, these personas won’t necessarily be “hot,” but they are a valuable summary of all types of old and new data, and they will available for everyone all the time as they are just model scores (refer to “No One Is One-Dimensional”).
We are living through constant communication via multiple devices on any given day, but too many messages that consumers receive are utterly irrelevant and inadequate. Even the messages that are based on real-time information can turn into repeated nuisances. (How many times can you, as a consumer, bear to see a constant barrage of ads for an item that you clicked and viewed once a week ago?). And if there is no hotline data in play? We get to see the same generic messages through multiple channels.
That is why we must not treat data atrophy just as a problem, but embrace aging data as opportunities for the future. Today’s data become historical data in a blink, but we still have a lot to mine there. And such mining is possible, only if we arrange the data properly and let it age gracefully using statistical techniques. That is the way to personalize messages constantly for everyone, instead of reacting to real-time data only sporadically for a fraction of your audience.
Stephen H. Yu is a world-class database marketer. He has a proven track record in comprehensive strategic planning and tactical execution, effectively bridging the gap between the marketing and technology world with a balanced view obtained from more than 30 years of experience in best practices of database marketing. Currently, Yu is president and chief consultant at Willow Data Strategy. Previously, he was the head of analytics and insights at eClerx, and VP, Data Strategy & Analytics at Infogroup. Prior to that, Yu was the founding CTO of I-Behavior Inc., which pioneered the use of SKU-level behavioral data. “As a long-time data player with plenty of battle experiences, I would like to share my thoughts and knowledge that I obtained from being a bridge person between the marketing world and the technology world. In the end, data and analytics are just tools for decision-makers; let’s think about what we should be (or shouldn’t be) doing with them first. And the tools must be wielded properly to meet the goals, so let me share some useful tricks in database design, data refinement process and analytics.” Reach him at email@example.com.