Customer Data Mining
How to determine if an in-house or outsourced solution is right for you
February 2006 By Jim Wheaton
|
Customer data mining is a complex process that involves highly trained professionals. Some companies handle data mining in house, while others farm it out, and still others follow a hybrid solution. Which option is right for you? Here are some factors to consider when you’re making this difficult decision.
What Can You Afford?
Most mid-size to large direct marketers have an in-house data mining department to handle at least some
of the analytics work. They feel it’s important to have total control of this critical function, and for the data miners to be continuously steeped in the business. Also, the cost of an in-house staff can be lower than an outsourced solution.
However, an in-house solution can be problematic for smaller direct marketers because employee turnover is an unfortunate fact of life. Having your own mining group is risky if your company is not large enough to support overlapping personnel.
Many smaller firms have been hurt badly when a key data miner has moved on to greener pastures. Generally, when this occurs, much or all “company memory” is lost forever. Therefore, when deciding whether to do customer data mining in house, you should reflect on how much staff you realistically can support.
For example, a well-known retail/catalog/e-commerce marketer recently lost its lone in-house data miner. Until it finds a replacement, the firm is in a tenuous situation in which predictive models are embedded in the production processes, driving decisions on whom to promote, but there is no statistician to interpret what’s going on.
Are You Seeking Ongoing Support or One-off Work?
Consistently excellent target marketing requires ongoing data mining support. However, some firms aren’t able to make that commitment. Instead, they pursue one-off assignments. Under such circumstances, the only logical choice is to outsource the work.
When farming out your data mining, beware that there is significant overhead the first time a vendor works with your customer file. The first project always is more expensive. To do an effective analytical job, it’s important for the analytics firm to totally understand your business and data.
For example, a major publisher with millions of subscribers across multiple titles decided to incorporate sophisticated data-driven marketing into its core business practices. As part if this initiative, it outsourced more than 40 custom-built predictive models over a one-year period. These models drove selections for the cost-effective micromarketing of dozens of products and services to the customer base.
The first model took the outside data mining company about eight weeks to build. However, several months into the relationship, the publisher asked if three models could be constructed in a single week. By then, the processes were so streamlined that this was feasible. As you can imagine, the data mining firm was able to charge a more favorable per-model price than it could have for a one-off project!
Where is Your Database Hosted?
It can make good sense for a direct marketer that hosts its own marketing database also to handle the data mining. When a company already has the data, it is relatively straightforward for it to do the mining.
Conversely, a direct marketer that has outsourced the housing of its database often will rely on the hosting company to do the data mining. This can be an efficient and beneficial relationship, provided data mining is a core competency of the data management firm.
Cost and Availability
Outsourcing will be of little value if your vendor provides project quotes that are well above your budget, or if its data mining department is so swamped that it’s difficult to schedule any of its time.
One way to ensure cost-effective availability is to negotiate a retainer relationship in exchange for an agreed-upon level of support. Often, such arrangements specify the identity of an individual who acts as a de facto employee of the client. Another advantage is that the dedicated data miner becomes totally steeped in your business.
Hybrid Solutions
An alternative to outsourcing data mining or performing these functions in house is to get the best of both worlds in a hybrid solution. One approach is to retain an outside consultant to mentor a one- or two-person data mining department. The consultant provides the real-world perspective and guidance on specific technical issues that only can be gained by years of in-the-trenches industry experience. An added benefit is that the consultant maintains the “company quantitative memory” during times of in-house staff turnover. The on-staff data miners, in turn, do all the heavy lifting, which keeps consulting fees at a reasonable level.
Another hybrid approach is to hire an outside firm for the most challenging assignments. That way, the ongoing heavy lifting is done by the in-house staff, which has a moderating effect on overall costs, while the outside firm focuses on “pushing the envelope.”
One example is a multibillion dollar company that sends hundreds of direct mail and e-mail promotions a year across its direct and store divisions. The firm has a keen interest in employing quantitative methods and test strategies to optimize this significant marketing investment. The company has a substantial budget for in-house database marketing, including many senior data miners and a state-of-the-art data repository.
Nevertheless, this firm always is on the lookout for new ways of thinking. It understands that its own people don’t have all the answers. Therefore, it engaged an outside data mining group to spearhead an approach to contact optimization. The project took about a year to complete, producing such strong results that the company hired the outside group to tackle other data challenges.
Important Intangibles
Regardless of whether you go with an in-house or outsourced solution, consider the experience and longevity of your data miners. Unfortunately, there are many service companies whose idea of a data mining department is to hire a handful of low- to mid-level staffers. Often, these individuals have solid academic credentials but little industry experience. What you learn in a statistics course has little bearing on most of the real-world decisions that data miners in the industry face. There are many non-statistics-based issues that must be navigated successfully to build an effective predictive model. Among them are:
* Which mailings/drops should comprise the analysis file?
* Should a single- or multiple-model strategy be employed?
* What should be predicted; that is, what is the definition of the dependent variable(s)?
* Of all the possible permutations of data elements in the marketing database, which should be included as potential predictor variables?
* For each potential predictor, does the statistical relationship to the dependent variable make ongoing business sense?
Also, make sure the group you work with maintains a collegial atmosphere with a clear path for career advancement. Stagnation can be a problem with small groups. Thinking can become in-bred. Such environments encourage a perverse sort of self-selection, where those with the best analytical minds grow restive and start looking for other employment. This is less of a problem with larger in-house departments and the best of the independent data mining companies, where a steady stream of diverse projects keeps advancing the collective knowledge.
Jim Wheaton is a co-founder and principal at Wheaton Group, a data mining and decision sciences practice based in Chicago. He can be reached at (919) 969-8859 or by e-mail at jim.wheaton@wheaton group.com.
What Can You Afford?
Most mid-size to large direct marketers have an in-house data mining department to handle at least some
of the analytics work. They feel it’s important to have total control of this critical function, and for the data miners to be continuously steeped in the business. Also, the cost of an in-house staff can be lower than an outsourced solution.
However, an in-house solution can be problematic for smaller direct marketers because employee turnover is an unfortunate fact of life. Having your own mining group is risky if your company is not large enough to support overlapping personnel.
Many smaller firms have been hurt badly when a key data miner has moved on to greener pastures. Generally, when this occurs, much or all “company memory” is lost forever. Therefore, when deciding whether to do customer data mining in house, you should reflect on how much staff you realistically can support.
For example, a well-known retail/catalog/e-commerce marketer recently lost its lone in-house data miner. Until it finds a replacement, the firm is in a tenuous situation in which predictive models are embedded in the production processes, driving decisions on whom to promote, but there is no statistician to interpret what’s going on.
Are You Seeking Ongoing Support or One-off Work?
Consistently excellent target marketing requires ongoing data mining support. However, some firms aren’t able to make that commitment. Instead, they pursue one-off assignments. Under such circumstances, the only logical choice is to outsource the work.
When farming out your data mining, beware that there is significant overhead the first time a vendor works with your customer file. The first project always is more expensive. To do an effective analytical job, it’s important for the analytics firm to totally understand your business and data.
For example, a major publisher with millions of subscribers across multiple titles decided to incorporate sophisticated data-driven marketing into its core business practices. As part if this initiative, it outsourced more than 40 custom-built predictive models over a one-year period. These models drove selections for the cost-effective micromarketing of dozens of products and services to the customer base.
The first model took the outside data mining company about eight weeks to build. However, several months into the relationship, the publisher asked if three models could be constructed in a single week. By then, the processes were so streamlined that this was feasible. As you can imagine, the data mining firm was able to charge a more favorable per-model price than it could have for a one-off project!
Where is Your Database Hosted?
It can make good sense for a direct marketer that hosts its own marketing database also to handle the data mining. When a company already has the data, it is relatively straightforward for it to do the mining.
Conversely, a direct marketer that has outsourced the housing of its database often will rely on the hosting company to do the data mining. This can be an efficient and beneficial relationship, provided data mining is a core competency of the data management firm.
Cost and Availability
Outsourcing will be of little value if your vendor provides project quotes that are well above your budget, or if its data mining department is so swamped that it’s difficult to schedule any of its time.
One way to ensure cost-effective availability is to negotiate a retainer relationship in exchange for an agreed-upon level of support. Often, such arrangements specify the identity of an individual who acts as a de facto employee of the client. Another advantage is that the dedicated data miner becomes totally steeped in your business.
Hybrid Solutions
An alternative to outsourcing data mining or performing these functions in house is to get the best of both worlds in a hybrid solution. One approach is to retain an outside consultant to mentor a one- or two-person data mining department. The consultant provides the real-world perspective and guidance on specific technical issues that only can be gained by years of in-the-trenches industry experience. An added benefit is that the consultant maintains the “company quantitative memory” during times of in-house staff turnover. The on-staff data miners, in turn, do all the heavy lifting, which keeps consulting fees at a reasonable level.
Another hybrid approach is to hire an outside firm for the most challenging assignments. That way, the ongoing heavy lifting is done by the in-house staff, which has a moderating effect on overall costs, while the outside firm focuses on “pushing the envelope.”
One example is a multibillion dollar company that sends hundreds of direct mail and e-mail promotions a year across its direct and store divisions. The firm has a keen interest in employing quantitative methods and test strategies to optimize this significant marketing investment. The company has a substantial budget for in-house database marketing, including many senior data miners and a state-of-the-art data repository.
Nevertheless, this firm always is on the lookout for new ways of thinking. It understands that its own people don’t have all the answers. Therefore, it engaged an outside data mining group to spearhead an approach to contact optimization. The project took about a year to complete, producing such strong results that the company hired the outside group to tackle other data challenges.
Important Intangibles
Regardless of whether you go with an in-house or outsourced solution, consider the experience and longevity of your data miners. Unfortunately, there are many service companies whose idea of a data mining department is to hire a handful of low- to mid-level staffers. Often, these individuals have solid academic credentials but little industry experience. What you learn in a statistics course has little bearing on most of the real-world decisions that data miners in the industry face. There are many non-statistics-based issues that must be navigated successfully to build an effective predictive model. Among them are:
* Which mailings/drops should comprise the analysis file?
* Should a single- or multiple-model strategy be employed?
* What should be predicted; that is, what is the definition of the dependent variable(s)?
* Of all the possible permutations of data elements in the marketing database, which should be included as potential predictor variables?
* For each potential predictor, does the statistical relationship to the dependent variable make ongoing business sense?
Also, make sure the group you work with maintains a collegial atmosphere with a clear path for career advancement. Stagnation can be a problem with small groups. Thinking can become in-bred. Such environments encourage a perverse sort of self-selection, where those with the best analytical minds grow restive and start looking for other employment. This is less of a problem with larger in-house departments and the best of the independent data mining companies, where a steady stream of diverse projects keeps advancing the collective knowledge.
Jim Wheaton is a co-founder and principal at Wheaton Group, a data mining and decision sciences practice based in Chicago. He can be reached at (919) 969-8859 or by e-mail at jim.wheaton@wheaton group.com.
Companies Mentioned:
 |
|
Social Media ROI
Email Marketing that Works (2nd Edition)