A Model Response
5 Steps to Building a Sound Response Model
By Forrestt Severtson
It is very easy to fall into the situation of putting the cart before the horse when it comes to response modeling in the direct mail arena. How tempting it is to let the technical aspects associated with modeling distract you from the most important principal in building a model: Business needs must define and drive the modeling process. If you don't clearly understand your business problem and how to measure it, even the best model in the world won't help you solve it.
Consider the typical problem of acquiring new customers. The naive solution is to do a mailing and work for the highest possible response rate by tweaking the various factors of a campaign, such as offer and presentation. But response by itself isn't good enough. You can get a phenomenal response rate to a low-interest credit card offer, for example, by mailing to those people with credit scores below 500 (the not-so-credit-worthy). But where would your business be in a year?
For a continuity mailer, the problem of getting new customers isn't just about getting response to an initial offer. It's about getting customers who respond to the initial offer, purchase, pay and repeat this pattern several times. The correct measure of this problem is the profitability at the third or fourth purchase cycle.
Before thinking about data or performance measurement, determine what business objectivee.g., reduce acquisition of slow-paying customers, find prospects who are likely to cross product categoriesyou want your model to address.
Define the outcome that achieves your objective.
Once you've identified the problem and how to measure it, the next step in the modeling process is to define the outcome you want the model to predict. This outcome is the result or event which, if you knew it in advance, would do the most to help you solve the business problem.
A good outcome to predict for a continuity program is not just response to a mailing, but response followed by acceptance of multiple future shipments, paid as agreed. Using this predicted outcome you can expect an increase in profitability of between 20 percent and 40 percent for a continuity programdepending on the list being usedcompared to a non-modeled mailing.
Put the prediction into action.
Decide how you are going to use the predicted outcome in your business process. For a response model, you obviously will want to mail those most likely to produce the desired outcome. But how deep in the scored list will you want to mail?
The answer to this question depends on your particular business problem and what you are measuring. For acquiring profitable customers, you probably only want to mail to those prospects most likely to have a profitability above a certain cut-off level.
Select data for the model.
Now comes the first nitty-gritty step of the process: decide what data and how much of it to use for predicting the desired outcome. The data must have some relation to the outcome you want to predict and be available at a reasonable price.
The best data for modeling is usually data that you have gathered on past performance and then augmented by external data sources.
Be careful about what data you use for modeling. There is a whole host of relatively new federal and state laws regulating what kind of data can and cannot be used for modeling in various settings. Credit bureau data historically has been the most predictive of business outcomes, but it is now the most tightly restricted data source. Your modeler can provide some guidance as to what data is usable, but be sure to check with an attorney.
The second part of the data question has to do with how much of the available data to use. There are two competing issues here: The first has to do with restricting the size of the universe, and the second has to do with expanding it.
You want to restrict your data because there are some people that you don't want to solicit. Offering a cookbook full of recipes that call for chicken, beef and pork to confirmed vegetarians is not a good business idea.
On the other hand, you want your universe to be large enough to hold up over time. People move, so addresses become invalid. You also want the universe to be large enough that in the future you can identify profitable segments that you might not have known about at the time you built the model. Lists also can shrink because of suppressions, such as marketers not wanting to mail to the same prospects more than once.
Once you have identified your universe, consider whether or not there may be different segments. Segments are groups of people that have appreciably different response rates or needs. One segment may respond well to one kind of offer, while another group may respond better to a different offer. A somewhat simple example is that of a foot-care catalog: Customers age 60 and older might respond well to a teaser offer for reduced-cost foot support shoe inserts, while the 20 to 30 age group might perform better to a sale on running shoes. Neither group would have much interest in the other's offer.
Segments can be identified from past experience or through analytic techniques. A good modeler can suggest techniques for identifying these segments. It may turn out that separate models should be built for the different segmentswith much better results than if only one model was built for the entire universe.
Set your criterion for measuring the model's performance.
Before a model is built, it is important to first decide how to measure the performance of it. A good modeler can take your performance criterion and find ways to improve that particular measure.
There are several statistical measures available. A gains chart often is one of the better ways of showing if the model is meeting your expectations. Gains charts are built by scoring the universe and showing various performance measures for each 10 percent of the score range. Response rate and profitability are most often shown in gains charts. Sometimes the separation between the best and worst 10-percent segments is a better measure of performance because of the business situation. In general, the bigger the separation between the response rates of the top and bottom deciles, the better the model.
In addition, this measurement gives you a way to gauge how the best decile will perform compared to just randomly mailing.
The sample gains chart, shown below, indicates a 4.4-percent response rate for the overall list to be mailed. The best decile brought in an 11.9-percent response rate, which is 269 percent better than the average.
The take-home message of this gains chart is that if the marketer only wanted to spend 10 percent of what it would cost to mail to all 98,990 prospects, it would still be able to get 27 percent of those that would have
responded for that 10-percent investment. Similarly, mailing 30 percent of the file would have brought in almost 60 percent of the responders.
Be able to tell if your model is still working.
The final question that most model users worry about is how long can the model be used? There is no single answer to this question, since it depends on the outcome being predicted and the data used to predict it.
The simplest way to tell if a model is still working is to track the response rate over time. As long as the modeled response rate is similar to what is shown in the gains chart, you can be fairly sure the model is working.
Another approach is to build a new model (with the same kind of data used to make the previous model) on a regular basis and compare the results of the old model to the new model. This is known as the champion/challenger method. It's a great way to ensure you have a well-performing model in use. If the challenger beats the champion, use it. If the champion beats the challenger, stay with the champion because it's a known winner.
Finally, the best way to measure the performance of a model, both from a business and an analytic perspective, is to make the same offer to random nth names on a list as you do to the best names on the same list that have been identified by the model. If the scored (modeled) group has a better response rate, the model is doing its job. If the random nth names are doing a better job, it's time for a new model ... or it could be that this random nth-name segment has some group that is particularly responsive and warrants further investigation. Using this approachwhich costs more than not mailing to the random nth namesis what distinguishes the great mailers from the good mailers.
Prior to becoming vice president of analytics at list marketing firm MarketTouch, Forrestt Severtson was the chief modeler at ChoicePoint and senior director of analytics at Equifax. He can be reached at (678) 596-4208.