Think You Know Churn? We’ll See About That.April 21, 2014 By Matt Shanahan
If you implement scoring rules for identifying at-risk customers, you will want to be sure the rules actually help your customer success organization, i.e., that the rules provide a return on investment (ROI). A scoring rule with low accuracy hurts your organization by reducing efficiency and effectiveness. Many scoring rules are developed from the experience and intuition of customer success managers, but most have not been validated as to their customer coverage or predictive accuracy—the two facets that determine ROI in predicting churn.
What Are These Facets and How Do They Affect Your Organization?
Customer coverage is the percentage measurement of the customer base for which the scoring rule provides insight. For example, if a rule is implemented based on survey response, there will be low coverage. Typically, 20 percent is considered an awesome response rate for a survey. Most survey response rates are in the 10 percent to 15 percent range. The implication is more than 80 percent of your customers will not respond to the survey—that means that scoring rule based on survey responses has low customer coverage. Creating rules around other metrics, such as number of open support cases or attendance at customer events, can fall into the low coverage category. Rules with high customer coverage include demographic, firmographic, purchase history and usage metrics.
Predictive accuracy is the percentage measurement of a rule's true vs. false predictions. In the case of churn, if a scoring rule predicts someone to be a non-churner and they do in fact churn, that is a false prediction. Likewise, another false prediction would be if the rule predicts a churner and, without any intervention from customer success, the churner renews. True predictions are, of course, the opposite of these. The percentage of true predictions vs. the total provides predictive accuracy.