Database : Get the Dupes Out

Modern tactics for ridding databases of duplicate customer records

Approaches to data hygiene introduced 30 or 40 years ago have been rendered obsolete. Back then, the average person moved only once in a decade, was less likely to divorce and typically relied on a single phone line for the entire household. That is a sharp contrast to today, when typical consumers make 11 moves during their lifetimes and 2 million people divorce each year. And, as we all know, the majority of households rely on multiple cell phones, often in lieu of a dedicated land line.

Today, any multichannel marketer can tell you that its customer files change almost daily, with new information contributed from stores, catalogs and the Web. Even with the most strenuous efforts in place to qualify new-to-file transactions, customer files continue to accumulate disparate, and seemingly unrelated, transactions.

Several dynamic factors make it very difficult to correctly identify and value each customer with traditional identity consolidation processes. Consider that:

• Most merge/purge processes were developed more than 30 years ago and were never intended to recognize the fluidity of movement, name changes and vastness of channels in which customers interact today. Multiple channels equate to name and address permutations that accrue as the customer's interactions across channels grow.

• In today's increasingly online and mobile society, about 30 percent of movers each year do not bother to complete the USPS's national change of address process. As a result, the NCOA file is not as accurate as it once was.

• In today's economy, the decline in consumer spending and rising postage costs for the marketer are making accurate data a critical must-have.

Surprisingly, most marketers still primarily rely on NCOA as their dominant tool to identify self-reported movers. While it might be easier than ever to update one's address—either in person, via mail or online—the fact remains that many consumers are not completing this information. In fact, on average, approximately 4 percent to 6 percent of marketers' files contain missed movers.

Data De-Duping Today
Relationship management starts with accurate data. To effectively reach your customers through their preferred channels, to be responsive to their contact preferences and requests, and to achieve cost savings on postage by reducing waste in your mail files—not to mention, to be mindful of conserving natural resources—you must keep your files as error-free as possible.

Older de-duplication processes use character-based logic and look-up tables that are easily deceived by minor variations in the name and address elements, such as married names versus maiden names, nicknames, typos and mis-keys.

Fortunately, marketers are starting to tap newer technology that uses referential transaction databases. Overall, these databases make it possible to provide current addresses for the majority of movers for which no information exists. This often happens when the mover does not complete the NCOA process or when an NCOA card or entry did not match the existing name and address on file.

These databases serve as a knowledge base of millions or billions of data elements culled from purchase transaction information, the marketer's own data and outside data sources. Including transactional data helps create a highly accurate view of consumers at any given address or point in time.

How It Works
Newer technology can automate processes that once required human intervention. These newer solutions employ advanced customer recognition technology and "fuzzy logic" to address the identity recognition issues caused by the dynamic nature of customers and the multiple methods of data collection in use today. These methods leverage newer and more sophisticated data matching algorithms and are applied to customer recognition challenges. Such solutions can now identify and "collapse" customers with multiple identities, regardless of name and address permutations, omissions or mis-keys.

Today's advanced customer recognition solutions incorporate several distinct types of advanced matching algorithms, each designed to group records into temporal data sets for the purpose of bringing visibility to distinct patterns of repetitious error.

Once patterns of error are identified, the records can be referenced to external data sources, such as a separate transactional database, to correct the errors and uncover the true identity of the customer. Comparing your data to an external data source provides transactional evidence and validates that a person exists at an address at a specific point in time (see an examplle, in the mediaplayer to the right, of a typical duplicates situation and the resulting consolidation).

Getting Started
Until recently, advanced de-duping tools were out of reach financially for the average marketer.

Fortunately, as the adoption of newer approaches increase, it is bringing the cost of these solutions within the realm of possibility for many marketers. Plus, more marketers have realized success in their multichannel programs, and this is driving overall interest in technological solutions that can bring about cost savings. Improvements in technology have, historically, been embraced by direct marketers, who realize that, by decreasing the number of undeliverables, the technology will help pay for itself.

Ideally, marketers should turn to tools that can readily integrate with their existing processes and can perform the following functions:

• Repair and perfect incomplete addresses.
• Accurately identify undeliverable addresses.
• Locate movers who were not found using NCOA-Link alone.

It is only a matter of time before automated "Data 2.0" solutions are widely adopted in the marketplace. After all, the results are significant. You will reduce your mail costs, and you will truly have the correct, consolidated identity and location of every customer for all of your direct marketing campaigns. You will then be able to determine each customer's individual value to your company. Plus, you will help increase customer loyalty and boost your image, because you will no longer be sending irrelevant or duplicate mailings to your top customers.

Rod Ford is president and CEO of CognitiveDATA (a Merkle Company), a marketing technology company serving multichannel merchants. He can be reached at (866) 243-7883 or rford@cognitivedata.com.

• MORE ON
• DATABASE & CRM