While this use of cross tabs seems, on the surface, to be quite mundane, there is a problem that the novice, or even the experienced analyst, may overlook. Let's take the results for both programs and combine them into a single analysis.
Table 3 simply takes Tables 1 and 2, and adds up the raw figures. The response rates then are calculated. Table 4 (see page 36) compares response rates for the three cross tabs referenced.
Wait a minute! The response rates on the total line clearly demonstrate that males are more responsive to this program. The combined statistics show this, but how could this be? Does this mean that management was incorrect in its assessment of the two solicitation efforts individually? Is there something wrong with our analysis?
G. Udny Yule, a British scientist in the early 1900s, first described the above statistical anomalies. In 1951, another statistician, Edward Simpson, further clarified the phenomenon, and, subsequently, these instances were referred to as Simpson's Paradox. The inconsistencies are characterized by the reversal of the direction of a comparison or an association when data from several groups are combined to form a single group.
Simpson's Paradox can occur whenever data are aggregated. If data are collapsed across a sub-classification (such as gender or age), then the overall change may not represent what is really happening.
The primary cause of the paradox has to do with the size of the samples being analyzed. When significantly different sample sizes are used to compare groups, you may encounter Simpson's Paradox. While trying to equate sample sizes frequently eliminates the problem, it's not always possible to assure equal sample sizes—and marketers and analysts must be suspect of sample sizes when using cross tab analyses.
Preventing this paradox from occurring may not be too difficult, either. When groups of different sizes emerging from different sources present themselves, avoid simply combining results and calculating averages. Generally, a well-designed experiment or survey, adjusting sample sizes accordingly, will not be plagued with this problem.