The Confluence of Data Mining and Market Research for Smarter CRM
Kenneth Elliott, Ph.D. Kenning Research Inc.
Richard Scionti SPSS Inc.
Mike Page The Kantar Group
Source: www.spss.com
Copyright SPSS, Inc. 2004

Part 4


Data Mining Process

Data Mining most commonly defines the customer as a set of trackable behaviors. This is due in large part to the fact that Data Mining requires large data sets. These are more often produced by operational systems than surveys. This means that the customer is defined as an acting entity with less input from intentions, attitudes or outside behaviors. Therefore, Data Mining focuses on capturing what is accessible via operational systems that interact with the customer.

These systems produce massive amounts of transactional data including purchases, customer service inquiries, web visits, phone logs and more. The data is stored in large data warehouses. The analysis of this data requires highly scalable algorithms that churn through the data looking for common aggregate patterns. Customer understanding is derived from interpreting behavioral patterns. Intentions are then inferred from actions. Finally, the insights gained through Data Mining are represented in the form of 'models' that can be used to score databases and real-time applications.

Market Research Process

Market Research defines the customer as a thinking affective entity where intentions and attitudes are more important than actions. Market Research often defines the ‘customer’ as a group within the general population. Being freed from the internal corporate database, Market Research is able to explore questions such as competitive-product assessments, intentions to defect and general satisfaction. The data outputs are subjective comments and ratings. The data is often captured in the form of spreadsheets or text files and delivered in the form of written reports. The analysis of this data is a subjective summary of the results and interpretation of meaning across the responses. Customer understanding is gained by linking the attitudes of general population segments to the assumed makeup of a client’s existing customer base. Deployment of market research results occurs through presentations to decision makers.

Combining Processes

Combining Data Mining and Market Research will require synergy at each stage of the research process. While the customer deserves to be seen as a thinking and acting entity, combining these disciplines provides the unique ability to analyze the gaps that are known to exist between espoused plans and practice. Thus data capture must expand to include all information, subjective and objective, intentions and actions. The storage of data must come together so that the analysis stage can leverage both. In addition, the analysis stage must leverage new processes that take advantage of the best of both disciplines, including empirical behavioral modeling and qualitative research methods. Finally, the deployment of insight, whether to human or machine, should take advantage of the knowledge gained from both Data Mining and Market Research. Only when a full perspective of the customer is available can holistic conclusions be drawn and the most accurate insight can be deployed.
For a more detailed examination of the convergence of Data Mining and Market Research practice, see Convergent Research Patterns (Kenning Research Inc., 2003).

Why aren’t Data Mining and Market Research Converged Today?

Despite their shared fit within customer intelligence, their commonality of application, and their similarity of research stages, Data Mining and Market Research are still not converged into a unified research environment today. While there are examples of leading companies who have converged disciplines for ad hoc research, systematic convergence has been hindered by several factors. Among the most challenging barriers to convergence are separations between Data Mining and Market Research with respect to organizational structure, culture, and infrastructure.

Organizational Separation

In most organizations today, Data Mining and Market Research operations are housed within different parts of the business. This physical separation hinders interaction and cooperation. Organizational separation also implies that two decision-makers, both tasked with customer intelligence, are operating under different strategies and objectives.

Cultural Separation

The cultural separation between Data Mining and Market Research can be seen from the executive and field level. At the executive level, there tends to be a decision-making culture that is based more heavily on either internal analytics or Market Research. The comfort of decision-makers toward one approach over the other perpetuates the separation of disciplines.

At the field level, there may exist an adversarial relationship between Data Miners and Market Researchers. This atmosphere of non-cooperation hinders the advancement of research.

"Anything where a person's identity is used isn't Market Research, it's spying...We [Market Researchers] are always at risk of getting a bad name from people who mistake Market Research and Data Mining, which is about finding out enough about people to sell them something." President of a Market Research Society

"What we need is not market research, it’s more transactional data. It is well known that past behavior is the best predictor of future behavior. Attitudinal research is weak at best." Data Mining Expert

Infrastructure

Today, Market Research and Data Mining rely on separate internal infrastructures. Bringing these two disciplines together will require the integration of technologies that are not widely integrated today. Such technologies include data collection, data management, data storage, data analysis/reporting, and deployment. As well as general applications such as project management and knowledge management.

What are the Benefits of Converging Data Mining and Market Research?

Maintaining two separate disciplines for consumer research, Data Mining and Market Research leads to:
• Non-optimized use of available data
• Non-optimized use of new learning
• Redundant treatment of similar research questions
• Sub optimal conclusions drawn when one discipline is used where the
   other would have been more effective
• Ultimately, the potential for non-optimized intelligence at a higher cost

Organizations that commission Data Mining and Market Research are often rich with data. In many cases, Data Mining and Market Research can be improved with the inclusion of data generated for use by the other discipline. Bringing these two research areas together can lead to the identification of available data, which can be leveraged to derive deeper, more accurate insight.

By not converging these disciplines there is the risk that knowledge gained from one research initiative isn't shared with the other. This can lead to the formation of conclusions that could have been improved by previous learning.

Certainly, an organization would want to avoid a situation where both disciplines are being used in an uncoordinated manner to address the same research question. For example, it is not uncommon for organizations to commission market research agencies to study the issue of customer loyalty, while in another initiative they have commissioned data analysts to develop models of customer retention. This is a good example of each discipline providing a unique and valuable contribution to the research question. Yet, the results will be sub-optimized and more expensive if they are not coordinated.