THE ASIA-PACIFIC CENTRE FOR RESEARCH, INC.

home > articles >

The Confluence of Data Mining and Market Research for Smarter CRM
Kenneth Elliott, Ph.D. Kenning Research Inc.
Richard Scionti SPSS Inc.
Mike Page The Kantar Group
Source: www.spss.com
Copyright SPSS, Inc. 2004

Part 1

In most companies, the realms of customer-behavior analysis and customer-attitudes analysis are worlds apart. They are like two swift flowing rivers that never meet. Behavioral analysis is typically the domain of business intelligence: tightly managed by IT and heavily focused on operational systems, data management, report servers, On-line Analytical Processing (OLAP) cube administration and Data Mining. While attitudinal analysis is the world of Market Research: owned by Marketing, often outsourced to a Market Research agency, resulting in tabular reports and executive briefing documents.

However, true holistic customer analysis demands that these worlds come together. Customers both think and act. An understanding of how customers think can help explain and predict customer behavior. Conversely, customer behaviors can help explain and predict customer attitudes. Ideally, behaviors and attitudes would be analyzed simultaneously for deeper customer understanding.

For companies with large numbers of customers, Data Mining and Market Research are often employed to gain intelligence into customer behavior and attitudes respectively. Therefore, truly holistic customer analysis requires that these two disciplines be integrated. The rivers must converge.
• Are Data Mining and Market Research integrated within your company?
• Are you optimizing your investment in behavioral and attitudinal data for a complete picture of your customers’ intentions and actions?
• Are you aware of the potential costs associated with redundant use of two disciplines to examine the same research question?
• Can you create a more efficient and accurate research operation by coordinating these disciplines for deeper customer understanding?
• What potential barriers will you face by trying to create a coordinated research operation?

In this paper, we will examine the issues surrounding the convergence of Data Mining and Market Research for deeper customer understanding. Much of the insight shared within this paper comes from the qualitative research study on the topic by SPSS Inc. and The Kantar Group entitled "Two Rivers".

We begin with a review of these two disciplines and a discussion of where they fit within an overall Customer Intelligence environment. From this foundation we will discuss where they are being applied today, how they can be converged into a collaborative research discipline and why they haven't been converged to date. Finally, we discuss some of the benefits of convergence and recommendations for creating a convergent research discipline within your organization.
What is Data Mining and Market Research?
Let’s begin our examination of the convergence of Data Mining and Market Research by exploring the basic principles and common uses of each discipline for customer2 understanding today.

Data Mining
There are several definitions of Data Mining in use today. Broad definitions suggest that Data Mining is the exploration and analysis of large data sets. Under such definitions reporting, graphing, traditional statistics and sophisticated machine learning are all considered Data Mining. In this document we use a more narrow definition of Data Mining that stresses the discovery aspect of the discipline. Specifically, we see Data Mining as the iterative process of using pattern discovery algorithms to find useful and previously unknown trends and relationships in large volumes of data. These patterns help explain past events as well as predict future events.

Data Mining is used in many industries where there is a need to find patterns in vast amounts of data. For example, Data Mining is being used to find sequences in DNA; predict manufacturing defects; identify drivers of student performance; optimize transportation logistics; forecast energy consumption; and, most recently, to identify threats to national security.

Perhaps the most widely recognized use of Data Mining is in the commercial market. Today’s businesses are using Data Mining to identify patterns in customers’ buying behavior; identify profitable customer segments; increase marketing return rates; prevent loss of valuable customers; estimate credit risk; identify fraudulent activity and much more.

The strength of Data Mining is in its ability to quickly sift through vast amounts of data to find patterns that are hidden and would otherwise be impossible to find. Data Mining often uncovers unexpected patterns, which fosters new learning and insight.

According to a 2002 report from IDC, the Data Mining market is expected to grow at a CAGR of 13% to reach $823 million in 2006. This growth can be attributed to at least four key factors.
1. There are more information sources available today than ever and the amount of information is growing exponentially.
2. The explosive growth in the capacity of databases along with the shrinking cost of data storage has made it possible to acquire, store and manage more data than ever.
3. Using Data Mining techniques used to require complex programming skills. Today, there are extremely powerful Data Mining tools on the market that are easy to use making Data Mining more accessible to a broader audience. Many operational suite vendors are beginning to embed data mining into their applications.
4. The highly competitive market environment and growing customer options makes customer intelligence more critical for business performance. This has created an increased appetite for rapidly finding knowledge from vast amounts of data.

Market Research
The American Marketing Association defines marketing research as the "systematic and objective approach to gathering marketing information which -- when processed, analyzed and interpreted -- will help identify problems and opportunities that allow for better-informed, lower-risk decisions."

In business, Market Research is typically focused on learning more about consumers, customers, competitors and market trends at large.
Depending on the source of information, Market Research is classified as either primary or secondary. Primary research uses information from original sources; that is, a Market Researcher collects data that have not been previously collected or published. Secondary research refers to collecting data from published sources such as information released by government agencies, and reports and publications available in a public library.

Primary research is classified as either qualitative or quantitative. Examples of qualitative research are focus groups and in-depth personal interviews. The most common form of quantitative research is a survey that uses a questionnaire to collect data.

The name qualitative research implies that its findings are not quantifiable. The research process is quite often a discussion in which the researcher poses open-ended questions to participants. Findings are participants' opinions, comments and impressions that cannot be tabulated to obtain averages or percentages.

Qualitative research defines issues, substantiates perception and identifies behavior. For instance, results of focus groups involving the users of a consumer product can clarify issues surrounding brand loyalty, and reveal users' likes and dislikes. Findings of personal interviews with corporate purchasing agents can aid the understanding of the criteria business firms use to select suppliers.

While qualitative research provides valuable information, it does not lend itself to rigorous data analysis that can reveal relationships among marketing variables. Quantitative research relies on survey questionnaires that are often responses to multiple-choice items or ratings on a scale. These surveys are typically conducted as either personal interviews, telephone interviews, mail surveys, or web-based surveys. Results from these surveys are then analyzed to generate averages, ranges and percentages.

When analyzing customer or consumer information, Market Research has many uses. Market segmentation studies provide information about the characteristics shared by customers. Purchasing power and buying habits studies uncover the financial strength and economic attributes shared by the target market. Psychological market studies reveal information regarding the perceived opinions and values held and shared by consumers in the market. Marketplace studies can provide insight into competitor strengths and directions. Environmental studies can provide insight into economical and political circumstances that can influence internal productivity and operations.