<<

. 9
( 137 .)



>>

Clustering is the task of segmenting a heterogeneous population into a num­
ber of more homogeneous subgroups or clusters. What distinguishes cluster­
ing from classification is that clustering does not rely on predefined classes. In
classification, each record is assigned a predefined class on the basis of a model
developed through training on preclassified examples.
In clustering, there are no predefined classes and no examples. The records
are grouped together on the basis of self-similarity. It is up to the user to deter­
mine what meaning, if any, to attach to the resulting clusters. Clusters of
symptoms might indicate different diseases. Clusters of customer attributes
might indicate different market segments.
Clustering is often done as a prelude to some other form of data mining or
modeling. For example, clustering might be the first step in a market segmen­
tation effort: Instead of trying to come up with a one-size-fits-all rule for “what
kind of promotion do customers respond to best,” first divide the customer
base into clusters or people with similar buying habits, and then ask what kind
of promotion works best for each cluster. Cluster detection is discussed in
detail in Chapter 11. Chapter 7 discusses self-organizing maps, another tech­
nique sometimes used for clustering.
12 Chapter 1


Profiling
Sometimes the purpose of data mining is simply to describe what is going on
in a complicated database in a way that increases our understanding of the
people, products, or processes that produced the data in the first place. A good
enough description of a behavior will often suggest an explanation for it as well.
At the very least, a good description suggests where to start looking for an
explanation. The famous gender gap in American politics is an example of
how a simple description, “women support Democrats in greater numbers
than do men,” can provoke large amounts of interest and further study on the
part of journalists, sociologists, economists, and political scientists, not to
mention candidates for public office.
Decision trees (discussed in Chapter 6) are a powerful tool for profiling




Y
customers (or anything else) with respect to a particular target or outcome.




FL
Association rules (discussed in Chapter 9) and clustering (discussed in
Chapter 11) can also be used to build profiles.
AM
Why Now?
TE

Most of the data mining techniques described in this book have existed, at
least as academic algorithms, for years or decades. However, it is only in the
last decade that commercial data mining has caught on in a big way. This is
due to the convergence of several factors:
The data is being produced.

––

The data is being warehoused.

––

Computing power is affordable.

––

Interest in customer relationship management is strong.

––

Commercial data mining software products are readily available.

––


Let™s look at each factor in turn.


Data Is Being Produced
Data mining makes the most sense when there are large volumes of data. In
fact, most data mining algorithms require large amounts of data in order to
build and train the models that will then be used to perform classification, pre­
diction, estimation, or other data mining tasks.
A few industries, including telecommunications and credit cards, have long
had an automated, interactive relationship with customers that generated




Team-Fly®
Why and What Is Data Mining? 13


many transaction records, but it is only relatively recently that the automation
of everyday life has become so pervasive. Today, the rise of supermarket point-
of-sale scanners, automatic teller machines, credit and debit cards, pay-
per-view television, online shopping, electronic funds transfer, automated
order processing, electronic ticketing, and the like means that data is being
produced and collected at unprecedented rates.


Data Is Being Warehoused
Not only is a large amount of data being produced, but also, more and more
often, it is being extracted from the operational billing, reservations, claims
processing, and order entry systems where it is generated and then fed into a
data warehouse to become part of the corporate memory.
Data warehousing brings together data from many different sources in a
common format with consistent definitions for keys and fields. It is generally
not possible (and certainly not advisable) to perform computer- and input/
output (I/O)“intensive data mining operations on an operational system that
the business depends on to survive. In any case, operational systems store data
in a format designed to optimize performance of the operational task. This for­
mat is generally not well suited to decision-support activities like data mining.
The data warehouse, on the other hand, should be designed exclusively for
decision support, which can simplify the job of the data miner.


Computing Power Is Affordable
Data mining algorithms typically require multiple passes over huge quantities
of data. Many are computationally intensive as well. The continuing dramatic
decrease in prices for disk, memory, processing power, and I/O bandwidth
has brought once-costly techniques that were used only in a few government-
funded laboratories into the reach of ordinary businesses.
The successful introduction of parallel relational database management
software by major suppliers such as Oracle, Teradata, and IBM, has brought
the power of parallel processing into many corporate data centers for the first
time. These parallel database server platforms provide an excellent environ­
ment for large-scale data mining.


Interest in Customer Relationship Management Is Strong
Across a wide spectrum of industries, companies have come to realize that
their customers are central to their business and that customer information is
one of their key assets.
14 Chapter 1


Every Business Is a Service Business
For companies in the service sector, information confers competitive advan­
tage. That is why hotel chains record your preference for a nonsmoking room
and car rental companies record your preferred type of car. In addition, com­
panies that have not traditionally thought of themselves as service providers
are beginning to think differently. Does an automobile dealer sell cars or trans­
portation? If the latter, it makes sense for the dealership to offer you a loaner
car whenever your own is in the shop, as many now do.
Even commodity products can be enhanced with service. A home heating
oil company that monitors your usage and delivers oil when you need more,
sells a better product than a company that expects you to remember to call to
arrange a delivery before your tank runs dry and the pipes freeze. Credit card
companies, long-distance providers, airlines, and retailers of all kinds often
compete as much or more on service as on price.

Information Is a Product
Many companies find that the information they have about their customers is
valuable not only to themselves, but to others as well. A supermarket with a
loyalty card program has something that the consumer packaged goods indus­
try would love to have”knowledge about who is buying which products. A
credit card company knows something that airlines would love to know”who
is buying a lot of airplane tickets. Both the supermarket and the credit card
company are in a position to be knowledge brokers or infomediaries. The super­
market can charge consumer packaged goods companies more to print
coupons when the supermarkets can promise higher redemption rates by
printing the right coupons for the right shoppers. The credit card company can
charge the airlines to target a frequent flyer promotion to people who travel a
lot, but fly on other airlines.
Google knows what people are looking for on the Web. It takes advantage of
this knowledge by selling sponsored links. Insurance companies pay to make
sure that someone searching on “car insurance” will be offered a link to their
site. Financial services pay for sponsored links to appear when someone
searches on the phrase “mortgage refinance.”
In fact, any company that collects valuable data is in a position to become an
information broker. The Cedar Rapids Gazette takes advantage of its dominant
position in a 22-county area of Eastern Iowa to offer direct marketing services
to local businesses. The paper uses its own obituary pages and wedding
announcements to keep its marketing database current.
Why and What Is Data Mining? 15


Commercial Data Mining Software Products
Have Become Available
There is always a lag between the time when new algorithms first appear in
academic journals and excite discussion at conferences and the time when
commercial software incorporating those algorithms becomes available. There
is another lag between the initial availability of the first products and the time
that they achieve wide acceptance. For data mining, the period of widespread
availability and acceptance has arrived.
Many of the techniques discussed in this book started out in the fields of
statistics, artificial intelligence, or machine learning. After a few years in uni­
versities and government labs, a new technique starts to be used by a few early
adopters in the commercial sector. At this point in the evolution of a new tech­
nique, the software is typically available in source code to the intrepid user
willing to retrieve it via FTP, compile it, and figure out how to use it by read­
ing the author™s Ph.D. thesis. Only after a few pioneers become successful with
a new technique, does it start to appear in real products that come with user™s
manuals and help lines.
Nowadays, new techniques are being developed; however, much work is
also devoted to extending and improving existing techniques. All the tech­
niques discussed in this book are available in commercial software products,
although there is no single product that incorporates all of them.


How Data Mining Is Being Used Today
This whirlwind tour of a few interesting applications of data mining is
intended to demonstrate the wide applicability of the data mining techniques
discussed in this book. These vignettes are intended to convey something of
the excitement of the field and possibly suggest ways that data mining could
be profitably employed in your own work.


A Supermarket Becomes an Information Broker
Thanks to point-of-sale scanners that record every item purchased and loyalty
card programs that link those purchases to individual customers, supermar­
kets are in a position to notice a lot about their customers these days.
Safeway was one of the first U.S. supermarket chains to take advantage of
this technology to turn itself into an information broker. Safeway purchases
address and demographic data directly from its customers by offering them
discounts in return for using loyalty cards when they make purchases. In order
16 Chapter 1


to obtain the card, shoppers voluntarily divulge personal information of the
sort that makes good input for actionable customer insight.
From then on, each time the shopper presents the discount card, his or her
transaction history is updated in a data warehouse somewhere. With every
trip to the store, shoppers teach the retailer a little more about themselves. The
supermarket itself is probably more interested in aggregate patterns (what
items sell well together, what should be shelved together) than in the behavior
of individual customers. The information gathered on individuals is of great
interest to the manufacturers of the products that line the stores™ aisles.
Of course, the store assures the customers that the information thus collected
will be kept private and it is. Rather than selling Coca-Cola a list of frequent
Pepsi buyers and vice versa, the chain sells access to customers who, based on
their known buying habits and the data they have supplied, are likely prospects
for a particular supplier™s product. Safeway charges several cents per name to
suppliers who want their coupon or special promotional offer to reach just the
right people. Since the coupon redemption also becomes an entry in the shop-
per™s transaction history file, the precise response rate of the targeted group is a
matter of record. Furthermore, a particular customer™s response or lack thereof
to the offer becomes input data for future predictive models.
American Express and other charge card suppliers do much the same thing,
selling advertising space in and on their billing envelopes. The price they can
charge for space in the envelope is directly tied to their ability to correctly iden­
tify people likely to respond to the ad. That is where data mining comes in.


A Recommendation-Based Business
Virgin Wines sells wine directly to consumers in the United Kingdom through
its Web site, www.virginwines.com. New customers are invited to complete a
survey, “the wine wizard,” when they first visit the site. The wine wizard asks
each customer to rate various styles of wines. The ratings are used to create a
profile of the customer™s tastes. During the course of building the profile, the
wine wizard makes some trial recommendations, and the customer has a
chance to agree or disagree with them in order to refine the profile. When the
wine wizard has been completed, the site knows enough about the customer
to start making recommendations.
Over time, the site keeps track of what each customer actually buys and uses
this information to update his or her customer profile. Customers can update
their profiles by redoing the wine wizard at any time. They can also browse
through their own past purchases by clicking on the “my cellar” tab. Any wine
a customer has ever purchased or rated on the site is in the cellar. Customers
may rate or rerate their past purchases at any time, providing still more feed­
back to the recommendation system. With these recommendations, the web
Why and What Is Data Mining? 17


site can offer customers new wines that they should like, emulating the way
that the stores like the Wine Cask have built loyal customer relationships.


Cross-Selling

<<

. 9
( 137 .)



>>