Data Mining
Introduction
What Question Does It Ask?
Data mining studies how to discover patterns in large data collections that are previously unknown, valid and useful. It uses the same algorithms as machine learning, but the goal is not prediction on new examples; it is finding rules, clusters and anomalies in the data itself.
At a glance
discovery-driven analysis
large tables or transaction data
rules, clusters, outliers
interestingness, support, confidence
medium
large
Introduction
A Short History
MacQueen publishes the clustering method he names "k-means".
The first workshop on knowledge discovery in databases (KDD) is held; the field gets its name.
Agrawal and Srikant popularize association rule mining with the Apriori algorithm.
Fayyad and colleagues define the KDD process; density-based DBSCAN is published the same year.
CRISP-DM 1.0 is released: an industry-neutral process model for data mining projects.
The Netflix Prize opens: one million dollars for improving recommendation accuracy by 10 percent.
The Course
Core Concepts
Selection, preprocessing, transformation, mining, interpretation.
Products often bought together: Apriori, FP-growth.
If X then Y; measured by support, confidence and lift.
k-means, hierarchical, DBSCAN; unlabeled groups.
A record unlike the rest: an error or fraud?
Readable, explanatory models such as decision trees.
Missing values, noise, discretization, normalization.
Event sequences that recur in time order.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- The data mining process: KDD and CRISP-DM
- Getting to know data: attribute types, similarity
- Preprocessing: cleaning, integration, reduction
- Data warehouses and OLAP cubes
- Frequent itemsets: Apriori and FP-growth
- Association rules and interestingness measures
- Classification: decision trees, rule-based methods
- Evaluating classifiers and imbalanced classes
- Clustering: k-means and hierarchical methods
- Density-based clustering and cluster validity
- Outlier detection
- Sequential pattern, text and graph mining
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Hidden Rules in the Shopping Basket
A corner shop looks at 14 receipts: nobody asked a question; let the data tell which items are bought together. Change the thresholds and Apriori finds the rules again; click a rule to see it in the baskets.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Probability and Statistics | support, confidence, significance |
| Databases and SQL | tables, joins and grouping: where the data comes from |
| Data Structures | hash tables and trees: building blocks of algorithms such as FP-growth |
| Algorithms | complexity: understanding the exponential growth of candidate sets |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you join and group two tables in one SQL query? | SQL from a database course: JOIN and GROUP BY |
| Can you explain conditional probability and independence with an example? | Probability and Statistics |
| Can you explain when a hash table beats a tree? | A data structures course: hash tables and trees |
| Can you read a CSV file in Python and count its missing values? | An intro programming course, then a short pandas tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Probability and Statistics | support, confidence, significance |
| application | Bioinformatics applies the methods of this course | sequence motifs, clustering |
| shares methods | Shares methods with Machine Learning | same algorithms; the goal is discovery, not prediction |
| shares methods | Shares methods with Natural Language Processing and Text Mining | text classification, topic modeling |
| shares methods | Shares methods with Data Science | exploratory data analysis |
| shares methods | Shares methods with Big Data Systems | scalable mining |
| shares methods | Shares methods with Information Retrieval and Recommender Systems | association, collaborative filtering |
| shares methods | Shares methods with Social Network Analysis | community detection |
Differences
Where It Differs (1/2)
| Criterion | Data Mining | Machine Learning | Data Science |
|---|---|---|---|
| Core question | How are hidden, useful patterns found in large piles of data? | How does a program generalize from examples? | How do we ask the right questions of data and turn the answers into decisions? |
| Approach | discovery-driven analysis | statistical learning | question, data, analysis, story |
| Input | large tables or transaction data | labeled or unlabeled examples | raw, messy data |
| Output | rules, clusters, outliers | a model that makes predictions | report, visualization, model |
| Success measure | interestingness, support, confidence | error on unseen data | impact on decisions, accuracy |
| Math load | medium | high: linear algebra, probability, optimization | medium: statistics |
| Data needs | large | moderate to large | moderate to large |
- Machine Learning: They share algorithms; ML aims at prediction on new examples, data mining at discovery in the data at hand.
- Data Science: Data science covers the whole path from question to decision; data mining focuses on the pattern-finding step.
Differences
Where It Differs (2/2)
| Criterion | Data Mining | Big Data | Probability & Statistics |
|---|---|---|---|
| Core question | How are hidden, useful patterns found in large piles of data? | How is data that does not fit on one machine stored and processed? | How is uncertainty measured, and how do we draw reliable conclusions from limited data? |
| Approach | discovery-driven analysis | distributed storage and computation | probability models and inference |
| Input | large tables or transaction data | huge, fast-flowing data | sample data |
| Output | rules, clusters, outliers | processed data, query results | estimate, confidence interval, test result |
| Success measure | interestingness, support, confidence | scalability, latency, cost | unbiasedness, power, significance |
| Math load | medium | low | the theory itself |
| Data needs | large | very large | a sample |
- Big Data: Big data systems store and process data in a distributed way; data mining asks which patterns to look for in it.
- Probability & Statistics: Statistics tests a hypothesis set in advance; data mining generates hypotheses from data, which must then be tested.
Practice
Applications
Items bought together, shelves and promotions.
Grouping similar customers.
Flagging unusual transactions.
Visitors' navigation patterns.
Error codes that occur together.
Diagnoses and drugs that occur together.
Practice
Risks and Ethics
Pregnancy, illness or money trouble can be inferred from a shopping basket, even if the person never shared it.
Netflix Prize data was linked to IMDb ratings to identify some users; the planned second contest was cancelled over it.
Search for enough patterns and chance ones look significant; multiple comparison corrections and testing on fresh data are needed.
Customer segments built from postcodes or shopping patterns can indirectly give some groups different prices and service.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Data mining is collecting data. | The name misleads: what is mined is knowledge, not data; the data already exists. |
| Every rule found is useful. | Thousands of rules come out; most are obvious or accidental. Interestingness measures and expert review are needed. |
| A strong association means causation. | Occurring together does not mean one causes the other; there may be a common cause. |
| Clustering has one right answer. | The result depends on the number of clusters, distance and scaling; different choices, different clusters. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Han, Kamber and Pei, Data Mining: Concepts and Techniques, 3rd ed., Morgan Kaufmann, 2011.
- Tan, Steinbach, Karpatne and Kumar, Introduction to Data Mining, 2nd ed., Pearson, 2019.
- Witten, Frank, Hall and Pal, Data Mining: Practical Machine Learning Tools and Techniques, 4th ed., Morgan Kaufmann, 2017.
- Leskovec, Rajaraman and Ullman, Mining of Massive Datasets, 3rd ed., Cambridge University Press, 2020.