Data Science
Introduction
What Question Does It Ask?
Data science is the process that starts with a question and covers finding, cleaning and analyzing data, then communicating the result so that it leads to a decision. It combines statistics, programming and domain knowledge; data mining and machine learning are its tools.
At a glance
question, data, analysis, story
raw, messy data
report, visualization, model
impact on decisions, accuracy
medium: statistics
moderate to large
Introduction
A Short History
In "The Future of Data Analysis", Tukey argues for data analysis as a science separate from statistics.
Tukey publishes "Exploratory Data Analysis": look first, then model.
Cleveland proposes "data science" as an academic field that extends statistics.
Kaggle is founded; data science competitions and open datasets spread.
Harvard Business Review calls data scientist "the sexiest job of the 21st century".
Project Jupyter splits from IPython; code, plots and text meet in one notebook.
The Course
Core Concepts
What do we want to know, and what decision will follow?
Each row an observation, each column a variable.
Fixing missing, wrong and inconsistent records.
Getting to know data with summaries and plots.
Showing the pattern and result with the right chart.
Checking whether a difference is due to chance.
Comparing two options with a randomized experiment.
Getting the same result again with code, data and versions.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- The data science process and asking good questions
- Reading data in Python or R; tidy data
- Data cleaning: missing values and outliers
- Exploratory data analysis
- Principles of visualization
- Sampling and descriptive statistics
- Confidence intervals and hypothesis tests
- A/B testing and experimental design
- Regression and predictive models
- Correlation, causation, confounders
- Getting data from a database: SQL
- Reproducible reports and telling the story
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
What Drives Grades?
The school asks: “Which habit should we advise students on?” We have survey and grade data for 26 students. Pick variables to see the relationship; click a point to drop it from the analysis.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Probability and Statistics | hypothesis testing, regression, visualization |
| Programming | reading files, loops and functions in Python or R |
| Databases and SQL | querying, joining and grouping data |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you explain the mean, median, standard deviation and the normal distribution? | Probability and Statistics |
| Can you read a table in Python or R and group it by a column? | An intro programming course, then a short pandas or dplyr tutorial |
| Can you write a simple SQL query with SELECT, WHERE and GROUP BY? | The SQL part of a database course, or online SQL exercises |
| Can you tell when a chart is misleading? | Not required, the course teaches it; if curious, Tufte's book |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Probability and Statistics | hypothesis testing, regression, visualization |
| shares methods | Shares methods with Machine Learning | predictive models |
| shares methods | Shares methods with Explainable AI and AI Ethics | data bias, fairness metrics |
| shares methods | Shares methods with Data Mining | exploratory data analysis |
| shares methods | Shares methods with Machine Learning Systems and MLOps | experiment tracking, data versioning |
Differences
Where It Differs (1/2)
| Criterion | Data Science | Data Mining | Machine Learning |
|---|---|---|---|
| Core question | How do we ask the right questions of data and turn the answers into decisions? | How are hidden, useful patterns found in large piles of data? | How does a program generalize from examples? |
| Approach | question, data, analysis, story | discovery-driven analysis | statistical learning |
| Input | raw, messy data | large tables or transaction data | labeled or unlabeled examples |
| Output | report, visualization, model | rules, clusters, outliers | a model that makes predictions |
| Success measure | impact on decisions, accuracy | interestingness, support, confidence | error on unseen data |
| Math load | medium: statistics | medium | high: linear algebra, probability, optimization |
| Data needs | moderate to large | large | moderate to large |
- Data Mining: Data mining focuses on finding patterns in large data; data science covers the whole path from question to decision.
- Machine Learning: ML builds models that predict well on new examples; data science uses a model as one tool for answering a question.
Differences
Where It Differs (2/2)
| Criterion | Data Science | Probability & Statistics | Big Data |
|---|---|---|---|
| Core question | How do we ask the right questions of data and turn the answers into decisions? | How is uncertainty measured, and how do we draw reliable conclusions from limited data? | How is data that does not fit on one machine stored and processed? |
| Approach | question, data, analysis, story | probability models and inference | distributed storage and computation |
| Input | raw, messy data | sample data | huge, fast-flowing data |
| Output | report, visualization, model | estimate, confidence interval, test result | processed data, query results |
| Success measure | impact on decisions, accuracy | unbiasedness, power, significance | scalability, latency, cost |
| Math load | medium: statistics | the theory itself | low |
| Data needs | moderate to large | a sample | very large |
- Probability & Statistics: Statistics provides the theory of inference; data science combines it with programming, messy real data and storytelling.
- Big Data: Big data systems store and process the data; data science draws meaning and decisions from it.
Practice
Applications
Is the new design really better?
Which campaign brings customers?
Tracking outbreaks and disease trends.
Measuring player and team performance.
Evaluating services with open data.
What keeps users, and what drives them away?
Practice
Risks and Ethics
Reading causes into correlations leads to wrong decisions: ice cream sales and drownings rise together because both happen in summer.
Run enough tests and something will come out "significant"; the hypothesis must be fixed before looking at the data.
In data with names removed, ZIP code, birth date and sex are enough to re-identify most people.
A truncated axis or a cherry-picked time window can tell the opposite story with the same data.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Data science is training models. | Most of the time goes into sharpening the question, finding and cleaning data, and communicating results. |
| Data speaks for itself. | Without a question, context and domain knowledge, the same data can lead to wrong conclusions. |
| A significant p-value is an important result. | Statistical significance says nothing about effect size; many tests produce misleading results. |
| A chart is decoration for the result. | Anscombe's quartet shows it: data with identical summaries look completely different when plotted. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Wickham, Çetinkaya-Rundel and Grolemund, R for Data Science, 2nd ed., O'Reilly, 2023.
- McKinney, Python for Data Analysis, 3rd ed., O'Reilly, 2022.
- James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning, 2nd ed., Springer, 2021.
- Tukey, Exploratory Data Analysis, Addison-Wesley, 1977.