Machine Learning
Introduction
What Question Does It Ask?
Machine learning studies algorithms that learn from examples without being explicitly programmed and generalize what they learn to new examples. In Mitchell's definition, a program learns if its performance on task T, measured by P, improves with experience E.
At a glance
statistical learning
labeled or unlabeled examples
a model that makes predictions
error on unseen data
high: linear algebra, probability, optimization
moderate to large
Introduction
A Short History
Rosenblatt introduces the perceptron, one of the first models to tune its weights from examples.
Samuel's checkers program improves by playing itself; the term "machine learning" spreads.
Quinlan publishes the ID3 decision tree; the same year backpropagation revives neural networks.
Cortes and Vapnik publish support vector machines (SVM).
Breiman introduces random forests: a vote among many trees.
AlexNet wins the ImageNet challenge by a wide margin; the deep learning era begins.
The Course
Core Concepts
Classification and regression from labeled examples.
Clustering and dimensionality reduction on unlabeled data.
The form and parameters of the function to be learned.
Measures how wrong a prediction is; training minimizes it.
Correct predictions on examples not seen in training.
Memorizing the training data; the bias-variance tradeoff.
Training, validation and test sets; cross-validation.
Turning raw data into numbers a model can use.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- The learning problem; supervised and unsupervised learning
- Linear regression and gradient descent
- Logistic regression and classification
- Evaluation: train/test split, cross-validation
- Overfitting, regularization, bias-variance
- Decision trees
- Ensembles: random forests, gradient boosting
- Support vector machines and kernels
- k-nearest neighbours and naive Bayes
- Clustering: k-means, hierarchical clustering
- Dimensionality reduction: PCA
- Feature engineering and model selection
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Should We Grant the Loan?
The bank knows the income, debt ratio and repayment outcome of past applications. Click the chart to add a new application; the model decides by looking at the k most similar past ones.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Artificial Intelligence | an agent that learns from data; statistical generalization instead of search |
| Linear Algebra | feature vectors, matrix multiplication, PCA |
| Probability and Statistics | generalization error, bias-variance, probabilistic models |
| Optimization | loss minimization, regularization |
| Programming | reading data, loops and functions in Python or Java |
| Calculus | derivatives, partial derivatives and the chain rule: the language of gradient descent |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you take a derivative and a partial derivative? | Derivatives and the chain rule from calculus |
| Can you multiply matrices and take a dot product? | Linear Algebra |
| Can you explain mean, variance and conditional probability? | Probability and Statistics |
| Can you write loops and functions in Python and read a CSV file? | An intro programming course, then a short pandas tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses (1/2)
| Relation | Course | Shared ground |
|---|---|---|
| subfield | This course is a subfield of Artificial Intelligence | an agent that learns from data; statistical generalization instead of search |
| subfield | Artificial Neural Networks is a subfield of this course | parametric model, loss function, gradient descent |
| subfield | Reinforcement Learning is a subfield of this course | learning from rewards instead of labels |
| builds on | This course builds on Linear Algebra | feature vectors, matrix multiplication, PCA |
| builds on | This course builds on Probability and Statistics | generalization error, bias-variance, probabilistic models |
| builds on | This course builds on Optimization | loss minimization, regularization |
| application | Machine Learning Systems and MLOps applies the methods of this course | deploying, monitoring and retraining models |
| application | AI in Cybersecurity applies the methods of this course | intrusion detection, malware classification |
Overlap
Overlapping Courses (2/2)
| Relation | Course | Shared ground |
|---|---|---|
| application | Bioinformatics applies the methods of this course | gene expression classification, protein structure |
| shares methods | Shares methods with Pattern Recognition | classifiers, decision boundaries |
| shares methods | Shares methods with Probabilistic Graphical Models | probabilistic classifiers, EM algorithm |
| shares methods | Shares methods with Explainable AI and AI Ethics | model explanation, feature importance |
| shares methods | Shares methods with Data Mining | same algorithms; the goal is discovery, not prediction |
| shares methods | Shares methods with Data Science | predictive models |
| shares methods | Shares methods with Information Theory | cross-entropy loss, decision tree split criterion |
Differences
Where It Differs (1/2)
| Criterion | Machine Learning | Artificial Intelligence | Data Mining |
|---|---|---|---|
| Core question | How does a program generalize from examples? | How does a machine act intelligently by searching, reasoning and planning? | How are hidden, useful patterns found in large piles of data? |
| Approach | statistical learning | search, logic, probability and learning combined | discovery-driven analysis |
| Input | labeled or unlabeled examples | problem definition: states, actions, goal | large tables or transaction data |
| Output | a model that makes predictions | action sequence, plan or decision | rules, clusters, outliers |
| Success measure | error on unseen data | reaching the goal, cost, rationality | interestingness, support, confidence |
| Math load | high: linear algebra, probability, optimization | medium: logic, probability, graphs | medium |
| Data needs | moderate to large | little; often a model and knowledge suffice | large |
- Artificial Intelligence: AI is the framework; ML focuses only on learning from data within it.
- Data Mining: They share algorithms; ML aims to predict new examples, data mining to discover patterns in data.
Differences
Where It Differs (2/2)
| Criterion | Machine Learning | Probability & Statistics | Deep Learning |
|---|---|---|---|
| Core question | How does a program generalize from examples? | How is uncertainty measured, and how do we draw reliable conclusions from limited data? | How do many-layered networks learn useful representations from raw data? |
| Approach | statistical learning | probability models and inference | multilayer representation learning |
| Input | labeled or unlabeled examples | sample data | raw data: images, text, audio |
| Output | a model that makes predictions | estimate, confidence interval, test result | learned representation and prediction |
| Success measure | error on unseen data | unbiasedness, power, significance | accuracy, loss, scaling |
| Math load | high: linear algebra, probability, optimization | the theory itself | high: derivatives, matrices, probability |
| Data needs | moderate to large | a sample | large; needs GPUs |
- Probability & Statistics: Statistics emphasizes a model's meaning and uncertainty, ML its predictive performance.
- Deep Learning: Deep learning is a branch of ML; it learns features from raw data instead of hand-designing them.
Practice
Applications
Classifying emails by their content.
Predicting an applicant's risk of default.
Products or content a user is likely to enjoy.
Estimating disease risk from patient data.
Sales and inventory planning.
Catching unusual card transactions.
Practice
Risks and Ethics
Discrimination in historical data passes straight into the model: hiring or credit models can systematically exclude some groups.
When test information leaks into training, a model looks flawless on paper and fails in the field.
Personal information in the training data can be extracted from the model; GDPR and similar laws limit how data may be used.
High accuracy collapses when the data distribution shifts, and the model looks just as confident on cases it does not know.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| A more complex model is always better. | A complex model can memorize the training data (overfitting) and do worse on new data. |
| Training accuracy shows success. | Success is measured on test data never used in training. |
| More data solves every problem. | More biased or noisy data only multiplies the errors. |
| The model learns cause and effect. | ML finds correlations; causation needs experiments or causal inference. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Mitchell, Machine Learning, McGraw-Hill, 1997.
- Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
- Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
- Alpaydın, Introduction to Machine Learning, 4th ed., MIT Press, 2020.
- Géron, Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow, 3rd ed., O'Reilly, 2022.