Deep Learning
Introduction
What Question Does It Ask?
Deep learning studies how multilayer neural networks learn useful intermediate representations from raw data (pixels, audio, text) on their own, layer by layer. The key idea is representation learning instead of hand-designed features; big data, GPUs and good architectures make it possible.
At a glance
multilayer representation learning
raw data: images, text, audio
learned representation and prediction
accuracy, loss, scaling
high: derivatives, matrices, probability
large; needs GPUs
Introduction
A Short History
Hochreiter and Schmidhuber propose the LSTM: a cure for vanishing gradients on long sequences.
LeCun et al.'s LeNet-5 convolutional network recognizes handwritten digits.
Hinton et al. train deep belief networks with layer-wise pretraining; interest revives.
AlexNet, a convolutional network trained on GPUs, wins ImageNet by a wide margin.
He et al.'s residual networks (ResNet) win ImageNet with 152 layers.
Vaswani et al. introduce the transformer architecture: attention only.
The Course
Core Concepts
Layers learn ever more abstract features, from edges to objects.
Small filters slide over the image; weights are shared.
Shrinks spatial size, adds robustness to small shifts.
RNN and LSTM: process a sequence step by step, carrying a state.
Each element weighs which other elements of the sequence to look at.
A path that skips a layer; makes very deep networks trainable.
Batch and layer normalization stabilize training.
Adapting a model trained on big data to a new task.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Deep feedforward networks and computational graphs
- Optimization: SGD, momentum, Adam
- Regularization: dropout, weight decay, augmentation
- Batch and layer normalization
- Convolutional networks: convolution and pooling
- CNN architectures: LeNet, AlexNet, VGG, ResNet
- Transfer learning and fine-tuning
- Recurrent networks: RNN, LSTM, GRU
- Sequence-to-sequence models and attention
- Transformers and self-attention
- Autoencoders and self-supervised learning
- Interpreting deep networks: saliency maps
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
How Do Filters See a Line?
A school app will recognize signs drawn by children. Click the left grid to draw; two convolution filters scan the picture, and class scores come out of the feature maps.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Artificial Neural Networks | multilayer networks, backpropagation, activation functions |
| GPU Programming and Parallel Computing | parallel training of large networks, tensor operations |
| Calculus | multivariable derivatives and the chain rule |
| Programming | comfort with Python, NumPy and a deep learning library |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you explain backpropagation step by step in a small network? | Artificial Neural Networks |
| Can you compute the gradient of a multivariable function? | Partial derivatives and the chain rule from calculus |
| Can you say how shapes must match in a matrix product? | Linear Algebra |
| Can you write a class in Python and work with NumPy arrays? | An intro programming course, then a short NumPy tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses (1/2)
| Relation | Course | Shared ground |
|---|---|---|
| subfield | This course is a subfield of Artificial Neural Networks | multilayer networks, backpropagation, activation functions |
| subfield | Generative Models is a subfield of this course | GANs, VAEs, diffusion, large language models |
| builds on | This course builds on GPU Programming and Parallel Computing | parallel training of large networks, tensor operations |
| application | Embedded AI applies the methods of this course | quantization, pruning, small models |
| shares methods | Shares methods with Natural Language Processing and Text Mining | word embeddings, transformer architecture |
| shares methods | Shares methods with Reinforcement Learning | deep Q-networks, deep policy networks |
| shares methods | Shares methods with Computer Vision | convolutional neural networks, vision transformers |
| shares methods | Shares methods with Speech Recognition and Audio Processing | end-to-end speech recognition |
Overlap
Overlapping Courses (2/2)
| Relation | Course | Shared ground |
|---|---|---|
| shares methods | Shares methods with Time Series Analysis and Forecasting | recurrent networks, attention-based forecasting |
Differences
Where It Differs (1/2)
| Criterion | Deep Learning | Neural Networks | Machine Learning |
|---|---|---|---|
| Core question | How do many-layered networks learn useful representations from raw data? | How do networks of simple units learn by adjusting their weights? | How does a program generalize from examples? |
| Approach | multilayer representation learning | network of simple weighted units | statistical learning |
| Input | raw data: images, text, audio | numeric feature vector | labeled or unlabeled examples |
| Output | learned representation and prediction | class or numeric prediction | a model that makes predictions |
| Success measure | accuracy, loss, scaling | loss and accuracy | error on unseen data |
| Math load | high: derivatives, matrices, probability | medium-high: derivatives, matrices | high: linear algebra, probability, optimization |
| Data needs | large; needs GPUs | moderate | moderate to large |
- Neural Networks: ANN builds neurons, loss and backpropagation on small networks; this course covers deep architectures and the practice of training them.
- Machine Learning: Classic ML works with hand-designed features; deep learning learns features from raw data by itself.
Differences
Where It Differs (2/2)
| Criterion | Deep Learning | Computer Vision | Generative Models |
|---|---|---|---|
| Core question | How do many-layered networks learn useful representations from raw data? | How does a computer understand what is in an image or a video? | How does a model generate new samples that look like real data? |
| Approach | multilayer representation learning | extracting meaning from images | learning the data distribution |
| Input | raw data: images, text, audio | image or video | huge, mostly unlabeled data |
| Output | learned representation and prediction | label, box, depth, 3D structure | new samples: images, text, audio |
| Success measure | accuracy, loss, scaling | accuracy, mAP, IoU | realism and diversity (FID etc.) |
| Math load | high: derivatives, matrices, probability | high: geometry, linear algebra | high |
| Data needs | large; needs GPUs | many labeled images | very large |
- Computer Vision: Computer vision is the problem of understanding images; deep learning is a toolkit shared by vision, language and speech.
- Generative Models: Generative models aim to sample from the data distribution; this course mostly focuses on discriminative, predictive networks.
Practice
Applications
Recognizing an object or a disease in a photo.
Turning an audio recording into text.
Transformer-based translation systems.
Finding pedestrians and vehicles for a self-driving car.
Predicting 3D structure from an amino acid sequence.
Atari and Go with deep reinforcement learning.
Practice
Risks and Ethics
Training large models takes thousands of GPU hours and real carbon emissions, concentrating research in a few large organizations.
Imperceptible pixel changes fool a classifier; researchers showed the same with stickers placed on a stop sign.
Web-scraped datasets such as ImageNet can contain photos taken without consent and offensive labels; ImageNet later removed many of them.
Hyperparameters tuned on the test set and results reported from a single seed make small differences look like real gains.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Deep learning is best for every problem. | On small tabular data, gradient-boosted trees are often better and cheaper. |
| A deep network is a black box that cannot be inspected. | Tools such as gradient-based saliency maps and attention weights give partial explanations. |
| Deep learning was invented in 2012. | The ideas are decades old; big data and GPUs brought the 2012 leap. |
| More layers always mean better results. | Without residual connections and normalization, very deep networks may not train. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Goodfellow, Bengio and Courville, Deep Learning, MIT Press, 2016.
- Zhang, Lipton, Li and Smola, Dive into Deep Learning, Cambridge University Press, 2023.
- Prince, Understanding Deep Learning, MIT Press, 2023.
- Chollet, Deep Learning with Python, 2nd ed., Manning, 2021.
- LeCun, Bengio and Hinton, "Deep learning", Nature 521, 2015.