Deep Learning

Guide to AI and Data Courses · Part I: Core
Representation learningConvolutional networksRecurrent networksTransformersTraining techniquesNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How do many-layered networks learn useful representations from raw data?

Deep learning studies how multilayer neural networks learn useful intermediate representations from raw data (pixels, audio, text) on their own, layer by layer. The key idea is representation learning instead of hand-designed features; big data, GPUs and good architectures make it possible.

At a glance

Approach

multilayer representation learning

Input

raw data: images, text, audio

Output

learned representation and prediction

Success measure

accuracy, loss, scaling

Math load

high: derivatives, matrices, probability

Data needs

large; needs GPUs

Introduction

A Short History

1997

Hochreiter and Schmidhuber propose the LSTM: a cure for vanishing gradients on long sequences.

1998

LeCun et al.'s LeNet-5 convolutional network recognizes handwritten digits.

2006

Hinton et al. train deep belief networks with layer-wise pretraining; interest revives.

2012

AlexNet, a convolutional network trained on GPUs, wins ImageNet by a wide margin.

2015

He et al.'s residual networks (ResNet) win ImageNet with 152 layers.

2017

Vaswani et al. introduce the transformer architecture: attention only.

The Course

Core Concepts

Representation learning

Layers learn ever more abstract features, from edges to objects.

Convolution

Small filters slide over the image; weights are shared.

Pooling

Shrinks spatial size, adds robustness to small shifts.

Recurrent network

RNN and LSTM: process a sequence step by step, carrying a state.

Attention

Each element weighs which other elements of the sequence to look at.

Residual connection

A path that skips a layer; makes very deep networks trainable.

Normalization

Batch and layer normalization stabilize training.

Transfer learning

Adapting a model trained on big data to a new task.

The Course

A Typical Workflow

Raw datapixels, audio,textChoosearchitectureCNN, RNN,transformerTrain onGPUmini-batch,AdamRegularizedropout,augmentationFine-tunetransferlearning
Example: An app that recognizes plant diseases collects leaf photos, takes a ResNet pretrained on ImageNet, replaces its last layer with the disease classes and fine-tunes it on a GPU with data augmentation. No feature is defined by hand.

The Course

What Does the Course Cover?

  1. Deep feedforward networks and computational graphs
  2. Optimization: SGD, momentum, Adam
  3. Regularization: dropout, weight decay, augmentation
  4. Batch and layer normalization
  5. Convolutional networks: convolution and pooling
  6. CNN architectures: LeNet, AlexNet, VGG, ResNet
  7. Transfer learning and fine-tuning
  8. Recurrent networks: RNN, LSTM, GRU
  9. Sequence-to-sequence models and attention
  10. Transformers and self-attention
  11. Autoencoders and self-supervised learning
  12. Interpreting deep networks: saliency maps

Skills you will gain

building and training models in PyTorchGPU training and mini-batch tuningfine-tuning pretrained modelsmonitoring training with TensorBoardusing Hugging Face modelsdesigning hyperparameter experiments

A typical semester; the order and coverage vary between universities.

Try It

How Do Filters See a Line?

A school app will recognize signs drawn by children. Click the left grid to draw; two convolution filters scan the picture, and class scores come out of the feature maps.

Draw a diagonal line: both filters barely respond. The network needs other filters for other patterns.
The nine numbers in each filter were written by hand; a deep network learns them from data and, by stacking layers, climbs from edges to shapes and from shapes to objects.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Artificial Neural Networksmultilayer networks, backpropagation, activation functions
GPU Programming and Parallel Computingparallel training of large networks, tensor operations
Calculusmultivariable derivatives and the chain rule
Programmingcomfort with Python, NumPy and a deep learning library

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you explain backpropagation step by step in a small network?Artificial Neural Networks
Can you compute the gradient of a multivariable function?Partial derivatives and the chain rule from calculus
Can you say how shapes must match in a matrix product?Linear Algebra
Can you write a class in Python and work with NumPy arrays?An intro programming course, then a short NumPy tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses (1/2)

RelationCourseShared ground
subfieldThis course is a subfield of Artificial Neural Networksmultilayer networks, backpropagation, activation functions
subfieldGenerative Models is a subfield of this courseGANs, VAEs, diffusion, large language models
builds onThis course builds on GPU Programming and Parallel Computingparallel training of large networks, tensor operations
applicationEmbedded AI applies the methods of this coursequantization, pruning, small models
shares methodsShares methods with Natural Language Processing and Text Miningword embeddings, transformer architecture
shares methodsShares methods with Reinforcement Learningdeep Q-networks, deep policy networks
shares methodsShares methods with Computer Visionconvolutional neural networks, vision transformers
shares methodsShares methods with Speech Recognition and Audio Processingend-to-end speech recognition

Overlap

Overlapping Courses (2/2)

RelationCourseShared ground
shares methodsShares methods with Time Series Analysis and Forecastingrecurrent networks, attention-based forecasting

Differences

Where It Differs (1/2)

CriterionDeep LearningNeural NetworksMachine Learning
Core questionHow do many-layered networks learn useful representations from raw data?How do networks of simple units learn by adjusting their weights?How does a program generalize from examples?
Approachmultilayer representation learningnetwork of simple weighted unitsstatistical learning
Inputraw data: images, text, audionumeric feature vectorlabeled or unlabeled examples
Outputlearned representation and predictionclass or numeric predictiona model that makes predictions
Success measureaccuracy, loss, scalingloss and accuracyerror on unseen data
Math loadhigh: derivatives, matrices, probabilitymedium-high: derivatives, matriceshigh: linear algebra, probability, optimization
Data needslarge; needs GPUsmoderatemoderate to large
  • Neural Networks: ANN builds neurons, loss and backpropagation on small networks; this course covers deep architectures and the practice of training them.
  • Machine Learning: Classic ML works with hand-designed features; deep learning learns features from raw data by itself.

Differences

Where It Differs (2/2)

CriterionDeep LearningComputer VisionGenerative Models
Core questionHow do many-layered networks learn useful representations from raw data?How does a computer understand what is in an image or a video?How does a model generate new samples that look like real data?
Approachmultilayer representation learningextracting meaning from imageslearning the data distribution
Inputraw data: images, text, audioimage or videohuge, mostly unlabeled data
Outputlearned representation and predictionlabel, box, depth, 3D structurenew samples: images, text, audio
Success measureaccuracy, loss, scalingaccuracy, mAP, IoUrealism and diversity (FID etc.)
Math loadhigh: derivatives, matrices, probabilityhigh: geometry, linear algebrahigh
Data needslarge; needs GPUsmany labeled imagesvery large
  • Computer Vision: Computer vision is the problem of understanding images; deep learning is a toolkit shared by vision, language and speech.
  • Generative Models: Generative models aim to sample from the data distribution; this course mostly focuses on discriminative, predictive networks.

Practice

Applications

Image classification

Recognizing an object or a disease in a photo.

Speech recognition

Turning an audio recording into text.

Machine translation

Transformer-based translation systems.

Object detection

Finding pedestrians and vehicles for a self-driving car.

Protein structure

Predicting 3D structure from an amino acid sequence.

Game playing

Atari and Go with deep reinforcement learning.

Practice

Risks and Ethics

Compute and energy cost

Training large models takes thousands of GPU hours and real carbon emissions, concentrating research in a few large organizations.

Adversarial examples

Imperceptible pixel changes fool a classifier; researchers showed the same with stickers placed on a stop sign.

Data scraped from the web

Web-scraped datasets such as ImageNet can contain photos taken without consent and offensive labels; ImageNet later removed many of them.

Illusory progress

Hyperparameters tuned on the test set and results reported from a single seed make small differences look like real gains.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

PyTorchTensorFlowKerasJAXHugging FaceCUDATensorBoard

Well-known datasets

ImageNetCIFAR-10/100MNISTCOCOLibriSpeechWikipedia text

Practice

Common Misconceptions

Common beliefIn fact
Deep learning is best for every problem.On small tabular data, gradient-boosted trees are often better and cheaper.
A deep network is a black box that cannot be inspected.Tools such as gradient-based saliency maps and attention weights give partial explanations.
Deep learning was invented in 2012.The ideas are decades old; big data and GPUs brought the 2012 leap.
More layers always mean better results.Without residual connections and normalization, very deep networks may not train.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

Deep learning engineerComputer vision engineerResearcherML infrastructure engineerApplied scientist
Deep learning is the most widely used tool on the map today: computer vision, natural language processing, speech recognition, generative models and deep reinforcement learning are built on it. On the systems side, GPU programming, embedded AI and MLOps teach how to speed these models up and take them to production.

Next Steps

Further Reading

  • Goodfellow, Bengio and Courville, Deep Learning, MIT Press, 2016.
  • Zhang, Lipton, Li and Smola, Dive into Deep Learning, Cambridge University Press, 2023.
  • Prince, Understanding Deep Learning, MIT Press, 2023.
  • Chollet, Deep Learning with Python, 2nd ed., Manning, 2021.
  • LeCun, Bengio and Hinton, "Deep learning", Nature 521, 2015.

The End

Chapter 5: Deep Learning
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.