Generative Models
Introduction
What Question Does It Ask?
Generative models learn the probability distribution of data and generate new samples from it. A discriminative model asks "is this image a cat?", while a generative model asks "how are cat images distributed, and how do I draw a new one?"
At a glance
learning the data distribution
huge, mostly unlabeled data
new samples: images, text, audio
realism and diversity (FID etc.)
high
very large
Introduction
A Short History
Kingma and Welling propose the variational autoencoder (VAE).
Goodfellow and colleagues introduce generative adversarial networks (GANs), pitting a generator against a discriminator.
Sohl-Dickstein and colleagues publish the idea of diffusion models that undo noise step by step.
OpenAI's first GPT: an autoregressive transformer pretrained on a large text corpus.
Ho, Jain and Abbeel's DDPM brings diffusion models to the front of image generation.
Stable Diffusion is released with open weights; ChatGPT brings generative models to a wide audience.
The Course
Core Concepts
The probability the model assigns to real data; training increases it.
Low-dimensional hidden variables that summarize samples.
Encode, decode; a VAE forces the latent space into a regular distribution.
A generator makes fake samples, a discriminator tries to tell them from real ones.
Adds noise to data; a network learns to remove it step by step.
Predicts each token from the previous ones: GPT.
Generation guided by text, a class or a sketch.
FID, perplexity and human evaluation.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Generative modelling; likelihood and KL divergence
- Autoregressive models: PixelCNN, transformers
- Variational autoencoders (VAE) and the ELBO
- Normalizing flows
- Generative adversarial networks and training stability
- Energy-based models
- Score matching and diffusion models
- Fast sampling and latent diffusion
- Conditional generation, classifier-free guidance
- Large language models: pretraining and fine-tuning
- Multimodal models: text to image
- Evaluation: FID, perplexity, human judgement
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
New Faces from Latent Space
A game studio wants new characters from 60 cartoon faces. The model learned to compress each face into two numbers (z₁, z₂); generate a new face with the sliders or by clicking the plane on the left.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Deep Learning | GANs, VAEs, diffusion, large language models |
| Information Theory | KL divergence, likelihood |
| Calculus | working comfortably with integrals, derivatives and probability densities |
| Programming | Python and a deep learning library (PyTorch) |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you build and train a neural network in PyTorch? | Deep Learning |
| Can you explain probability densities, expectation and Bayes' rule? | Probability and Statistics |
| Can you explain how backpropagation uses the chain rule to get gradients? | Artificial Neural Networks |
| Can you compute an integral and the derivative of a logarithm? | Integrals and derivatives from calculus |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| subfield | This course is a subfield of Deep Learning | GANs, VAEs, diffusion, large language models |
| builds on | This course builds on Information Theory | KL divergence, likelihood |
| shares methods | Shares methods with Natural Language Processing and Text Mining | text generation |
| shares methods | Shares methods with Probabilistic Graphical Models | latent variable models, variational inference |
| shares methods | Shares methods with Computer Graphics | image and scene synthesis |
Differences
Where It Differs (1/2)
| Criterion | Generative Models | Deep Learning | Graphical Models |
|---|---|---|---|
| Core question | How does a model generate new samples that look like real data? | How do many-layered networks learn useful representations from raw data? | How do we model and query many interdependent, uncertain variables? |
| Approach | learning the data distribution | multilayer representation learning | probability and graphs combined |
| Input | huge, mostly unlabeled data | raw data: images, text, audio | variables, dependencies, data |
| Output | new samples: images, text, audio | learned representation and prediction | conditional probability, best explanation |
| Success measure | realism and diversity (FID etc.) | accuracy, loss, scaling | likelihood, inference accuracy |
| Math load | high | high: derivatives, matrices, probability | high: probability |
| Data needs | very large | large; needs GPUs | moderate |
- Deep Learning: Deep learning is the general toolkit and mostly makes predictions; this course uses those tools to generate new samples from a data distribution.
- Graphical Models: Graphical models build the structure between variables explicitly; the deep generative models here usually learn it implicitly.
Differences
Where It Differs (2/2)
| Criterion | Generative Models | NLP | Computer Graphics |
|---|---|---|---|
| Core question | How does a model generate new samples that look like real data? | How does a computer analyze, understand and generate human language? | How is a realistic image produced from the description of a scene? |
| Approach | learning the data distribution | linguistics, statistics and deep learning | geometry and light simulation |
| Input | huge, mostly unlabeled data | text | scene: models, lights, camera |
| Output | new samples: images, text, audio | label, translation, summary, answer | image or animation |
| Success measure | realism and diversity (FID etc.) | accuracy, BLEU, human evaluation | realism, frame rate |
| Math load | high | medium | high: linear algebra, geometry |
| Data needs | very large | large text corpora | none |
- NLP: NLP also covers parsing and understanding language; this course looks at the shared principles of generating text, images and audio.
- Computer Graphics: Computer graphics computes images from a physical description of a scene; generative models produce images by learning from data.
Practice
Applications
Pictures and illustrations from a prompt.
Text generation with large language models.
Natural-sounding speech and music.
Synthetic samples for classes with little data.
Proposing new candidate molecules.
Inpainting missing regions, super-resolution.
Practice
Risks and Ethics
A person's face and voice are imitated without consent and used for fraud, blackmail and election disinformation.
Web-scraped datasets such as LAION-5B contain works without their owners' permission; copyright lawsuits are ongoing.
Models retrained on generated data lose the tails of the distribution over generations, and their outputs grow uniform.
Metrics such as FID do not always agree with human judgement; a model tuned to the metric can look worse to the eye.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| A generative model copies its training data. | The goal is to learn the distribution; memorization can still happen and must be checked. |
| Generative AI is just chatbots. | Models that generate images, audio, video, molecules and code are also part of this course. |
| Realistic-looking output is correct output. | A model can produce plausible but wrong content; correctness is checked separately. |
| GANs are outdated; diffusion is always better. | Each family trades off speed, quality and diversity differently; a GAN generates in one step. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Foster, Generative Deep Learning, 2nd ed., O'Reilly, 2023.
- Tomczak, Deep Generative Modeling, Springer, 2022.
- Goodfellow et al., "Generative Adversarial Nets", Advances in Neural Information Processing Systems 27, 2014.
- Kingma and Welling, "Auto-Encoding Variational Bayes", ICLR, 2014.
- Ho, Jain and Abbeel, "Denoising Diffusion Probabilistic Models", Advances in Neural Information Processing Systems 33, 2020.