GPU Programming and Parallel Computing
Introduction
What Question Does It Ask?
GPU programming studies data-parallel computation, which applies the same operation to thousands of data elements at once. Using a model such as CUDA, the course teaches how to organize threads, use the memory hierarchy efficiently and measure speedup; this is also the hardware that makes deep learning practical.
At a glance
data-parallel computation
parallelizable algorithm
accelerated program
speedup, memory bandwidth
low-medium
none
Introduction
A Short History
Flynn classifies computers as SISD, SIMD, MISD and MIMD.
Amdahl shows that the remaining sequential part limits parallel speedup (Amdahl's law).
NVIDIA GeForce 256 is marketed under the name "GPU".
NVIDIA releases CUDA: the GPU becomes general-purpose programmable in a C-like language.
AlexNet is trained on two GPUs and wins ImageNet; the GPU becomes the engine of deep learning.
The Volta architecture introduces tensor cores: hardware units dedicated to matrix multiplication.
The Course
Core Concepts
32 threads execute the same instruction together.
The grid, block and thread hierarchy.
Registers, shared memory, global memory.
Neighbouring threads should access neighbouring addresses.
Barriers, atomic operations and race conditions.
The sequential part caps the achievable speedup.
Enough threads to keep the cores busy.
Map, reduction, prefix sum, stencil.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Parallel architectures: Flynn's taxonomy, CPU vs GPU
- The CUDA model: grids, blocks and threads
- Memory hierarchy and CPU-GPU data transfer
- Coalesced access and shared memory
- Warp execution, branch divergence, occupancy
- Synchronization, atomics and race conditions
- Parallel patterns: reduction and prefix sum
- Stencils, histograms and sparse matrix operations
- Tiled matrix multiplication and tensor cores
- Streams, asynchronous copies and multiple GPUs
- Profiling: Nsight and the roofline model
- OpenMP on the CPU, MPI on clusters
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Add Thousands of Numbers at Once
Computing a network's loss means adding millions of numbers; a GPU does it as a tree, adding pairs at the same time. Change the thread count and watch step by step; Amdahl's law shows the limit.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Programming with C | pointers, arrays and memory allocation: the basis of CUDA C |
| Operating Systems | threads, synchronization and race conditions |
| Microprocessors | registers, caches and the memory hierarchy |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you manage a dynamic array in C with pointers and malloc? | A C programming course |
| Can you explain the race when two threads write the same variable? | Threads and synchronization from operating systems |
| Can you explain the speed gap between cache and main memory? | A microprocessors or computer architecture course |
| Can you write the product of two matrices with nested loops? | Linear Algebra |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | Deep Learning builds on this course | parallel training of large networks, tensor operations |
| shares methods | Shares methods with Computer Graphics | graphics pipeline, shaders |
| shares methods | Shares methods with Big Data Systems | parallel and distributed computing |
| shares methods | Shares methods with Embedded AI | hardware accelerators |
Differences
Where It Differs (1/2)
| Criterion | GPU & Parallel Computing | Deep Learning | Big Data |
|---|---|---|---|
| Core question | How do thousands of cores share the same computation? | How do many-layered networks learn useful representations from raw data? | How is data that does not fit on one machine stored and processed? |
| Approach | data-parallel computation | multilayer representation learning | distributed storage and computation |
| Input | parallelizable algorithm | raw data: images, text, audio | huge, fast-flowing data |
| Output | accelerated program | learned representation and prediction | processed data, query results |
| Success measure | speedup, memory bandwidth | accuracy, loss, scaling | scalability, latency, cost |
| Math load | low-medium | high: derivatives, matrices, probability | low |
| Data needs | none | large; needs GPUs | very large |
- Deep Learning: Deep learning uses the GPU through ready-made libraries; this course writes the kernels and memory access beneath them.
- Big Data: Big data systems spread work across many machines; this course makes efficient use of thousands of cores on one device.
Differences
Where It Differs (2/2)
| Criterion | GPU & Parallel Computing | Computer Graphics | Embedded AI |
|---|---|---|---|
| Core question | How do thousands of cores share the same computation? | How is a realistic image produced from the description of a scene? | How do AI models run on small, low-power devices? |
| Approach | data-parallel computation | geometry and light simulation | resource-constrained inference |
| Input | parallelizable algorithm | scene: models, lights, camera | trained model, device constraints |
| Output | accelerated program | image or animation | small, fast model |
| Success measure | speedup, memory bandwidth | realism, frame rate | latency, energy, memory |
| Math load | low-medium | high: linear algebra, geometry | medium |
| Data needs | none | none | training data already exists |
- Computer Graphics: Computer graphics programs the GPU with shaders to draw images; this course uses it for general-purpose computation.
- Embedded AI: Embedded AI teaches making do with little power; this course teaches getting peak performance from the largest accelerator.
Practice
Applications
Matrix multiplication and convolution on tensor cores.
Fluids, climate and molecular dynamics.
Real-time filters, encoding and decoding.
Monte Carlo risk and pricing.
Sequence alignment and protein structure computation.
Table and database operations on the GPU.
Practice
Risks and Ethics
Leaving out data transfer or comparing against a weak CPU baseline makes the speedup look far larger than it is.
A kernel with missing synchronization is right on most runs and wrong on some; the bug is hard to catch.
Parallel sums change the order of operations, so results differ slightly from the CPU and from run to run.
GPU clusters that train large models draw a lot of power; inefficient code inflates that cost for nothing.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic example problems
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| A GPU speeds up any program. | Only parts with many independent, similar operations speed up; branching and sequential parts do not. |
| Doubling the cores halves the run time. | Amdahl's law: the sequential part sets an upper bound on the achievable speedup. |
| Computation is the most expensive part. | In most GPU programs the bottleneck is memory bandwidth and data transfer between CPU and GPU. |
| Parallel programming is only for deep learning. | Simulation, graphics and scientific computing used GPUs long before deep learning. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Kirk, Hwu and El Hajj, Programming Massively Parallel Processors, 4th ed., Morgan Kaufmann, 2022.
- Sanders and Kandrot, CUDA by Example, Addison-Wesley, 2010.
- Hennessy and Patterson, Computer Architecture: A Quantitative Approach, 6th ed., Morgan Kaufmann, 2017.
- Amdahl, "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities", AFIPS Conference Proceedings 30, 1967.