Computer Vision
Introduction
What Question Does It Ask?
Computer vision studies how to extract meaning from an image or video: which objects are there, where they are, what 3D structure the scene has. The input is an image but the output is not; it is a label, a box, a mask or depth. Transforming the image is the job of image processing.
At a glance
extracting meaning from images
image or video
label, box, depth, 3D structure
accuracy, mAP, IoU
high: geometry, linear algebra
many labeled images
Introduction
A Short History
Larry Roberts's PhD thesis recovers 3D blocks from line drawings: the field's first work.
MIT's "Summer Vision Project" expects to solve vision in one summer; it does not.
Marr's book Vision appears, describing vision as a layered computational process.
Lowe introduces SIFT features: local point matching invariant to scale and rotation.
Viola and Jones publish a real-time face detector; it ships in digital cameras.
AlexNet wins ImageNet by a wide margin; the deep learning era in vision begins.
The Course
Core Concepts
Corners and keypoints: Harris, SIFT, ORB.
One label for the whole image: "cat".
A box and label per object: YOLO, Faster R-CNN.
A class for every pixel: road, pedestrian, car.
Pinhole model, calibration, intrinsics and extrinsics.
Stereo, epipolar geometry, structure from motion.
Optical flow; following an object frame to frame.
CNNs and vision transformers (ViT).
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Image formation and the camera model
- Filters and edge detection (image processing review)
- Local features: corners, SIFT, ORB
- Feature matching and RANSAC
- Homographies and panorama stitching
- Camera calibration and stereo vision
- Structure from motion and multi-view geometry
- Optical flow and object tracking
- Classification with convolutional networks
- Object detection: the R-CNN family, YOLO
- Semantic and instance segmentation
- Vision-language models and current trends
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Find the Rings in the Image
On an assembly line a camera must count the ring-shaped gaskets on the belt. A template slides over every position and similarity is measured. Change the confidence threshold and suppression, and check the boxes.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Image Processing | edges, filters, segmentation |
| Linear algebra | matrices, eigenvalues and projection: the language of camera geometry |
| Programming | array operations and using libraries in Python |
| Algorithms | search, matching and complexity: feature matching and tracking |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you explain matrix multiplication, inverses and eigenvalues? | Linear Algebra |
| Do you know that an image is a matrix of pixels, and what convolution does? | Image Processing |
| Can you explain how a neural network is trained with a loss and gradient descent? | Artificial Neural Networks |
| Can you work with NumPy arrays in Python? | An intro programming course, then a short NumPy tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Image Processing | edges, filters, segmentation |
| application | Medical Image Processing applies the methods of this course | lesion detection, classification |
| application | Remote Sensing applies the methods of this course | land cover classification, change detection |
| application | Biometric Systems applies the methods of this course | face recognition |
| shares methods | Shares methods with Deep Learning | convolutional neural networks, vision transformers |
| shares methods | Shares methods with Pattern Recognition | object classification |
| shares methods | Shares methods with Computer Graphics | camera model; graphics renders scenes, vision recovers them |
| shares methods | Shares methods with Robotics and Robot Vision | robot vision, SLAM |
Differences
Where It Differs (1/2)
| Criterion | Computer Vision | Image Processing | Deep Learning |
|---|---|---|---|
| Core question | How does a computer understand what is in an image or a video? | How is an image enhanced, transformed and divided into regions? | How do many-layered networks learn useful representations from raw data? |
| Approach | extracting meaning from images | pixel-level transformations | multilayer representation learning |
| Input | image or video | image | raw data: images, text, audio |
| Output | label, box, depth, 3D structure | processed image or regions | learned representation and prediction |
| Success measure | accuracy, mAP, IoU | visual quality, PSNR | accuracy, loss, scaling |
| Math load | high: geometry, linear algebra | medium: convolution, Fourier | high: derivatives, matrices, probability |
| Data needs | many labeled images | little; learning not required | large; needs GPUs |
- Image Processing: In image processing both input and output are images; in computer vision the input is an image and the output is meaning: a label, a box, depth.
- Deep Learning: Deep learning is a general tool; vision uses it together with camera geometry, multi-view and motion knowledge.
Differences
Where It Differs (2/2)
| Criterion | Computer Vision | Computer Graphics | Pattern Recognition |
|---|---|---|---|
| Core question | How does a computer understand what is in an image or a video? | How is a realistic image produced from the description of a scene? | How do we decide from measurements which class an object belongs to? |
| Approach | extracting meaning from images | geometry and light simulation | statistical decision theory |
| Input | image or video | scene: models, lights, camera | measurement or feature vector |
| Output | label, box, depth, 3D structure | image or animation | class label |
| Success measure | accuracy, mAP, IoU | realism, frame rate | classification error |
| Math load | high: geometry, linear algebra | high: linear algebra, geometry | medium-high: probability, linear algebra |
| Data needs | many labeled images | none | labeled examples |
- Computer Graphics: Graphics produces an image from a scene description; vision does the reverse, rebuilding the scene from the image.
- Pattern Recognition: Pattern recognition classifies any kind of measurement; vision treats classification together with spatial structure, position and 3D.
Practice
Applications
Detecting lanes, pedestrians and traffic signs.
Phone unlock, access control.
Checkout-free stores, shelf monitoring.
Locating the camera and adding objects to the scene.
Spotting defective products and removing them from the line.
Tracking players and the ball, statistics.
Practice
Risks and Ethics
Identifying people in a crowd can turn into tracking without consent; many countries restrict its use in public spaces.
Error rates can be markedly higher for skin tones and age groups that are under-represented in the training data.
A small sticker the human eye ignores can make a model read a stop sign as a different sign.
In self-driving, a single missed pedestrian is a cost that average accuracy does not show.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Computer vision is another name for image processing. | Image processing transforms the image; vision extracts meaning from it. Vision uses image processing as a first step. |
| Vision is a solved problem. | Networks still fail noticeably under lighting, weather or cameras that differ from the training data. |
| Vision is just deep learning. | Calibration, stereo, SLAM and 3D reconstruction rest largely on geometry. |
| A model "sees" an object the way a human does. | Networks can over-rely on texture and context; a small pixel change can flip a decision. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., Springer, 2022.
- Hartley and Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004.
- Prince, Computer Vision: Models, Learning, and Inference, Cambridge University Press, 2012.
- Marr, Vision, W. H. Freeman, 1982.