Image Processing
Introduction
What Question Does It Ask?
Image processing studies methods that take an image and turn it into a new image or into regions: reducing noise, boosting contrast, finding edges and object boundaries. Both input and output are images; understanding what is in the image is the job of computer vision.
At a glance
pixel-level transformations
image
processed image or regions
visual quality, PSNR
medium: convolution, Fourier
little; learning not required
Introduction
A Short History
The Bartlane system sends digitally coded newspaper photos across the Atlantic by cable.
Russell Kirsch makes the first digitally scanned photo: a 176×176-pixel image of his infant son.
JPL corrects Ranger 7's lunar images by computer; digital image processing is born.
Otsu publishes his method for picking a threshold from the histogram; still the most used thresholding.
The Canny edge detector appears: noise-robust, thin and continuous edges.
The JPEG standard is adopted; DCT-based lossy compression becomes the common language of photos.
The Course
Core Concepts
An image is a matrix of numbers; color has three channels.
Brightness distribution; contrast via histogram equalization.
Convolution with neighboring pixels: blurring, sharpening.
2D Fourier transform; filtering periodic noise.
Brightness jumps via gradient, Sobel and Canny.
Separating foreground from background; Otsu's method.
Erosion and dilation: cleaning up binary shapes.
Splitting an image into regions: region growing, watershed.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Image formation, sampling and quantization
- Point operations and histogram equalization
- Spatial filtering: smoothing and sharpening
- The 2D Fourier transform and frequency-domain filtering
- Image restoration: noise models, the Wiener filter
- Colour spaces and colour image processing
- Geometric transforms and interpolation
- Edge detection: gradients, Sobel, Canny
- Thresholding and Otsu's method
- Morphological operations
- Segmentation: region growing, watershed, labelling
- Image compression: the DCT and JPEG
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Finding Edges in a Noisy Image
A quality control camera returned a 32×32 grayscale image of a part; the sensor is noisy. Toggle the steps and slide the threshold: how many of the detected edges lie on a real shape boundary?
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Digital Signal Processing | 2D signals, convolution, Fourier transform |
| Linear algebra and calculus | matrix operations, derivatives and gradients: the language of filters and edges |
| Programming | loops over 2D arrays in Python or C |
| Data Structures | queues and union-find: region growing and connected-component labeling |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you visit every element of a matrix with nested loops? | An intro programming course, then a short NumPy tutorial |
| Can you explain what a partial derivative and a gradient are? | Derivatives and partial derivatives from calculus |
| Can you multiply matrices and take a dot product? | Linear Algebra |
| Can you explain what the frequency components of a signal are? | Digital Signal Processing |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Digital Signal Processing | 2D signals, convolution, Fourier transform |
| builds on | Computer Vision builds on this course | edges, filters, segmentation |
| application | Medical Image Processing applies the methods of this course | denoising, segmentation, registration |
| application | Remote Sensing applies the methods of this course | multispectral images, correction |
| shares methods | Shares methods with Pattern Recognition | feature extraction, template matching |
Differences
Where It Differs (1/2)
| Criterion | Image Processing | Computer Vision | Signal Processing |
|---|---|---|---|
| Core question | How is an image enhanced, transformed and divided into regions? | How does a computer understand what is in an image or a video? | How are sampled signals analyzed, filtered and transformed? |
| Approach | pixel-level transformations | extracting meaning from images | mathematical transforms and filters |
| Input | image | image or video | sampled signal |
| Output | processed image or regions | label, box, depth, 3D structure | filtered signal, spectrum |
| Success measure | visual quality, PSNR | accuracy, mAP, IoU | signal-to-noise ratio, frequency response |
| Math load | medium: convolution, Fourier | high: geometry, linear algebra | high: complex numbers, Fourier |
| Data needs | little; learning not required | many labeled images | no learning; the signal suffices |
- Computer Vision: In image processing both input and output are images; in computer vision the input is an image and the output is meaning: a label, a box, depth.
- Signal Processing: Signal processing works with a 1D time signal; image processing carries the same convolution and Fourier tools to two dimensions.
Differences
Where It Differs (2/2)
| Criterion | Image Processing | Computer Graphics | Medical Imaging |
|---|---|---|---|
| Core question | How is an image enhanced, transformed and divided into regions? | How is a realistic image produced from the description of a scene? | How do X-ray, CT and MRI images help diagnosis and treatment? |
| Approach | pixel-level transformations | geometry and light simulation | image processing plus clinical knowledge |
| Input | image | scene: models, lights, camera | CT, MRI, X-ray, ultrasound |
| Output | processed image or regions | image or animation | segmentation, measurement, diagnostic suggestion |
| Success measure | visual quality, PSNR | realism, frame rate | sensitivity, specificity, Dice |
| Math load | medium: convolution, Fourier | high: linear algebra, geometry | medium-high |
| Data needs | little; learning not required | none | little; labels are expensive |
- Computer Graphics: Graphics produces an image from a scene description; image processing takes an existing image and transforms it.
- Medical Imaging: Medical image processing applies this course's methods to medical images such as CT and MRI, under clinical constraints.
Practice
Applications
Brightness, sharpness, noise reduction.
Deskewing, converting to black and white.
JPEG, PNG and video coding.
Isolating scratches and stains on products.
Segmenting and counting cells.
Clean input for vision and medical models.
Practice
Risks and Ethics
The same tools can remove or add objects in a photo, and an edited image can be passed off as evidence.
Lossy JPEG and sharpening change detail in forensic or medical images; the original file must be kept.
Super-resolution and denoising can produce detail that was never in the scene, and it may be taken as real.
Thresholds and parameters picked by looking at one image fail silently when the lighting changes.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic test images and datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Image processing and computer vision are the same thing. | Image processing transforms the image; vision extracts meaning from it. One is a step before the other. |
| Sharpening brings back lost detail. | Sharpening accentuates existing edges; detail that was never recorded does not come back. |
| Deep learning does everything now. | Classical methods are still essential in calibration, compression, preprocessing and fast industrial inspection. |
| Enlarging an image increases its resolution. | Interpolation creates new pixels but no new information; the image just gets bigger. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Gonzalez and Woods, Digital Image Processing, 4th ed., Pearson, 2018.
- Jain, Fundamentals of Digital Image Processing, Prentice Hall, 1989.
- Burger and Burge, Digital Image Processing: An Algorithmic Introduction, 3rd ed., Springer, 2022.
- Canny, "A Computational Approach to Edge Detection", IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6), 1986.