Pattern Recognition
Introduction
What Question Does It Ask?
Pattern recognition studies how to decide which class an object belongs to by looking at a feature vector extracted from measurements. The course treats the chain from sensor to decision as a whole: preprocessing, feature selection, classifier design and error analysis.
At a glance
statistical decision theory
measurement or feature vector
class label
classification error
medium-high: probability, linear algebra
labeled examples
Introduction
A Short History
Fisher publishes linear discriminant analysis; the Iris data from that paper is still the first example.
Hu introduces moment features invariant to translation, scale and rotation.
Cover and Hart show that the nearest neighbor rule's error is at most twice the Bayes error.
Duda and Hart publish the field's core textbook, "Pattern Classification and Scene Analysis".
Turk and Pentland build face recognition on PCA with eigenfaces.
LeCun et al. publish LeNet-5 and the MNIST handwritten digit dataset.
The Course
Core Concepts
Measurements that represent an object: length, color, texture.
Each example is a point; classes are regions in this space.
Choose the class with the largest posterior; minimum error.
The surface that splits feature space into class regions.
Parametric (Gaussian) or nonparametric (Parzen).
Assign a new example to the class of the nearest labeled ones.
Few but discriminative features with PCA and LDA.
Shows which class is confused with which.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Recognition systems: sensing, features, decision
- Bayesian decision theory, risk and loss
- The normal distribution and discriminant functions
- Parameter estimation: maximum likelihood, Bayesian
- Nonparametric density estimation: Parzen windows
- Nearest-neighbour rule and distance measures
- Linear discriminants and the perceptron
- Support vector machines
- Feature extraction, selection, curse of dimensionality
- PCA and Fisher linear discriminant analysis (LDA)
- Evaluating and comparing classifiers
- Clustering and Gaussian mixture models
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Reject the Defective Part
Every screw on the line is measured; defective ones are longer on average, but the distributions overlap. Change the defect rate and error cost to move the Bayes boundary; click the chart to measure a part.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Probability and Statistics | Bayesian decision theory, density estimation |
| Programming | arrays, loops and functions in Python or Java |
| Linear Algebra | vectors, distances, eigenvalues: the language of PCA and LDA |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you explain the normal distribution, mean and covariance? | Probability and Statistics |
| Can you explain Euclidean distance, eigenvalues and eigenvectors? | Linear Algebra |
| Can you find the minimum of a function by taking its derivative? | Derivatives and extrema from calculus |
| Can you load and process a dataset with NumPy arrays in Python? | An intro programming course, then a short NumPy tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Probability and Statistics | Bayesian decision theory, density estimation |
| application | Biometric Systems applies the methods of this course | matching, verification and identification |
| shares methods | Shares methods with Machine Learning | classifiers, decision boundaries |
| shares methods | Shares methods with Image Processing | feature extraction, template matching |
| shares methods | Shares methods with Computer Vision | object classification |
Differences
Where It Differs (1/2)
| Criterion | Pattern Recognition | Machine Learning | Computer Vision |
|---|---|---|---|
| Core question | How do we decide from measurements which class an object belongs to? | How does a program generalize from examples? | How does a computer understand what is in an image or a video? |
| Approach | statistical decision theory | statistical learning | extracting meaning from images |
| Input | measurement or feature vector | labeled or unlabeled examples | image or video |
| Output | class label | a model that makes predictions | label, box, depth, 3D structure |
| Success measure | classification error | error on unseen data | accuracy, mAP, IoU |
| Math load | medium-high: probability, linear algebra | high: linear algebra, probability, optimization | high: geometry, linear algebra |
| Data needs | labeled examples | moderate to large | many labeled images |
- Machine Learning: ML studies learning algorithms in general; this course focuses on the design from measurement to class decision and on the Bayes error.
- Computer Vision: Computer vision tries to understand the scene in an image; this course treats class decisions from any kind of measurement.
Differences
Where It Differs (2/2)
| Criterion | Pattern Recognition | Image Processing | Biometrics |
|---|---|---|---|
| Core question | How do we decide from measurements which class an object belongs to? | How is an image enhanced, transformed and divided into regions? | How is a person recognized from their face, fingerprint or voice? |
| Approach | statistical decision theory | pixel-level transformations | matching and verification |
| Input | measurement or feature vector | image | face, fingerprint, iris, voice |
| Output | class label | processed image or regions | identity decision |
| Success measure | classification error | visual quality, PSNR | FAR, FRR, EER |
| Math load | medium-high: probability, linear algebra | medium: convolution, Fourier | medium |
| Data needs | labeled examples | little; learning not required | few samples per person |
- Image Processing: Image processing enhances and transforms images, and the result is still an image; this course outputs a class label.
- Biometrics: Biometrics is an application of pattern recognition; this course teaches general decision methods, not person-specific ones.
Practice
Applications
Reading postcodes, bank checks and forms.
Turning a scanned document into editable text.
Identity verification and identification.
Rejecting defective products on the line.
Classifying ECGs, blood tests or tissue samples.
Finding out who is speaking from the voice.
Practice
Risks and Ethics
Face recognition enables covert tracking; Clearview AI was fined in several European countries for faces scraped from the internet.
In the Gender Shades study, commercial gender classifiers erred far more often on darker-skinned women than on lighter-skinned men.
If a missed cancer weighs the same as a false alarm, total error looks low while outcomes are dangerous; choose the loss matrix with care.
Selecting features on the full dataset and then cross-validating makes the error look far lower than it really is.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Pattern recognition is the old name for machine learning. | The two overlap heavily; pattern recognition is an engineering view that also covers sensing and feature design. |
| Choosing the classifier is the key step. | In most problems feature selection decides success; a good classifier cannot rescue bad features. |
| With the right features the error drops to zero. | If classes overlap, even the best decision errs as much as the Bayes error. |
| The more features, the better. | As dimension grows, examples become sparse (the curse of dimensionality); useless features hurt performance. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Duda, Hart and Stork, Pattern Classification, 2nd ed., Wiley, 2001.
- Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
- Theodoridis and Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2009.
- Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed., Academic Press, 1990.