Pattern Recognition

Guide to AI and Data Courses · Part II: Branches of AI
Feature extractionBayes decisionsClassifiersClusteringMeasuring performanceNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How do we decide from measurements which class an object belongs to?

Pattern recognition studies how to decide which class an object belongs to by looking at a feature vector extracted from measurements. The course treats the chain from sensor to decision as a whole: preprocessing, feature selection, classifier design and error analysis.

At a glance

Approach

statistical decision theory

Input

measurement or feature vector

Output

class label

Success measure

classification error

Math load

medium-high: probability, linear algebra

Data needs

labeled examples

Introduction

A Short History

1936

Fisher publishes linear discriminant analysis; the Iris data from that paper is still the first example.

1962

Hu introduces moment features invariant to translation, scale and rotation.

1967

Cover and Hart show that the nearest neighbor rule's error is at most twice the Bayes error.

1973

Duda and Hart publish the field's core textbook, "Pattern Classification and Scene Analysis".

1991

Turk and Pentland build face recognition on PCA with eigenfaces.

1998

LeCun et al. publish LeNet-5 and the MNIST handwritten digit dataset.

The Course

Core Concepts

Feature vector

Measurements that represent an object: length, color, texture.

Feature space

Each example is a point; classes are regions in this space.

Bayesian decision theory

Choose the class with the largest posterior; minimum error.

Decision boundary

The surface that splits feature space into class regions.

Density estimation

Parametric (Gaussian) or nonparametric (Parzen).

Nearest neighbor

Assign a new example to the class of the nearest labeled ones.

Dimensionality reduction

Few but discriminative features with PCA and LDA.

Confusion matrix

Shows which class is confused with which.

The Course

A Typical Workflow

Sensecamera,microphonePreprocessnoise,normalizationExtractfeaturesdiscriminativemeasuresClassifydecision ruleEvaluateerror rate,confusion
Example: A mail sorting machine scans the postcode on an envelope, binarizes the image and splits it into digits, extracts shape features from each digit, decides which digit it is with nearest neighbor or an SVM, and sends misread envelopes to manual sorting.

The Course

What Does the Course Cover?

  1. Recognition systems: sensing, features, decision
  2. Bayesian decision theory, risk and loss
  3. The normal distribution and discriminant functions
  4. Parameter estimation: maximum likelihood, Bayesian
  5. Nonparametric density estimation: Parzen windows
  6. Nearest-neighbour rule and distance measures
  7. Linear discriminants and the perceptron
  8. Support vector machines
  9. Feature extraction, selection, curse of dimensionality
  10. PCA and Fisher linear discriminant analysis (LDA)
  11. Evaluating and comparing classifiers
  12. Clustering and Gaussian mixture models

Skills you will gain

designing features from measurementsbuilding a Bayes classifierk-NN and SVM with scikit-learnextracting image features with OpenCVerror analysis with a confusion matrix

A typical semester; the order and coverage vary between universities.

Try It

Reject the Defective Part

Every screw on the line is measured; defective ones are longer on average, but the distributions overlap. Change the defect rate and error cost to move the Bayes boundary; click the chart to measure a part.

Pattern recognition decides with probability and cost: the same measurement changes class when the defect rate or the cost of an error changes. The Bayes rule gives the boundary that minimizes expected loss.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Probability and StatisticsBayesian decision theory, density estimation
Programmingarrays, loops and functions in Python or Java
Linear Algebravectors, distances, eigenvalues: the language of PCA and LDA

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you explain the normal distribution, mean and covariance?Probability and Statistics
Can you explain Euclidean distance, eigenvalues and eigenvectors?Linear Algebra
Can you find the minimum of a function by taking its derivative?Derivatives and extrema from calculus
Can you load and process a dataset with NumPy arrays in Python?An intro programming course, then a short NumPy tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
builds onThis course builds on Probability and StatisticsBayesian decision theory, density estimation
applicationBiometric Systems applies the methods of this coursematching, verification and identification
shares methodsShares methods with Machine Learningclassifiers, decision boundaries
shares methodsShares methods with Image Processingfeature extraction, template matching
shares methodsShares methods with Computer Visionobject classification

Differences

Where It Differs (1/2)

CriterionPattern RecognitionMachine LearningComputer Vision
Core questionHow do we decide from measurements which class an object belongs to?How does a program generalize from examples?How does a computer understand what is in an image or a video?
Approachstatistical decision theorystatistical learningextracting meaning from images
Inputmeasurement or feature vectorlabeled or unlabeled examplesimage or video
Outputclass labela model that makes predictionslabel, box, depth, 3D structure
Success measureclassification errorerror on unseen dataaccuracy, mAP, IoU
Math loadmedium-high: probability, linear algebrahigh: linear algebra, probability, optimizationhigh: geometry, linear algebra
Data needslabeled examplesmoderate to largemany labeled images
  • Machine Learning: ML studies learning algorithms in general; this course focuses on the design from measurement to class decision and on the Bayes error.
  • Computer Vision: Computer vision tries to understand the scene in an image; this course treats class decisions from any kind of measurement.

Differences

Where It Differs (2/2)

CriterionPattern RecognitionImage ProcessingBiometrics
Core questionHow do we decide from measurements which class an object belongs to?How is an image enhanced, transformed and divided into regions?How is a person recognized from their face, fingerprint or voice?
Approachstatistical decision theorypixel-level transformationsmatching and verification
Inputmeasurement or feature vectorimageface, fingerprint, iris, voice
Outputclass labelprocessed image or regionsidentity decision
Success measureclassification errorvisual quality, PSNRFAR, FRR, EER
Math loadmedium-high: probability, linear algebramedium: convolution, Fouriermedium
Data needslabeled exampleslittle; learning not requiredfew samples per person
  • Image Processing: Image processing enhances and transforms images, and the result is still an image; this course outputs a class label.
  • Biometrics: Biometrics is an application of pattern recognition; this course teaches general decision methods, not person-specific ones.

Practice

Applications

Handwriting recognition

Reading postcodes, bank checks and forms.

Optical character recognition

Turning a scanned document into editable text.

Face and fingerprint

Identity verification and identification.

Industrial inspection

Rejecting defective products on the line.

Medical diagnosis

Classifying ECGs, blood tests or tissue samples.

Speaker recognition

Finding out who is speaking from the voice.

Practice

Risks and Ethics

Face recognition without consent

Face recognition enables covert tracking; Clearview AI was fined in several European countries for faces scraped from the internet.

Error gaps between groups

In the Gender Shades study, commercial gender classifiers erred far more often on darker-skinned women than on lighter-skinned men.

Treating all errors alike

If a missed cancer weighs the same as a false alarm, total error looks low while outcomes are dangerous; choose the loss matrix with care.

Leakage in feature selection

Selecting features on the full dataset and then cross-validating makes the error look far lower than it really is.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

scikit-learnNumPyOpenCVMATLAB Statistics and ML ToolboxWekaPRTools

Well-known datasets

IrisMNISTOptical Recognition of Handwritten Digits (UCI)Wisconsin Breast CancerAT&T (ORL) face datasetEMNIST

Practice

Common Misconceptions

Common beliefIn fact
Pattern recognition is the old name for machine learning.The two overlap heavily; pattern recognition is an engineering view that also covers sensing and feature design.
Choosing the classifier is the key step.In most problems feature selection decides success; a good classifier cannot rescue bad features.
With the right features the error drops to zero.If classes overlap, even the best decision errs as much as the Bayes error.
The more features, the better.As dimension grows, examples become sparse (the curse of dimensionality); useless features hurt performance.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

Image and signal processing engineerMachine learning engineerBiometrics specialistQuality inspection systems engineer
Pattern recognition is the shared core of computer vision, speech recognition and biometrics. After this course you can move to a sensing area (vision, audio, biometrics) or go on to deep learning, which also learns the features from data.

Next Steps

Further Reading

  • Duda, Hart and Stork, Pattern Classification, 2nd ed., Wiley, 2001.
  • Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
  • Theodoridis and Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2009.
  • Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed., Academic Press, 1990.

The End

Chapter 8: Pattern Recognition
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.