Linear Algebra
Introduction
What Question Does It Ask?
Linear algebra studies vectors, matrices and linear transformations. In AI and data courses almost everything is a vector: an image, a word, a user. A neural network layer is, to a large extent, a matrix multiplication.
At a glance
vector spaces and transformations
vectors, matrices
solution, decomposition, transformation
accuracy, numerical stability
the theory itself
none
Introduction
A Short History
Grassmann sets out the idea of a multidimensional vector space systematically for the first time.
Cayley treats matrices as algebraic objects in their own right; matrix multiplication is defined.
Pearson finds the lines and planes that best fit data: principal component analysis (PCA).
Eckart and Young show that the SVD gives the best low-rank approximation of a matrix.
Golub and Kahan give a numerically stable way to compute the SVD on a computer.
BLAS is published: the basic matrix operations beneath today's scientific and AI libraries.
The Course
Core Concepts
Turning data into a point in many dimensions.
Applying a transformation or a network layer.
Measuring similarity (cosine) and distance.
How many independent directions the data really spans.
Ax = b; the least squares solution.
Vectors whose direction a map leaves unchanged.
Splitting any matrix into rotate, scale, rotate.
The closest point to a vector in a subspace.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Vectors, linear combinations and dot products
- Linear systems and Gaussian elimination
- Matrix operations, inverses and LU factorization
- Vector spaces and the four fundamental subspaces
- Linear independence, basis, dimension and rank
- Orthogonality, projections and least squares
- Gram-Schmidt and QR factorization
- Determinants and their properties
- Eigenvalues, eigenvectors and diagonalization
- Symmetric and positive definite matrices
- Singular value decomposition (SVD) and PCA
- Linear transformations and change of basis
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Augmenting a Digit Image
A handwritten digit recognizer has few examples; making new ones by rotating and shearing each image (data augmentation) means multiplying every pixel by a 2×2 matrix. Change the matrix and watch the “7”.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| High school mathematics | solving equations, functions and the coordinate plane |
| Programming | arrays and loops: trying matrix operations in code |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you solve a system of two equations in two unknowns by elimination? | Systems of equations from high school maths |
| Can you find the distance between two points in the plane? | Coordinate geometry from high school maths |
| Can you find an angle of a right triangle with sine and cosine? | High school trigonometry |
| Can you multiply two Python lists element by element in a loop and sum them? | An intro programming course, then a short NumPy tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | Machine Learning builds on this course | feature vectors, matrix multiplication, PCA |
| builds on | Artificial Neural Networks builds on this course | weight matrices, vectorized computation |
| builds on | Computer Graphics builds on this course | transformation matrices, projection |
| builds on | Robotics and Robot Vision builds on this course | coordinate transforms, kinematics |
| builds on | Information Retrieval and Recommender Systems builds on this course | vector space model, matrix factorization |
| builds on | Social Network Analysis builds on this course | adjacency matrix, spectral methods |
| builds on | Optimization builds on this course | gradient, Hessian, convexity |
Differences
Where It Differs (1/2)
| Criterion | Linear Algebra | Optimization | Probability & Statistics |
|---|---|---|---|
| Core question | How do vectors and matrices represent data and transformations? | How do we find the values that make an objective function best? | How is uncertainty measured, and how do we draw reliable conclusions from limited data? |
| Approach | vector spaces and transformations | mathematical model of objective and constraints | probability models and inference |
| Input | vectors, matrices | objective function and constraints | sample data |
| Output | solution, decomposition, transformation | optimal or near-optimal solution | estimate, confidence interval, test result |
| Success measure | accuracy, numerical stability | objective value, convergence rate | unbiasedness, power, significance |
| Math load | the theory itself | the theory itself | the theory itself |
| Data needs | none | none | a sample |
- Optimization: Optimization searches for the best values; linear algebra supplies the gradients, Hessians and system solves used in that search.
- Probability & Statistics: Probability measures uncertainty; linear algebra represents and decomposes the data and the covariance matrix.
Differences
Where It Differs (2/2)
| Criterion | Linear Algebra | Machine Learning | Computer Graphics |
|---|---|---|---|
| Core question | How do vectors and matrices represent data and transformations? | How does a program generalize from examples? | How is a realistic image produced from the description of a scene? |
| Approach | vector spaces and transformations | statistical learning | geometry and light simulation |
| Input | vectors, matrices | labeled or unlabeled examples | scene: models, lights, camera |
| Output | solution, decomposition, transformation | a model that makes predictions | image or animation |
| Success measure | accuracy, numerical stability | error on unseen data | realism, frame rate |
| Math load | the theory itself | high: linear algebra, probability, optimization | high: linear algebra, geometry |
| Data needs | none | moderate to large | none |
- Machine Learning: ML learns a model from data; linear algebra is the language of the computation inside that model and learns nothing itself.
- Computer Graphics: Computer graphics applies 3D transformation matrices; this course gives the general theory of those matrices.
Practice
Applications
Layer = matrix product plus activation.
Dimensionality reduction and visualization.
Factorizing the user-item matrix.
Similar meaning = nearby vectors.
Dominant eigenvector of the link matrix.
Transforms, compression and filters.
Practice
Risks and Ethics
With a large condition number, a tiny input error is magnified many times; the answer fills up with meaningless digits.
Forming AᵀA for least squares squares the condition number; solving with QR or SVD is safer.
NumPy broadcasting combines mismatched shapes without an error; the result is wrong but looks fine.
On unscaled features PCA favours the variable with the largest units; a component is not always a meaningful factor.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic example problems
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Linear algebra is just solving equations. | Its real power is understanding transformations and decompositions: PCA, SVD and embeddings are built on them. |
| Computing the inverse and multiplying is the right way. | In practice you do not invert; you solve with LU, QR or Cholesky, which is faster and more stable. |
| High-dimensional space behaves like 3D. | In high dimensions points sit at similar distances from each other; intuition misleads (curse of dimensionality). |
| With libraries around, theory is unnecessary. | Rank deficiency, ill-conditioning and shape errors make sense only if you know the theory. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Strang, Introduction to Linear Algebra, 6th ed., Wellesley-Cambridge Press, 2023.
- Strang, Linear Algebra and Learning from Data, Wellesley-Cambridge Press, 2019.
- Trefethen and Bau, Numerical Linear Algebra, SIAM, 1997.
- Deisenroth, Faisal and Ong, Mathematics for Machine Learning, Cambridge University Press, 2020.