Optimization
Introduction
What Question Does It Ask?
Optimization studies how to find the values that minimize or maximize an objective function while respecting constraints. Training an AI model means searching for the weights that minimize a loss function, so an optimization method runs inside every learning algorithm.
At a glance
mathematical model of objective and constraints
objective function and constraints
optimal or near-optimal solution
objective value, convergence rate
the theory itself
none
Introduction
A Short History
Cauchy proposes steepest descent (gradient descent) for solving systems of equations.
Dantzig develops the simplex method for linear programming.
Kuhn and Tucker publish optimality conditions for constrained problems (KKT).
Bellman's book Dynamic Programming is published.
Karmarkar introduces a polynomial-time interior point method for linear programming.
Kingma and Ba propose Adam, soon widely used to train deep networks.
The Course
Core Concepts
The cost or loss to be minimized.
Equalities and inequalities a solution must satisfy.
Linear objective and constraints; the simplex method.
Every local optimum is also the global optimum.
Small steps against the slope.
Using a small part of the data at each step.
Folding constraints into the objective; KKT conditions.
Integer variables; often very hard.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Formulating problems: variables, objective, constraints
- Linear programming and the simplex method
- Duality and sensitivity analysis
- Convex sets and convex functions
- Convex problems: QP, SOCP, SDP
- Optimality conditions: Lagrange and KKT
- Gradient descent and step size selection
- Newton and quasi-Newton methods (BFGS)
- Constrained optimization and interior point methods
- Stochastic gradient, momentum and Adam
- Integer programming and branch and bound
- Dynamic programming
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Training a Price Model
A model predicts house prices as “price = w·area + b”; gradient descent moves its two weights toward the lowest loss. Click the map to set the start and change the learning rate.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Linear Algebra | gradient, Hessian, convexity |
| Calculus | derivatives, partial derivatives and Taylor series: the language of gradients and Hessians |
| Algorithms | dynamic programming, greedy methods and complexity |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you compute the gradient of a function of several variables? | Partial derivatives and the chain rule from calculus |
| Can you write the Taylor expansion of a function up to the second term? | Taylor series from calculus |
| Can you multiply a matrix by a vector and solve a linear system? | Linear Algebra |
| Can you update NumPy arrays step by step inside a loop in Python? | An intro programming course, then a short NumPy tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | Machine Learning builds on this course | loss minimization, regularization |
| builds on | Artificial Neural Networks builds on this course | gradient descent, learning rate |
| builds on | This course builds on Linear Algebra | gradient, Hessian, convexity |
| shares methods | Shares methods with Artificial Intelligence | state-space search, constraint satisfaction |
| shares methods | Shares methods with Reinforcement Learning | dynamic programming, policy gradient |
| shares methods | Shares methods with Evolutionary and Metaheuristic Optimization | derivative-free and global optimization |
Differences
Where It Differs (1/2)
| Criterion | Optimization | Evolutionary Optimization | Artificial Intelligence |
|---|---|---|---|
| Core question | How do we find the values that make an objective function best? | How do we find a good solution when there is no gradient or the search space is huge? | How does a machine act intelligently by searching, reasoning and planning? |
| Approach | mathematical model of objective and constraints | population-based heuristic search | search, logic, probability and learning combined |
| Input | objective function and constraints | objective, solution encoding | problem definition: states, actions, goal |
| Output | optimal or near-optimal solution | good, not proven optimal, solution | action sequence, plan or decision |
| Success measure | objective value, convergence rate | quality, evaluation count | reaching the goal, cost, rationality |
| Math load | the theory itself | low-medium | medium: logic, probability, graphs |
| Data needs | none | none; evaluations suffice | little; often a model and knowledge suffice |
- Evolutionary Optimization: Evolutionary methods search without derivatives; this course teaches methods that exploit problem structure (derivatives, convexity).
- Artificial Intelligence: AI starts by modeling a problem as an agent and an environment; optimization solves a mathematically stated problem.
Differences
Where It Differs (2/2)
| Criterion | Optimization | Machine Learning | Reinforcement Learning |
|---|---|---|---|
| Core question | How do we find the values that make an objective function best? | How does a program generalize from examples? | How can good decisions be learned by trial and error from a reward signal alone? |
| Approach | mathematical model of objective and constraints | statistical learning | trial and error with rewards |
| Input | objective function and constraints | labeled or unlabeled examples | environment: state, reward |
| Output | optimal or near-optimal solution | a model that makes predictions | policy: state → action |
| Success measure | objective value, convergence rate | error on unseen data | total expected reward |
| Math load | the theory itself | high: linear algebra, probability, optimization | high: probability, dynamic programming |
| Data needs | none | moderate to large | generated through interaction |
- Machine Learning: ML lowers training loss but also wants success on new data; optimization only makes the given objective as good as possible.
- Reinforcement Learning: Reinforcement learning maximizes reward by trial and error; here the objective function is written down explicitly from the start.
Practice
Applications
Finding the weights that minimize the loss.
The maximum-margin separator: a convex problem.
Choosing learning rate and regularization.
Routing, transport and warehouse location.
Shifts, production and timetables.
Balancing return against risk.
Practice
Risks and Ethics
Optimizing what is measurable can defeat the real goal: a system that maximizes clicks alone promotes clickbait.
A constraint left out of the model is ignored: the cheapest route may use a banned road, the best rota may skip legal rest.
An optimum computed from estimated parameters can degrade badly under small errors; Markowitz portfolios are notoriously sensitive.
Variables on wildly different scales slow a solver down or can lead it to a false "infeasible" verdict.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic example problems
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Gradient descent always finds the best solution. | On non-convex problems it can get stuck in a local optimum or at a saddle point. |
| The larger the learning rate, the faster. | Too large a step overshoots the minimum and the loss diverges; too small a step drags training out. |
| Integer problems are solved by rounding. | Rounding the linear solution can break constraints or land far from the optimum. |
| The lowest loss means the best model. | Driving training loss to zero can mean overfitting; in AI the goal is generalization. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Boyd and Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
- Nocedal and Wright, Numerical Optimization, 2nd ed., Springer, 2006.
- Bertsimas and Tsitsiklis, Introduction to Linear Optimization, Athena Scientific, 1997.
- Bottou, Curtis and Nocedal, "Optimization Methods for Large-Scale Machine Learning", SIAM Review 60(2), 2018.