Optimization

Guide to AI and Data Courses · Part V: Mathematical Foundations
Objective, constraintsLinear programmingConvexityGradient methodsWhere in AINeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How do we find the values that make an objective function best?

Optimization studies how to find the values that minimize or maximize an objective function while respecting constraints. Training an AI model means searching for the weights that minimize a loss function, so an optimization method runs inside every learning algorithm.

At a glance

Approach

mathematical model of objective and constraints

Input

objective function and constraints

Output

optimal or near-optimal solution

Success measure

objective value, convergence rate

Math load

the theory itself

Data needs

none

Introduction

A Short History

1847

Cauchy proposes steepest descent (gradient descent) for solving systems of equations.

1947

Dantzig develops the simplex method for linear programming.

1951

Kuhn and Tucker publish optimality conditions for constrained problems (KKT).

1957

Bellman's book Dynamic Programming is published.

1984

Karmarkar introduces a polynomial-time interior point method for linear programming.

2014

Kingma and Ba propose Adam, soon widely used to train deep networks.

The Course

Core Concepts

Objective function

The cost or loss to be minimized.

Constraints

Equalities and inequalities a solution must satisfy.

Linear programming

Linear objective and constraints; the simplex method.

Convexity

Every local optimum is also the global optimum.

Gradient descent

Small steps against the slope.

Stochastic gradient

Using a small part of the data at each step.

Lagrange multipliers

Folding constraints into the objective; KKT conditions.

Integer programming

Integer variables; often very hard.

The Course

A Typical Workflow

Modelvariables,goal, limitsClassifylinear?convex?Pick methodsimplex,gradient, ...SolvewatchconvergenceValidatesensitivity,constraints
Example: Training a neural network is an optimization problem: the variables are the weights and the objective is the average loss. The problem is not convex, so stochastic gradient descent on mini-batches or Adam is used, and the loss curve is watched on a validation set.

The Course

What Does the Course Cover?

  1. Formulating problems: variables, objective, constraints
  2. Linear programming and the simplex method
  3. Duality and sensitivity analysis
  4. Convex sets and convex functions
  5. Convex problems: QP, SOCP, SDP
  6. Optimality conditions: Lagrange and KKT
  7. Gradient descent and step size selection
  8. Newton and quasi-Newton methods (BFGS)
  9. Constrained optimization and interior point methods
  10. Stochastic gradient, momentum and Adam
  11. Integer programming and branch and bound
  12. Dynamic programming

Skills you will gain

modelling convex problems in CVXPYwriting linear models with PuLPusing scipy.optimizeproving convexitymonitoring and tuning convergence

A typical semester; the order and coverage vary between universities.

Try It

Training a Price Model

A model predicts house prices as “price = w·area + b”; gradient descent moves its two weights toward the lowest loss. Click the map to set the start and change the learning rate.

With a small learning rate the model crawls; with a large one it bounces across the narrow valley and diverges. Scaling the features makes the valley round, which is why normalization speeds up neural network training.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Linear Algebragradient, Hessian, convexity
Calculusderivatives, partial derivatives and Taylor series: the language of gradients and Hessians
Algorithmsdynamic programming, greedy methods and complexity

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you compute the gradient of a function of several variables?Partial derivatives and the chain rule from calculus
Can you write the Taylor expansion of a function up to the second term?Taylor series from calculus
Can you multiply a matrix by a vector and solve a linear system?Linear Algebra
Can you update NumPy arrays step by step inside a loop in Python?An intro programming course, then a short NumPy tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
builds onMachine Learning builds on this courseloss minimization, regularization
builds onArtificial Neural Networks builds on this coursegradient descent, learning rate
builds onThis course builds on Linear Algebragradient, Hessian, convexity
shares methodsShares methods with Artificial Intelligencestate-space search, constraint satisfaction
shares methodsShares methods with Reinforcement Learningdynamic programming, policy gradient
shares methodsShares methods with Evolutionary and Metaheuristic Optimizationderivative-free and global optimization

Differences

Where It Differs (1/2)

CriterionOptimizationEvolutionary OptimizationArtificial Intelligence
Core questionHow do we find the values that make an objective function best?How do we find a good solution when there is no gradient or the search space is huge?How does a machine act intelligently by searching, reasoning and planning?
Approachmathematical model of objective and constraintspopulation-based heuristic searchsearch, logic, probability and learning combined
Inputobjective function and constraintsobjective, solution encodingproblem definition: states, actions, goal
Outputoptimal or near-optimal solutiongood, not proven optimal, solutionaction sequence, plan or decision
Success measureobjective value, convergence ratequality, evaluation countreaching the goal, cost, rationality
Math loadthe theory itselflow-mediummedium: logic, probability, graphs
Data needsnonenone; evaluations sufficelittle; often a model and knowledge suffice
  • Evolutionary Optimization: Evolutionary methods search without derivatives; this course teaches methods that exploit problem structure (derivatives, convexity).
  • Artificial Intelligence: AI starts by modeling a problem as an agent and an environment; optimization solves a mathematically stated problem.

Differences

Where It Differs (2/2)

CriterionOptimizationMachine LearningReinforcement Learning
Core questionHow do we find the values that make an objective function best?How does a program generalize from examples?How can good decisions be learned by trial and error from a reward signal alone?
Approachmathematical model of objective and constraintsstatistical learningtrial and error with rewards
Inputobjective function and constraintslabeled or unlabeled examplesenvironment: state, reward
Outputoptimal or near-optimal solutiona model that makes predictionspolicy: state → action
Success measureobjective value, convergence rateerror on unseen datatotal expected reward
Math loadthe theory itselfhigh: linear algebra, probability, optimizationhigh: probability, dynamic programming
Data needsnonemoderate to largegenerated through interaction
  • Machine Learning: ML lowers training loss but also wants success on new data; optimization only makes the given objective as good as possible.
  • Reinforcement Learning: Reinforcement learning maximizes reward by trial and error; here the objective function is written down explicitly from the start.

Practice

Applications

Model training

Finding the weights that minimize the loss.

SVM

The maximum-margin separator: a convex problem.

Hyperparameter search

Choosing learning rate and regularization.

Logistics

Routing, transport and warehouse location.

Scheduling

Shifts, production and timetables.

Portfolio selection

Balancing return against risk.

Practice

Risks and Ethics

The wrong objective

Optimizing what is measurable can defeat the real goal: a system that maximizes clicks alone promotes clickbait.

A forgotten constraint

A constraint left out of the model is ignored: the cheapest route may use a banned road, the best rota may skip legal rest.

Fragile optima

An optimum computed from estimated parameters can degrade badly under small errors; Markowitz portfolios are notoriously sensitive.

Poor scaling

Variables on wildly different scales slow a solver down or can lead it to a false "infeasible" verdict.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

SciPy (scipy.optimize)CVXPYGoogle OR-ToolsPuLPGurobiPyTorch (torch.optim)

Classic example problems

Diet problemTransportation problemKnapsack problemTraveling salesmanMarkowitz portfolioRosenbrock function

Practice

Common Misconceptions

Common beliefIn fact
Gradient descent always finds the best solution.On non-convex problems it can get stuck in a local optimum or at a saddle point.
The larger the learning rate, the faster.Too large a step overshoots the minimum and the loss diverges; too small a step drags training out.
Integer problems are solved by rounding.Rounding the linear solution can break constraints or land far from the optimum.
The lowest loss means the best model.Driving training loss to zero can mean overfitting; in AI the goal is generalization.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

Operations researcherMachine learning engineerLogistics and planning analystQuantitative analyst
After this course, how training works in machine learning and neural networks becomes clear. From here you can move to evolutionary and metaheuristic optimization for problems without derivatives, reinforcement learning for sequential decisions, and operations research for allocating resources.

Next Steps

Further Reading

  • Boyd and Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
  • Nocedal and Wright, Numerical Optimization, 2nd ed., Springer, 2006.
  • Bertsimas and Tsitsiklis, Introduction to Linear Optimization, Athena Scientific, 1997.
  • Bottou, Curtis and Nocedal, "Optimization Methods for Large-Scale Machine Learning", SIAM Review 60(2), 2018.

The End

Chapter 33: Optimization
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.