Artificial Neural Networks
Introduction
What Question Does It Ask?
Artificial neural networks are models built by connecting, in layers, simple units that take a weighted sum and pass it through an activation function. Learning means adjusting the weights so as to reduce a loss function; the main tools for this are backpropagation and gradient descent.
At a glance
network of simple weighted units
numeric feature vector
class or numeric prediction
loss and accuracy
medium-high: derivatives, matrices
moderate
Introduction
A Short History
McCulloch and Pitts propose the first artificial neuron model with threshold logic units.
Hebb suggests that connections between neurons that fire together grow stronger: the Hebbian rule.
Rosenblatt introduces the perceptron; its weights are tuned from examples.
Minsky and Papert show that a single-layer perceptron cannot solve problems such as XOR.
Hopfield publishes a recurrent network that recalls memories by minimizing energy.
Rumelhart, Hinton and Williams popularize backpropagation; multilayer networks become trainable.
The Course
Core Concepts
Weighted sum of inputs, a bias and an output.
Sigmoid, tanh, ReLU: adds nonlinearity to the network.
Single-layer network that separates linearly separable classes.
Hidden layers learn nonlinear boundaries.
Computing the prediction layer by layer from input to output.
Finding the gradient of every weight with the chain rule.
Moves the weights in small steps where the loss decreases.
Weight penalties and early stopping prevent overfitting.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Biological and artificial neurons; McCulloch-Pitts
- Hebbian learning and the perceptron algorithm
- Adaline, least mean squares and the delta rule
- Linear separability and the XOR problem
- Multilayer networks and activation functions
- Backpropagation: computational graphs, chain rule
- Loss functions: squared error, cross-entropy
- Gradient descent, learning rate and momentum
- Weight initialization and vanishing gradients
- Generalization: regularization, early stopping
- Radial basis function networks
- Hopfield networks and self-organizing maps (SOM)
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Tangerine or Lemon?
A camera on the sorting belt measures the weight and color hue of each fruit. A single artificial neuron (a perceptron) learns to separate the two kinds: train it, or click a fruit to flip its label.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Machine Learning | parametric model, loss function, gradient descent |
| Linear Algebra | weight matrices, vectorized computation |
| Optimization | gradient descent, learning rate |
| Calculus | derivatives, partial derivatives and the chain rule: the math of backpropagation |
| Programming | arrays, loops and functions in Python |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you differentiate a composite function with the chain rule? | Derivatives and the chain rule from calculus |
| Can you multiply a matrix by a vector and state the result's shape? | Linear Algebra |
| Can you do vectorized operations on NumPy arrays without loops? | An intro programming course, then a short NumPy tutorial |
| Can you explain the difference between training and test error? | Machine Learning |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| subfield | This course is a subfield of Machine Learning | parametric model, loss function, gradient descent |
| subfield | Deep Learning is a subfield of this course | multilayer networks, backpropagation, activation functions |
| builds on | This course builds on Linear Algebra | weight matrices, vectorized computation |
| builds on | This course builds on Optimization | gradient descent, learning rate |
| shares methods | Shares methods with Evolutionary and Metaheuristic Optimization | neuroevolution, hyperparameter search |
Differences
Where It Differs (1/2)
| Criterion | Neural Networks | Machine Learning | Deep Learning |
|---|---|---|---|
| Core question | How do networks of simple units learn by adjusting their weights? | How does a program generalize from examples? | How do many-layered networks learn useful representations from raw data? |
| Approach | network of simple weighted units | statistical learning | multilayer representation learning |
| Input | numeric feature vector | labeled or unlabeled examples | raw data: images, text, audio |
| Output | class or numeric prediction | a model that makes predictions | learned representation and prediction |
| Success measure | loss and accuracy | error on unseen data | accuracy, loss, scaling |
| Math load | medium-high: derivatives, matrices | high: linear algebra, probability, optimization | high: derivatives, matrices, probability |
| Data needs | moderate | moderate to large | large; needs GPUs |
- Machine Learning: ML covers many model families from decision trees to SVMs; this course focuses only on networks built from neurons.
- Deep Learning: Deep learning moves on to convolutional, recurrent and transformer architectures; this course builds the basic network and training they rest on.
Differences
Where It Differs (2/2)
| Criterion | Neural Networks | Optimization | Evolutionary Optimization |
|---|---|---|---|
| Core question | How do networks of simple units learn by adjusting their weights? | How do we find the values that make an objective function best? | How do we find a good solution when there is no gradient or the search space is huge? |
| Approach | network of simple weighted units | mathematical model of objective and constraints | population-based heuristic search |
| Input | numeric feature vector | objective function and constraints | objective, solution encoding |
| Output | class or numeric prediction | optimal or near-optimal solution | good, not proven optimal, solution |
| Success measure | loss and accuracy | objective value, convergence rate | quality, evaluation count |
| Math load | medium-high: derivatives, matrices | the theory itself | low-medium |
| Data needs | moderate | none | none; evaluations suffice |
- Optimization: Optimization studies gradient methods in general; this course uses them to learn weights.
- Evolutionary Optimization: Neuroevolution searches weights or architectures by evolution; this course learns weights by gradients and backpropagation.
Practice
Applications
Handwritten digits on postcodes and checks.
Classification tasks such as credit risk or churn.
Learning nonlinear relations from measurements.
Simple robot and process controllers.
Detecting faults or motion from sensor data.
Completing a corrupted pattern with a Hopfield network.
Practice
Risks and Ethics
A large network on a small dataset drives training error to zero and then fails on new examples; without a validation set you never see it.
A network may learn traces of data collection instead of the real feature: a ruler in the photo, the background or a scanner tag.
Poor initialization or a large learning rate makes the loss blow up; a result from a single random seed may not reproduce.
Weights do not read as rules; that is a serious problem where a justification is required, as in credit or medical decisions.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Neural networks imitate the brain. | Biology is the inspiration, but the artificial neuron is a very crude model and its learning rule differs from the brain's. |
| More neurons are always better. | A large network can memorize the training data; size is chosen with a validation set. |
| Backpropagation is a separate learning method. | Backpropagation only computes the gradient; gradient descent updates the weights. |
| Training gives the same result every time. | Random initialization and shuffling can lead to different local minima. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Haykin, Neural Networks and Learning Machines, 3rd ed., Pearson, 2009.
- Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.
- Nielsen, Neural Networks and Deep Learning, Determination Press, 2015 (online book).
- Rumelhart, Hinton and Williams, "Learning representations by back-propagating errors", Nature 323, 1986.