Artificial Neural Networks

Guide to AI and Data Courses · Part I: Core
Artificial neuronPerceptronMultilayer networksBackpropagationTraining and generalizationNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How do networks of simple units learn by adjusting their weights?

Artificial neural networks are models built by connecting, in layers, simple units that take a weighted sum and pass it through an activation function. Learning means adjusting the weights so as to reduce a loss function; the main tools for this are backpropagation and gradient descent.

At a glance

Approach

network of simple weighted units

Input

numeric feature vector

Output

class or numeric prediction

Success measure

loss and accuracy

Math load

medium-high: derivatives, matrices

Data needs

moderate

Introduction

A Short History

1943

McCulloch and Pitts propose the first artificial neuron model with threshold logic units.

1949

Hebb suggests that connections between neurons that fire together grow stronger: the Hebbian rule.

1957

Rosenblatt introduces the perceptron; its weights are tuned from examples.

1969

Minsky and Papert show that a single-layer perceptron cannot solve problems such as XOR.

1982

Hopfield publishes a recurrent network that recalls memories by minimizing energy.

1986

Rumelhart, Hinton and Williams popularize backpropagation; multilayer networks become trainable.

The Course

Core Concepts

Artificial neuron

Weighted sum of inputs, a bias and an output.

Activation

Sigmoid, tanh, ReLU: adds nonlinearity to the network.

Perceptron

Single-layer network that separates linearly separable classes.

Multilayer network

Hidden layers learn nonlinear boundaries.

Forward pass

Computing the prediction layer by layer from input to output.

Backpropagation

Finding the gradient of every weight with the chain rule.

Gradient descent

Moves the weights in small steps where the loss decreases.

Regularization

Weight penalties and early stopping prevent overfitting.

The Course

A Typical Workflow

Initializeweightssmall randomvaluesForwardpasscomputepredictionMeasurelosserror,cross-entropyBack-propagategradientsUpdatelearning rate,epoch by epoch
Example: A small network that reads handwritten digits takes a 28×28-pixel image as 784 inputs, passes it through a hidden layer and outputs probabilities for 10 classes. A wrong guess means a large loss; backpropagation computes each weight's share and gradient descent corrects the weights a little.

The Course

What Does the Course Cover?

  1. Biological and artificial neurons; McCulloch-Pitts
  2. Hebbian learning and the perceptron algorithm
  3. Adaline, least mean squares and the delta rule
  4. Linear separability and the XOR problem
  5. Multilayer networks and activation functions
  6. Backpropagation: computational graphs, chain rule
  7. Loss functions: squared error, cross-entropy
  8. Gradient descent, learning rate and momentum
  9. Weight initialization and vanishing gradients
  10. Generalization: regularization, early stopping
  11. Radial basis function networks
  12. Hopfield networks and self-organizing maps (SOM)

Skills you will gain

writing a network from scratch in NumPyderiving backpropagation by handnumerical gradient checkingtraining networks in PyTorch and Kerasreading learning curves

A typical semester; the order and coverage vary between universities.

Try It

Tangerine or Lemon?

A camera on the sorting belt measures the weight and color hue of each fruit. A single artificial neuron (a perceptron) learns to separate the two kinds: train it, or click a fruit to flip its label.

● tangerine   ● lemon. Relabel a fruit deep among the others: no line can separate them, and the error never reaches zero.
Nobody wrote the rule: after each mistake the neuron nudged its weights toward the error. A single neuron can only draw a linear boundary; data that cannot be separated that way needs a multilayer network.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Machine Learningparametric model, loss function, gradient descent
Linear Algebraweight matrices, vectorized computation
Optimizationgradient descent, learning rate
Calculusderivatives, partial derivatives and the chain rule: the math of backpropagation
Programmingarrays, loops and functions in Python

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you differentiate a composite function with the chain rule?Derivatives and the chain rule from calculus
Can you multiply a matrix by a vector and state the result's shape?Linear Algebra
Can you do vectorized operations on NumPy arrays without loops?An intro programming course, then a short NumPy tutorial
Can you explain the difference between training and test error?Machine Learning
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
subfieldThis course is a subfield of Machine Learningparametric model, loss function, gradient descent
subfieldDeep Learning is a subfield of this coursemultilayer networks, backpropagation, activation functions
builds onThis course builds on Linear Algebraweight matrices, vectorized computation
builds onThis course builds on Optimizationgradient descent, learning rate
shares methodsShares methods with Evolutionary and Metaheuristic Optimizationneuroevolution, hyperparameter search

Differences

Where It Differs (1/2)

CriterionNeural NetworksMachine LearningDeep Learning
Core questionHow do networks of simple units learn by adjusting their weights?How does a program generalize from examples?How do many-layered networks learn useful representations from raw data?
Approachnetwork of simple weighted unitsstatistical learningmultilayer representation learning
Inputnumeric feature vectorlabeled or unlabeled examplesraw data: images, text, audio
Outputclass or numeric predictiona model that makes predictionslearned representation and prediction
Success measureloss and accuracyerror on unseen dataaccuracy, loss, scaling
Math loadmedium-high: derivatives, matriceshigh: linear algebra, probability, optimizationhigh: derivatives, matrices, probability
Data needsmoderatemoderate to largelarge; needs GPUs
  • Machine Learning: ML covers many model families from decision trees to SVMs; this course focuses only on networks built from neurons.
  • Deep Learning: Deep learning moves on to convolutional, recurrent and transformer architectures; this course builds the basic network and training they rest on.

Differences

Where It Differs (2/2)

CriterionNeural NetworksOptimizationEvolutionary Optimization
Core questionHow do networks of simple units learn by adjusting their weights?How do we find the values that make an objective function best?How do we find a good solution when there is no gradient or the search space is huge?
Approachnetwork of simple weighted unitsmathematical model of objective and constraintspopulation-based heuristic search
Inputnumeric feature vectorobjective function and constraintsobjective, solution encoding
Outputclass or numeric predictionoptimal or near-optimal solutiongood, not proven optimal, solution
Success measureloss and accuracyobjective value, convergence ratequality, evaluation count
Math loadmedium-high: derivatives, matricesthe theory itselflow-medium
Data needsmoderatenonenone; evaluations suffice
  • Optimization: Optimization studies gradient methods in general; this course uses them to learn weights.
  • Evolutionary Optimization: Neuroevolution searches weights or architectures by evolution; this course learns weights by gradients and backpropagation.

Practice

Applications

Digit recognition

Handwritten digits on postcodes and checks.

Tabular data

Classification tasks such as credit risk or churn.

Function approximation

Learning nonlinear relations from measurements.

Control

Simple robot and process controllers.

Signal classification

Detecting faults or motion from sensor data.

Associative memory

Completing a corrupted pattern with a Hopfield network.

Practice

Risks and Ethics

Memorization and overfitting

A large network on a small dataset drives training error to zero and then fails on new examples; without a validation set you never see it.

Shortcut learning

A network may learn traces of data collection instead of the real feature: a ruler in the photo, the background or a scanner tag.

Unstable training

Poor initialization or a large learning rate makes the loss blow up; a result from a single random seed may not reproduce.

Decisions without reasons

Weights do not read as rules; that is a serious problem where a justification is required, as in credit or medical decisions.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

NumPyPyTorchKerasscikit-learn (MLPClassifier)TensorFlow PlaygroundMATLAB Deep Learning Toolbox

Well-known datasets

XOR problemIrisMNISTFashion-MNISTTwo spiralsCIFAR-10
In this course, writing a network by hand in NumPy first is the best way to see what backpropagation really does.

Practice

Common Misconceptions

Common beliefIn fact
Neural networks imitate the brain.Biology is the inspiration, but the artificial neuron is a very crude model and its learning rule differs from the brain's.
More neurons are always better.A large network can memorize the training data; size is chosen with a validation set.
Backpropagation is a separate learning method.Backpropagation only computes the gradient; gradient descent updates the weights.
Training gives the same result every time.Random initialization and shuffling can lead to different local minima.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

Machine learning engineerDeep learning engineerResearcherEmbedded AI developer
This course is the gateway to deep learning: convolutional networks, recurrent networks and transformers are built on the ideas of neuron, loss and backpropagation introduced here. It also reinforces optimization and linear algebra.

Next Steps

Further Reading

  • Haykin, Neural Networks and Learning Machines, 3rd ed., Pearson, 2009.
  • Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.
  • Nielsen, Neural Networks and Deep Learning, Determination Press, 2015 (online book).
  • Rumelhart, Hinton and Williams, "Learning representations by back-propagating errors", Nature 323, 1986.

The End

Chapter 4: Artificial Neural Networks
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.