Machine Learning

Guide to AI and Data Courses · Part I: Core
Types of learningModel and lossGeneralizationEvaluationCore algorithmsNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How does a program generalize from examples?

Machine learning studies algorithms that learn from examples without being explicitly programmed and generalize what they learn to new examples. In Mitchell's definition, a program learns if its performance on task T, measured by P, improves with experience E.

At a glance

Approach

statistical learning

Input

labeled or unlabeled examples

Output

a model that makes predictions

Success measure

error on unseen data

Math load

high: linear algebra, probability, optimization

Data needs

moderate to large

Introduction

A Short History

1957

Rosenblatt introduces the perceptron, one of the first models to tune its weights from examples.

1959

Samuel's checkers program improves by playing itself; the term "machine learning" spreads.

1986

Quinlan publishes the ID3 decision tree; the same year backpropagation revives neural networks.

1995

Cortes and Vapnik publish support vector machines (SVM).

2001

Breiman introduces random forests: a vote among many trees.

2012

AlexNet wins the ImageNet challenge by a wide margin; the deep learning era begins.

The Course

Core Concepts

Supervised learning

Classification and regression from labeled examples.

Unsupervised learning

Clustering and dimensionality reduction on unlabeled data.

Model and hypothesis

The form and parameters of the function to be learned.

Loss function

Measures how wrong a prediction is; training minimizes it.

Generalization

Correct predictions on examples not seen in training.

Overfitting

Memorizing the training data; the bias-variance tradeoff.

Validation

Training, validation and test sets; cross-validation.

Features

Turning raw data into numbers a model can use.

The Course

A Typical Workflow

Collectdatasample,labelsPreparecleaning,featuresTrainminimizethe lossEvaluateheld-outtest setDeploy andmonitorproduction,data drift
Example: A spam filter collects past messages and their labels, turns word counts into features, trains a logistic regression, measures accuracy on held-out test messages and classifies new incoming mail. No rules are written by hand; they are learned from data.

The Course

What Does the Course Cover?

  1. The learning problem; supervised and unsupervised learning
  2. Linear regression and gradient descent
  3. Logistic regression and classification
  4. Evaluation: train/test split, cross-validation
  5. Overfitting, regularization, bias-variance
  6. Decision trees
  7. Ensembles: random forests, gradient boosting
  8. Support vector machines and kernels
  9. k-nearest neighbours and naive Bayes
  10. Clustering: k-means, hierarchical clustering
  11. Dimensionality reduction: PCA
  12. Feature engineering and model selection

Skills you will gain

Python and NumPydata preparation with pandasscikit-learncross-validationchoosing the right metricreporting results with plots and tables

A typical semester; the order and coverage vary between universities.

Try It

Should We Grant the Loan?

The bank knows the income, debt ratio and repayment outcome of past applications. Click the chart to add a new application; the model decides by looking at the k most similar past ones.

Click somewhere on the chart.
● repaid   ● defaulted. With k = 1, see how the regions split around single examples: that is overfitting.
Nobody wrote the rule: the decision comes entirely from examples. With a small k the model memorizes noise, with a large k it misses detail; the right k is chosen on held-out test data.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Artificial Intelligencean agent that learns from data; statistical generalization instead of search
Linear Algebrafeature vectors, matrix multiplication, PCA
Probability and Statisticsgeneralization error, bias-variance, probabilistic models
Optimizationloss minimization, regularization
Programmingreading data, loops and functions in Python or Java
Calculusderivatives, partial derivatives and the chain rule: the language of gradient descent

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you take a derivative and a partial derivative?Derivatives and the chain rule from calculus
Can you multiply matrices and take a dot product?Linear Algebra
Can you explain mean, variance and conditional probability?Probability and Statistics
Can you write loops and functions in Python and read a CSV file?An intro programming course, then a short pandas tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses (1/2)

RelationCourseShared ground
subfieldThis course is a subfield of Artificial Intelligencean agent that learns from data; statistical generalization instead of search
subfieldArtificial Neural Networks is a subfield of this courseparametric model, loss function, gradient descent
subfieldReinforcement Learning is a subfield of this courselearning from rewards instead of labels
builds onThis course builds on Linear Algebrafeature vectors, matrix multiplication, PCA
builds onThis course builds on Probability and Statisticsgeneralization error, bias-variance, probabilistic models
builds onThis course builds on Optimizationloss minimization, regularization
applicationMachine Learning Systems and MLOps applies the methods of this coursedeploying, monitoring and retraining models
applicationAI in Cybersecurity applies the methods of this courseintrusion detection, malware classification

Overlap

Overlapping Courses (2/2)

RelationCourseShared ground
applicationBioinformatics applies the methods of this coursegene expression classification, protein structure
shares methodsShares methods with Pattern Recognitionclassifiers, decision boundaries
shares methodsShares methods with Probabilistic Graphical Modelsprobabilistic classifiers, EM algorithm
shares methodsShares methods with Explainable AI and AI Ethicsmodel explanation, feature importance
shares methodsShares methods with Data Miningsame algorithms; the goal is discovery, not prediction
shares methodsShares methods with Data Sciencepredictive models
shares methodsShares methods with Information Theorycross-entropy loss, decision tree split criterion

Differences

Where It Differs (1/2)

CriterionMachine LearningArtificial IntelligenceData Mining
Core questionHow does a program generalize from examples?How does a machine act intelligently by searching, reasoning and planning?How are hidden, useful patterns found in large piles of data?
Approachstatistical learningsearch, logic, probability and learning combineddiscovery-driven analysis
Inputlabeled or unlabeled examplesproblem definition: states, actions, goallarge tables or transaction data
Outputa model that makes predictionsaction sequence, plan or decisionrules, clusters, outliers
Success measureerror on unseen datareaching the goal, cost, rationalityinterestingness, support, confidence
Math loadhigh: linear algebra, probability, optimizationmedium: logic, probability, graphsmedium
Data needsmoderate to largelittle; often a model and knowledge sufficelarge
  • Artificial Intelligence: AI is the framework; ML focuses only on learning from data within it.
  • Data Mining: They share algorithms; ML aims to predict new examples, data mining to discover patterns in data.

Differences

Where It Differs (2/2)

CriterionMachine LearningProbability & StatisticsDeep Learning
Core questionHow does a program generalize from examples?How is uncertainty measured, and how do we draw reliable conclusions from limited data?How do many-layered networks learn useful representations from raw data?
Approachstatistical learningprobability models and inferencemultilayer representation learning
Inputlabeled or unlabeled examplessample dataraw data: images, text, audio
Outputa model that makes predictionsestimate, confidence interval, test resultlearned representation and prediction
Success measureerror on unseen dataunbiasedness, power, significanceaccuracy, loss, scaling
Math loadhigh: linear algebra, probability, optimizationthe theory itselfhigh: derivatives, matrices, probability
Data needsmoderate to largea samplelarge; needs GPUs
  • Probability & Statistics: Statistics emphasizes a model's meaning and uncertainty, ML its predictive performance.
  • Deep Learning: Deep learning is a branch of ML; it learns features from raw data instead of hand-designing them.

Practice

Applications

Spam filtering

Classifying emails by their content.

Credit scoring

Predicting an applicant's risk of default.

Recommendation

Products or content a user is likely to enjoy.

Medical prediction

Estimating disease risk from patient data.

Demand forecasting

Sales and inventory planning.

Fraud detection

Catching unusual card transactions.

Practice

Risks and Ethics

Biased data, biased decisions

Discrimination in historical data passes straight into the model: hiring or credit models can systematically exclude some groups.

Data leakage

When test information leaks into training, a model looks flawless on paper and fails in the field.

Privacy

Personal information in the training data can be extracted from the model; GDPR and similar laws limit how data may be used.

Misplaced confidence

High accuracy collapses when the data distribution shifts, and the model looks just as confident on cases it does not know.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

scikit-learnpandasNumPyXGBoostWekaR

Well-known datasets

IrisMNISTUCI Machine Learning RepositoryTitanic (Kaggle)Adult (income)Wine

Practice

Common Misconceptions

Common beliefIn fact
A more complex model is always better.A complex model can memorize the training data (overfitting) and do worse on new data.
Training accuracy shows success.Success is measured on test data never used in training.
More data solves every problem.More biased or noisy data only multiplies the errors.
The model learns cause and effect.ML finds correlations; causation needs experiments or causal inference.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

Machine learning engineerData scientistResearcherMLOps engineer
Machine learning is the crossroads of the map: neural networks and reinforcement learning are its branches; MLOps, cybersecurity and bioinformatics apply its methods. After this course you can go deeper (neural networks, deep learning) or move on to an application area.

Next Steps

Further Reading

  • Mitchell, Machine Learning, McGraw-Hill, 1997.
  • Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
  • Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
  • Alpaydın, Introduction to Machine Learning, 4th ed., MIT Press, 2020.
  • Géron, Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow, 3rd ed., O'Reilly, 2022.

The End

Chapter 3: Machine Learning
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.