Probability and Statistics

Guide to AI and Data Courses · Part V: Mathematical Foundations
Probability and BayesDistributionsEstimationHypothesis testingWhere in AINeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How is uncertainty measured, and how do we draw reliable conclusions from limited data?

Probability is the language for measuring uncertainty; statistics uses it to draw general conclusions from limited data. Every model in AI and data courses works under uncertainty, and only this course can say how far a prediction can be trusted.

At a glance

Approach

probability models and inference

Input

sample data

Output

estimate, confidence interval, test result

Success measure

unbiasedness, power, significance

Math load

the theory itself

Data needs

a sample

Introduction

A Short History

1654

Pascal and Fermat exchange letters on dividing the stakes of an unfinished game of chance.

1763

Bayes's essay is published after his death: probability updated by evidence.

1908

Gosset, writing as "Student", publishes the t distribution for small samples.

1925

Fisher's Statistical Methods for Research Workers popularizes significance testing.

1933

Kolmogorov builds probability on axioms; probability theory rests on measure theory.

1979

Efron proposes the bootstrap: measuring uncertainty by resampling.

The Course

Core Concepts

Conditional probability

How a probability changes given information.

Bayes' rule

Turning a prior belief into a posterior with evidence.

Random variable

Expected value, variance and distribution.

Normal distribution

Central limit theorem: averages tend to normal.

Maximum likelihood

Choosing the parameter that makes the data most likely.

Confidence interval

Reporting an estimate's uncertainty as a range.

Hypothesis testing

Is the observed difference chance or real?

Regression

Explaining one variable by others, and testing it.

The Course

A Typical Workflow

Askwhat do wewant to know?SamplerandomsampleModeldistributionassumptionEstimateparameter,intervalDecidetest, risk
Example: An AI team asks whether a new recommendation model raises clicks. Half of the users see the new model at random (an A/B test), click rates are estimated, a confidence interval for the difference is computed, and if chance cannot explain the difference the model ships.

The Course

What Does the Course Cover?

  1. Counting, probability axioms, conditional probability
  2. Bayes' rule and independence
  3. Discrete distributions: Bernoulli, binomial, Poisson
  4. Continuous distributions: uniform, exponential, normal
  5. Expectation, variance and covariance
  6. Joint and conditional distributions
  7. Law of large numbers and central limit theorem
  8. Point estimation: method of moments, maximum likelihood
  9. Confidence intervals and the bootstrap
  10. Hypothesis testing: t-test, chi-square, p-values
  11. Linear regression and inference
  12. Bayesian inference: priors and posteriors

Skills you will gain

running tests with scipy.statsregression in R or statsmodelscomputing confidence intervalschecking probabilities by simulationdesigning an A/B test

A typical semester; the order and coverage vary between universities.

Try It

Does the New Recommender Work?

Does the new recommendation model raise the purchase rate? Half of the visitors see the old model (A), half the new one (B). Change the sample size and the true difference, and repeat the experiment.

With a small sample the intervals are wide: a real improvement comes out “not significant” in most experiments. Computing the required n before starting the test keeps you from throwing away a good model.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Calculusintegrals and derivatives: continuous distributions and maximum likelihood
Discrete Mathematicssets and counting (permutations, combinations)

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you integrate a function to find the area under a curve?Definite integrals from calculus
Can you find the maximum of a function by taking its derivative?Derivatives and extreme values from calculus
Can you count selections with permutations and combinations?Counting from discrete mathematics
Can you compute the mean of a list in Python and plot its histogram?An intro programming course, then a short NumPy tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
builds onArtificial Intelligence builds on this coursereasoning under uncertainty, Bayes rule
builds onMachine Learning builds on this coursegeneralization error, bias-variance, probabilistic models
builds onPattern Recognition builds on this courseBayesian decision theory, density estimation
builds onProbabilistic Graphical Models builds on this courseconditional independence, joint distribution
builds onData Mining builds on this coursesupport, confidence, significance
builds onData Science builds on this coursehypothesis testing, regression, visualization
builds onTime Series Analysis and Forecasting builds on this coursestationarity, autocorrelation, ARIMA
shares methodsShares methods with Information Theoryentropy, mutual information

Differences

Where It Differs (1/2)

CriterionProbability & StatisticsMachine LearningData Science
Core questionHow is uncertainty measured, and how do we draw reliable conclusions from limited data?How does a program generalize from examples?How do we ask the right questions of data and turn the answers into decisions?
Approachprobability models and inferencestatistical learningquestion, data, analysis, story
Inputsample datalabeled or unlabeled examplesraw, messy data
Outputestimate, confidence interval, test resulta model that makes predictionsreport, visualization, model
Success measureunbiasedness, power, significanceerror on unseen dataimpact on decisions, accuracy
Math loadthe theory itselfhigh: linear algebra, probability, optimizationmedium: statistics
Data needsa samplemoderate to largemoderate to large
  • Machine Learning: ML puts predictive performance first; statistics asks about a model's meaning, assumptions and uncertainty.
  • Data Science: Data science covers the whole process (data, analysis, communication); this course gives the theory of inference behind it.

Differences

Where It Differs (2/2)

CriterionProbability & StatisticsGraphical ModelsInformation Theory
Core questionHow is uncertainty measured, and how do we draw reliable conclusions from limited data?How do we model and query many interdependent, uncertain variables?How is information measured, and how is it compressed and transmitted with the least loss?
Approachprobability models and inferenceprobability and graphs combinedmeasuring information with probability
Inputsample datavariables, dependencies, datasource and channel model
Outputestimate, confidence interval, test resultconditional probability, best explanationbounds, codes, compression rate
Success measureunbiasedness, power, significancelikelihood, inference accuracybits, channel capacity
Math loadthe theory itselfhigh: probabilityhigh: probability
Data needsa samplemoderatenone
  • Graphical Models: Probabilistic graphical models structure multivariate distributions with graphs; this course starts with one or a few variables.
  • Information Theory: Information theory measures uncertainty with entropy; this course is built on estimation and decisions with probability.

Practice

Applications

A/B testing

Whether a new model is really better.

Naive Bayes

Text and spam classification.

Uncertainty

Confidence intervals and calibration of predictions.

Model comparison

Is the gap between two models significant?

Generative models

Learning a data distribution and sampling it.

Medical tests

Sensitivity, specificity and base rate.

Practice

Risks and Ethics

p-hacking

Trying many analysis choices and reporting only the "significant" one makes chance look like a finding.

Multiple testing

At the 0.05 level, a hundred comparisons yield about five false positives; without a correction, spurious findings appear.

Simpson's paradox

A trend can reverse when groups are combined; the 1973 Berkeley graduate admissions data is the well-known example.

Selection bias

A non-random sample misleads however large it is, as the 1936 Literary Digest poll showed.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

RSciPy (scipy.stats)statsmodelspandasPyMCStan

Classic example problems

Monty HallBirthday paradoxMedical test and base rateGalton's height dataA/B testGambler's fallacy

Practice

Common Misconceptions

Common beliefIn fact
If p = 0.03, the hypothesis is 97% likely to be true.The p-value is the probability of data this extreme if the null hypothesis holds; it is not the probability of the hypothesis.
Correlation implies causation.A common cause or chance also produces correlation; causation needs an experiment.
If a test is 99% accurate, a positive means 99% disease.If the disease is rare, most positives may be false alarms; Bayes' rule brings in the base rate.
With enough data, statistics is unnecessary.In big data even trivial differences come out "significant"; effect size and bias still matter.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

StatisticianData scientistMachine learning researcherBiostatisticianActuary
After this course, machine learning, probabilistic graphical models, pattern recognition and information theory become easier to approach. Bayesian statistics and causal inference are two natural next steps that treat uncertainty and cause and effect in more depth.

Next Steps

Further Reading

  • Wasserman, All of Statistics, Springer, 2004.
  • DeGroot and Schervish, Probability and Statistics, 4th ed., Pearson, 2012.
  • Blitzstein and Hwang, Introduction to Probability, 2nd ed., CRC Press, 2019.
  • Gelman, Carlin, Stern, Dunson, Vehtari and Rubin, Bayesian Data Analysis, 3rd ed., CRC Press, 2013.

The End

Chapter 32: Probability and Statistics
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.