Probability and Statistics
Introduction
What Question Does It Ask?
Probability is the language for measuring uncertainty; statistics uses it to draw general conclusions from limited data. Every model in AI and data courses works under uncertainty, and only this course can say how far a prediction can be trusted.
At a glance
probability models and inference
sample data
estimate, confidence interval, test result
unbiasedness, power, significance
the theory itself
a sample
Introduction
A Short History
Pascal and Fermat exchange letters on dividing the stakes of an unfinished game of chance.
Bayes's essay is published after his death: probability updated by evidence.
Gosset, writing as "Student", publishes the t distribution for small samples.
Fisher's Statistical Methods for Research Workers popularizes significance testing.
Kolmogorov builds probability on axioms; probability theory rests on measure theory.
Efron proposes the bootstrap: measuring uncertainty by resampling.
The Course
Core Concepts
How a probability changes given information.
Turning a prior belief into a posterior with evidence.
Expected value, variance and distribution.
Central limit theorem: averages tend to normal.
Choosing the parameter that makes the data most likely.
Reporting an estimate's uncertainty as a range.
Is the observed difference chance or real?
Explaining one variable by others, and testing it.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Counting, probability axioms, conditional probability
- Bayes' rule and independence
- Discrete distributions: Bernoulli, binomial, Poisson
- Continuous distributions: uniform, exponential, normal
- Expectation, variance and covariance
- Joint and conditional distributions
- Law of large numbers and central limit theorem
- Point estimation: method of moments, maximum likelihood
- Confidence intervals and the bootstrap
- Hypothesis testing: t-test, chi-square, p-values
- Linear regression and inference
- Bayesian inference: priors and posteriors
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Does the New Recommender Work?
Does the new recommendation model raise the purchase rate? Half of the visitors see the old model (A), half the new one (B). Change the sample size and the true difference, and repeat the experiment.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Calculus | integrals and derivatives: continuous distributions and maximum likelihood |
| Discrete Mathematics | sets and counting (permutations, combinations) |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you integrate a function to find the area under a curve? | Definite integrals from calculus |
| Can you find the maximum of a function by taking its derivative? | Derivatives and extreme values from calculus |
| Can you count selections with permutations and combinations? | Counting from discrete mathematics |
| Can you compute the mean of a list in Python and plot its histogram? | An intro programming course, then a short NumPy tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | Artificial Intelligence builds on this course | reasoning under uncertainty, Bayes rule |
| builds on | Machine Learning builds on this course | generalization error, bias-variance, probabilistic models |
| builds on | Pattern Recognition builds on this course | Bayesian decision theory, density estimation |
| builds on | Probabilistic Graphical Models builds on this course | conditional independence, joint distribution |
| builds on | Data Mining builds on this course | support, confidence, significance |
| builds on | Data Science builds on this course | hypothesis testing, regression, visualization |
| builds on | Time Series Analysis and Forecasting builds on this course | stationarity, autocorrelation, ARIMA |
| shares methods | Shares methods with Information Theory | entropy, mutual information |
Differences
Where It Differs (1/2)
| Criterion | Probability & Statistics | Machine Learning | Data Science |
|---|---|---|---|
| Core question | How is uncertainty measured, and how do we draw reliable conclusions from limited data? | How does a program generalize from examples? | How do we ask the right questions of data and turn the answers into decisions? |
| Approach | probability models and inference | statistical learning | question, data, analysis, story |
| Input | sample data | labeled or unlabeled examples | raw, messy data |
| Output | estimate, confidence interval, test result | a model that makes predictions | report, visualization, model |
| Success measure | unbiasedness, power, significance | error on unseen data | impact on decisions, accuracy |
| Math load | the theory itself | high: linear algebra, probability, optimization | medium: statistics |
| Data needs | a sample | moderate to large | moderate to large |
- Machine Learning: ML puts predictive performance first; statistics asks about a model's meaning, assumptions and uncertainty.
- Data Science: Data science covers the whole process (data, analysis, communication); this course gives the theory of inference behind it.
Differences
Where It Differs (2/2)
| Criterion | Probability & Statistics | Graphical Models | Information Theory |
|---|---|---|---|
| Core question | How is uncertainty measured, and how do we draw reliable conclusions from limited data? | How do we model and query many interdependent, uncertain variables? | How is information measured, and how is it compressed and transmitted with the least loss? |
| Approach | probability models and inference | probability and graphs combined | measuring information with probability |
| Input | sample data | variables, dependencies, data | source and channel model |
| Output | estimate, confidence interval, test result | conditional probability, best explanation | bounds, codes, compression rate |
| Success measure | unbiasedness, power, significance | likelihood, inference accuracy | bits, channel capacity |
| Math load | the theory itself | high: probability | high: probability |
| Data needs | a sample | moderate | none |
- Graphical Models: Probabilistic graphical models structure multivariate distributions with graphs; this course starts with one or a few variables.
- Information Theory: Information theory measures uncertainty with entropy; this course is built on estimation and decisions with probability.
Practice
Applications
Whether a new model is really better.
Text and spam classification.
Confidence intervals and calibration of predictions.
Is the gap between two models significant?
Learning a data distribution and sampling it.
Sensitivity, specificity and base rate.
Practice
Risks and Ethics
Trying many analysis choices and reporting only the "significant" one makes chance look like a finding.
At the 0.05 level, a hundred comparisons yield about five false positives; without a correction, spurious findings appear.
A trend can reverse when groups are combined; the 1973 Berkeley graduate admissions data is the well-known example.
A non-random sample misleads however large it is, as the 1936 Literary Digest poll showed.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic example problems
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| If p = 0.03, the hypothesis is 97% likely to be true. | The p-value is the probability of data this extreme if the null hypothesis holds; it is not the probability of the hypothesis. |
| Correlation implies causation. | A common cause or chance also produces correlation; causation needs an experiment. |
| If a test is 99% accurate, a positive means 99% disease. | If the disease is rare, most positives may be false alarms; Bayes' rule brings in the base rate. |
| With enough data, statistics is unnecessary. | In big data even trivial differences come out "significant"; effect size and bias still matter. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Wasserman, All of Statistics, Springer, 2004.
- DeGroot and Schervish, Probability and Statistics, 4th ed., Pearson, 2012.
- Blitzstein and Hwang, Introduction to Probability, 2nd ed., CRC Press, 2019.
- Gelman, Carlin, Stern, Dunson, Vehtari and Rubin, Bayesian Data Analysis, 3rd ed., CRC Press, 2013.