Information Theory
Introduction
What Question Does It Ask?
Information theory measures the information in a message by probability: an unlikely event carries more information, and the average information is called entropy. The course derives the limits on how far data can be compressed and how fast it can be sent reliably over a noisy channel.
At a glance
measuring information with probability
source and channel model
bounds, codes, compression rate
bits, channel capacity
high: probability
none
Introduction
A Short History
Hartley proposes measuring information by the logarithm of the number of possible messages.
Shannon defines entropy and channel capacity in "A Mathematical Theory of Communication".
Kullback and Leibler publish a divergence measure between two probability distributions.
Huffman publishes the algorithm that builds the prefix code with minimum average length.
Lempel and Ziv publish the dictionary-based LZ77 compressor, ancestor of ZIP and PNG.
Turbo codes arrive: a practical code comes within about half a decibel of the Shannon limit.
The Course
Core Concepts
Average information per symbol of a source, in bits.
Uncertainty left in one variable once the other is known.
How much two variables tell about each other.
Extra cost of coding one distribution with a code built for another.
The most common loss function in classification.
Lossless compression: Huffman, arithmetic coding, LZ.
Highest rate that can pass reliably through a noisy channel.
Adding redundancy to detect and correct errors.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Entropy, joint and conditional entropy
- Mutual information and KL divergence
- Jensen's and the data processing inequalities
- Typical sets and the AEP
- Source coding theorem and Kraft inequality
- Huffman codes and arithmetic coding
- Universal compression: Lempel-Ziv
- Channel capacity and the channel coding theorem
- Differential entropy and continuous variables
- The Gaussian channel and Shannon-Hartley
- Rate-distortion theory and lossy compression
- Error-correcting codes: Hamming, LDPC
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Send the Message in Fewer Bits
A battery-powered sensor sends its messages by radio, and every bit costs energy. Type a text or pick a preset: the Huffman tree gives frequent letters short codes. Click a letter in the tree to see its code.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Discrete Mathematics | logarithms, sets, counting and binary representation |
| Algorithms | greedy algorithms and priority queues: building a Huffman code |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you use the rules of logarithms and compute in base 2? | Logarithms from high school maths |
| Can you explain conditional probability and expected value? | Probability and Statistics |
| Can you write a number in binary and tell how many bits it needs? | Number bases from discrete mathematics |
| Can you write a greedy algorithm that uses a priority queue? | Greedy methods and heaps from an algorithms course |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | Generative Models builds on this course | KL divergence, likelihood |
| shares methods | Shares methods with Machine Learning | cross-entropy loss, decision tree split criterion |
| shares methods | Shares methods with Digital Signal Processing | compression, sampling, channels |
| shares methods | Shares methods with Probability and Statistics | entropy, mutual information |
Differences
Where It Differs (1/2)
| Criterion | Information Theory | Probability & Statistics | Signal Processing |
|---|---|---|---|
| Core question | How is information measured, and how is it compressed and transmitted with the least loss? | How is uncertainty measured, and how do we draw reliable conclusions from limited data? | How are sampled signals analyzed, filtered and transformed? |
| Approach | measuring information with probability | probability models and inference | mathematical transforms and filters |
| Input | source and channel model | sample data | sampled signal |
| Output | bounds, codes, compression rate | estimate, confidence interval, test result | filtered signal, spectrum |
| Success measure | bits, channel capacity | unbiasedness, power, significance | signal-to-noise ratio, frequency response |
| Math load | high: probability | the theory itself | high: complex numbers, Fourier |
| Data needs | none | a sample | no learning; the signal suffices |
- Probability & Statistics: Probability and statistics describe distributions and uncertainty; this course derives information content and transmission limits from them.
- Signal Processing: Signal processing filters and transforms a signal; this course asks how much information it can carry and how fast.
Differences
Where It Differs (2/2)
| Criterion | Information Theory | Machine Learning | Generative Models |
|---|---|---|---|
| Core question | How is information measured, and how is it compressed and transmitted with the least loss? | How does a program generalize from examples? | How does a model generate new samples that look like real data? |
| Approach | measuring information with probability | statistical learning | learning the data distribution |
| Input | source and channel model | labeled or unlabeled examples | huge, mostly unlabeled data |
| Output | bounds, codes, compression rate | a model that makes predictions | new samples: images, text, audio |
| Success measure | bits, channel capacity | error on unseen data | realism and diversity (FID etc.) |
| Math load | high: probability | high: linear algebra, probability, optimization | high |
| Data needs | none | moderate to large | very large |
- Machine Learning: ML uses cross-entropy as a loss function; this course explains where that measure comes from and what it means.
- Generative Models: Generative models minimize KL divergence in training; this course studies the divergence itself and its link to code length.
Practice
Applications
ZIP, gzip and PNG: lossless compression.
JPEG and MP3: lossy compression.
Turbo codes in 4G, LDPC and polar codes in 5G.
Error-correcting codes in QR codes, CDs and SSDs.
Cross-entropy loss, information gain in decision trees.
The most informative features by mutual information.
Practice
Risks and Ethics
Histogram estimates of mutual information are biased upward on small samples; an irrelevant feature looks informative.
KL(p‖q) and KL(q‖p) give different models: one pushes to cover every mode, the other to lock onto a single mode.
Compressing secret data together with attacker-chosen text lets the size reveal the secret; the CRIME attack on TLS did this.
Treating symbols as independent overstates entropy; compression estimates and randomness assessments go wrong.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic example problems
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Information is measured by the meaning of a message. | Shannon information ignores meaning; it measures how unexpected a message is, that is, its probability. |
| A good program can shrink any file. | No lossless method shrinks every input; data cannot be compressed below its entropy. |
| Error-free transmission over a noisy channel is impossible. | Below capacity the error probability can be made as small as desired (Shannon). |
| KL divergence is a distance. | It is not symmetric and violates the triangle inequality: p to q and q to p give different values. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Cover and Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
- MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.
- Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27, 1948.
- Shannon and Weaver, The Mathematical Theory of Communication, University of Illinois Press, 1949.