Information Theory

Guide to AI and Data Courses · Part V: Mathematical Foundations
EntropyCompressionChannel capacityError-correcting codesMeasures in MLNeighbouring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How is information measured, and how is it compressed and transmitted with the least loss?

Information theory measures the information in a message by probability: an unlikely event carries more information, and the average information is called entropy. The course derives the limits on how far data can be compressed and how fast it can be sent reliably over a noisy channel.

At a glance

Approach

measuring information with probability

Input

source and channel model

Output

bounds, codes, compression rate

Success measure

bits, channel capacity

Math load

high: probability

Data needs

none

Introduction

A Short History

1928

Hartley proposes measuring information by the logarithm of the number of possible messages.

1948

Shannon defines entropy and channel capacity in "A Mathematical Theory of Communication".

1951

Kullback and Leibler publish a divergence measure between two probability distributions.

1952

Huffman publishes the algorithm that builds the prefix code with minimum average length.

1977

Lempel and Ziv publish the dictionary-based LZ77 compressor, ancestor of ZIP and PNG.

1993

Turbo codes arrive: a practical code comes within about half a decibel of the Shannon limit.

The Course

Core Concepts

Entropy

Average information per symbol of a source, in bits.

Conditional entropy

Uncertainty left in one variable once the other is known.

Mutual information

How much two variables tell about each other.

KL divergence

Extra cost of coding one distribution with a code built for another.

Cross-entropy

The most common loss function in classification.

Source coding

Lossless compression: Huffman, arithmetic coding, LZ.

Channel capacity

Highest rate that can pass reliably through a noisy channel.

Error-correcting codes

Adding redundancy to detect and correct errors.

The Course

A Typical Workflow

Sourcemessage,probabilitiesCompresssourcecodingProtectchannel code,redundancyTransmitnoisychannelDecodefix errors,decompress
Example: A photo sent from a phone is first compressed with JPEG; the wireless channel code then adds controlled redundancy. Bits corrupted in the air are corrected with that redundancy at the receiver and the photo opens. Compression and protection look like opposites, yet both come from the same theory.

The Course

What Does the Course Cover?

  1. Entropy, joint and conditional entropy
  2. Mutual information and KL divergence
  3. Jensen's and the data processing inequalities
  4. Typical sets and the AEP
  5. Source coding theorem and Kraft inequality
  6. Huffman codes and arithmetic coding
  7. Universal compression: Lempel-Ziv
  8. Channel capacity and the channel coding theorem
  9. Differential entropy and continuous variables
  10. The Gaussian channel and Shannon-Hartley
  11. Rate-distortion theory and lossy compression
  12. Error-correcting codes: Hamming, LDPC

Skills you will gain

computing entropy and mutual informationwriting a Huffman codermeasuring compression with zlibcomputing channel capacitywriting proofs with inequalities

A typical semester; the order and coverage vary between universities.

Try It

Send the Message in Fewer Bits

A battery-powered sensor sends its messages by radio, and every bit costs energy. Type a text or pick a preset: the Huffman tree gives frequent letters short codes. Click a letter in the tree to see its code.

Information theory sets the limit: no lossless code can be shorter than the entropy on average. Huffman stays within one bit of it. When all letters are equally frequent, the saving comes only from the small alphabet.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Discrete Mathematicslogarithms, sets, counting and binary representation
Algorithmsgreedy algorithms and priority queues: building a Huffman code

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you use the rules of logarithms and compute in base 2?Logarithms from high school maths
Can you explain conditional probability and expected value?Probability and Statistics
Can you write a number in binary and tell how many bits it needs?Number bases from discrete mathematics
Can you write a greedy algorithm that uses a priority queue?Greedy methods and heaps from an algorithms course
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
builds onGenerative Models builds on this courseKL divergence, likelihood
shares methodsShares methods with Machine Learningcross-entropy loss, decision tree split criterion
shares methodsShares methods with Digital Signal Processingcompression, sampling, channels
shares methodsShares methods with Probability and Statisticsentropy, mutual information

Differences

Where It Differs (1/2)

CriterionInformation TheoryProbability & StatisticsSignal Processing
Core questionHow is information measured, and how is it compressed and transmitted with the least loss?How is uncertainty measured, and how do we draw reliable conclusions from limited data?How are sampled signals analyzed, filtered and transformed?
Approachmeasuring information with probabilityprobability models and inferencemathematical transforms and filters
Inputsource and channel modelsample datasampled signal
Outputbounds, codes, compression rateestimate, confidence interval, test resultfiltered signal, spectrum
Success measurebits, channel capacityunbiasedness, power, significancesignal-to-noise ratio, frequency response
Math loadhigh: probabilitythe theory itselfhigh: complex numbers, Fourier
Data needsnonea sampleno learning; the signal suffices
  • Probability & Statistics: Probability and statistics describe distributions and uncertainty; this course derives information content and transmission limits from them.
  • Signal Processing: Signal processing filters and transforms a signal; this course asks how much information it can carry and how fast.

Differences

Where It Differs (2/2)

CriterionInformation TheoryMachine LearningGenerative Models
Core questionHow is information measured, and how is it compressed and transmitted with the least loss?How does a program generalize from examples?How does a model generate new samples that look like real data?
Approachmeasuring information with probabilitystatistical learninglearning the data distribution
Inputsource and channel modellabeled or unlabeled exampleshuge, mostly unlabeled data
Outputbounds, codes, compression ratea model that makes predictionsnew samples: images, text, audio
Success measurebits, channel capacityerror on unseen datarealism and diversity (FID etc.)
Math loadhigh: probabilityhigh: linear algebra, probability, optimizationhigh
Data needsnonemoderate to largevery large
  • Machine Learning: ML uses cross-entropy as a loss function; this course explains where that measure comes from and what it means.
  • Generative Models: Generative models minimize KL divergence in training; this course studies the divergence itself and its link to code length.

Practice

Applications

File compression

ZIP, gzip and PNG: lossless compression.

Image and audio

JPEG and MP3: lossy compression.

Mobile networks

Turbo codes in 4G, LDPC and polar codes in 5G.

Data storage

Error-correcting codes in QR codes, CDs and SSDs.

Machine learning

Cross-entropy loss, information gain in decision trees.

Feature selection

The most informative features by mutual information.

Practice

Risks and Ethics

Mutual information from little data

Histogram estimates of mutual information are biased upward on small samples; an irrelevant feature looks informative.

The wrong KL direction

KL(p‖q) and KL(q‖p) give different models: one pushes to cover every mode, the other to lock onto a single mode.

Secrets leaked by compression

Compressing secret data together with attacker-chosen text lets the size reveal the secret; the CRIME attack on TLS did this.

Assuming independence

Treating symbols as independent overstates entropy; compression estimates and randomness assessments go wrong.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

PythonNumPySciPyzlibscikit-learnMATLAB

Classic example problems

Loaded die and coin tossBinary symmetric channelEntropy of English textBuilding a Huffman treeTwenty questions gameCanterbury corpus
Information theory is mostly done with proofs and calculation; code is written to measure entropy and to test compression ratios on real files.

Practice

Common Misconceptions

Common beliefIn fact
Information is measured by the meaning of a message.Shannon information ignores meaning; it measures how unexpected a message is, that is, its probability.
A good program can shrink any file.No lossless method shrinks every input; data cannot be compressed below its entropy.
Error-free transmission over a noisy channel is impossible.Below capacity the error probability can be made as small as desired (Shannon).
KL divergence is a distance.It is not symmetric and violates the triangle inequality: p to q and q to p give different values.

Next Steps

After This Course

Courses that build on or extend this one

Related careers

Communications engineerCompression and codec developerMachine learning researcherCryptography specialistData storage engineer
Information theory is the mathematical gateway to ML: cross-entropy, KL divergence and mutual information reappear in generative models, decision trees and representation learning. On the communications side, continue with digital signal processing and coding theory.

Next Steps

Further Reading

  • Cover and Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
  • MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.
  • Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27, 1948.
  • Shannon and Weaver, The Mathematical Theory of Communication, University of Illinois Press, 1949.

The End

Chapter 34: Information Theory
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.