Chapter 2: Greedy Algorithms

Advanced Algorithms
The greedy approachMaking changeActivity selectionJob sequencing Fractional knapsackMST and the cut propertyHuffman coding
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Approach

Define the greedy strategy; explain when the greedy-choice property and optimal substructure guarantee a correct result.

Classic problems

Solve making change, activity selection, job sequencing, and fractional knapsack greedily; give counterexamples where greedy fails.

Spanning trees

Explain why the cut property makes the greedy choices of Kruskal, Prim, and Borůvka correct; compare the three by selection rule, data structure, and running time.

Data compression

Build a Huffman tree, derive prefix-free codes, encode and decode a text, and compute the savings in bits.

The Greedy Approach

Winning Step by Step

  • Greedy algorithms proceed by choosing, at each step, the option that looks best right now.
  • Like a treasure hunter: at every stage, take the option with the largest immediate gain.
  • It works in stages: instead of looking at the problem as a whole, it focuses on the best option at hand.
  • A decision is never undone; there is no backtracking.
  • Like planning a trip by taking the road that looks shortest at every junction.
Gain and risk

Fast and simple; the code is usually short and clear.

However, it does not guarantee an optimal solution; in many cases it yields a good approximate one.

The Greedy Approach

The Greedy Paradigm: General Template

Solution greedy(Candidate[] candidates) {
  Solution s = new Solution(); // empty
  sort(candidates);        // by criterion
  for (Candidate a : candidates) {
    if (s.isFeasible(a))   // constraints hold
      s.add(a);            // never undone
  }
  return s;
}
  1. Define the set of feasible solutions (which choices keep the constraints satisfied?).
  2. Pick a greedy criterion: cheapest, earliest finishing, most profitable, most frequent…
  3. Choose the locally best candidate by the criterion; add it to the solution if feasible.
  4. Repeat until the solution is complete.
sorted candidatesa1✓a2✓a3✗a4✓a5✗a6✓one pass: each candidate seen once, no decision undone →solution = {a1, a2, a4, a6}
The cost of most greedy algorithms comes from sorting or priority queue operations: typically O(n log n).

The Greedy Approach

When Is Greedy Correct?

Greedy-choice property

There is at least one optimal solution that contains the locally best choice. The first greedy choice never leads us "down the wrong path".

Optimal substructure

Once a choice is made, an optimal solution of the remaining subproblem, together with that choice, gives an optimal solution of the whole problem.

Proof technique: the exchange argument

Take any optimal solution O. If O does not contain the greedy choice, exchange one of its elements for the greedy choice; the solution stays feasible and does not get worse. So an optimal solution containing the greedy choice exists; induction extends this to every step.

If neither property holds, the greedy algorithm is only a heuristic: fast, but its result may be worse than optimal.

The Greedy Approach

Advantages and Disadvantages

Advantages

  • Easy to understand and implement; short code.
  • Fast: usually a sort plus one pass, O(n log n).
  • Little memory: only the current solution is kept.
  • Optimal on the right problems (MST, Huffman, activity selection).
  • On hard problems it gives a good approximation or a starting point.

Disadvantages

  • No guarantee of optimality (making change with {1, 3, 4}, 0/1 knapsack).
  • Correctness must be proved separately for each problem.
  • The criterion is critical: the wrong one gives the wrong result.
  • Decisions are never undone; there is no chance to fix a mistake.

The Greedy Approach

The Change-Making Problem (Coin Change)

Pay a given amount with the fewest coins.

Greedy criterion: take the largest coin not exceeding the remaining amount; repeat until the amount is zero.

// coins[] sorted in decreasing order
static int makeChange(int[] coins, int amount) {
  int count = 0;
  for (int p : coins) {
    while (amount >= p) {
      amount -= p;
      count++;
    }
  }
  return count;
}
Is it always correct?

Yes for canonical coin systems such as {50, 25, 10, 5, 1}: 68 = 50 + 10 + 5 + 1 + 1 + 1.

With {4, 3, 1}, greedy pays 6 as 4 + 1 + 1 (3 coins); the optimum is 3 + 3 (2 coins). In general, dynamic programming is needed.

Time: O(k + count), where k is the number of coin types.

The Greedy Approach · Run the code step by step

makeChange({4, 3, 1}, 10): Code Trace

// coins[] sorted in decreasing order
static int makeChange(int[] coins, int amount) {
  int count = 0;
  for (int p : coins) {
    while (amount >= p) {
      amount -= p;
      count++;
    }
  }
  return count;
}
coins431amountleft 10count0

The coins {4, 3, 1} are sorted in decreasing order; the amount is 10. This system is not canonical, so the greedy result may not be optimal.

coins431amountleft 10count0

p = 4: 10 ≥ 4 → this coin can be used.

coins431amount4left 6count1

amount −= 4 → 6, count = 1. The loop continues.

coins431amount44left 2count2

amount −= 4 → 2, count = 2. 2 < 4: the while loop ends.

coins431amount44left 2count2

p = 3: 2 < 3 → the 3-coin is never used; move on to the next coin.

coins431amount44left 2count2

p = 1: 2 ≥ 1 → this coin can be used.

coins431amount441left 1count3

amount −= 1 → 1, count = 3. The loop continues.

coins431amount4411left 0count4

amount −= 1 → 0, count = 4. 0 < 1: the while loop ends.

coins431amount4411left 0count4optimal4333 coins

Greedy returns 4 + 4 + 1 + 1 = 4 coins, yet 4 + 3 + 3 = 3 coins suffice (found by DP). Taking two 4-coins first blocked the use of the 3-coins.

The Greedy Approach · Step by step

Making Change: Step by Step

coins50251051chosenremaining68coins used: 0

Example 1: coins {50, 25, 10, 5, 1}, amount 68. Start with the largest coin.

coins50251051chosen50remaining18coins used: 1

The largest coin that fits is 50 → 68 − 50 = 18 left.

coins50251051chosen5010remaining8coins used: 2

The largest coin that fits is 10 → 18 − 10 = 8 left.

coins50251051chosen50105remaining3coins used: 3

The largest coin that fits is 5 → 8 − 5 = 3 left.

coins50251051chosen501051remaining2coins used: 4

The largest coin that fits is 1 → 3 − 1 = 2 left.

coins50251051chosen5010511remaining1coins used: 5

The largest coin that fits is 1 → 2 − 1 = 1 left.

coins50251051chosen50105111remaining0coins used: 6

The largest coin that fits is 1 → 1 − 1 = 0 left. 6 coins in total — optimal for this coin system.

coins431chosenremaining6coins used: 0

Example 2: coins {4, 3, 1}, amount 6. Start with the largest coin.

coins431chosen4remaining2coins used: 1

The largest coin that fits is 4 → 6 − 4 = 2 left.

coins431chosen41remaining1coins used: 2

The largest coin that fits is 1 → 2 − 1 = 1 left.

coins431chosen411remaining0coins used: 3

The largest coin that fits is 1 → 1 − 1 = 0 left.

coins431chosen33remaining0coins used: 2optimal: 2 coins

Greedy gave 4 + 1 + 1 = 3 coins, yet 3 + 3 = 2 coins suffice. The locally best choice (4) missed the global optimum.

The Greedy Approach

Try It: Greedy vs. Optimal Change

The optimum is found by dynamic programming: minCoins[t] = 1 + min minCoins[t − p]. Coins are integers from 1 to 999, at most 8 kinds; the amount is at most 999. Enter = compute.

The Greedy Approach

The Greedy Approach vs. Divide and Conquer

GreedyDivide and conquer
Decisiona single locally best choice per step; never undonesplit by a fixed rule; no choice is made
Subproblemsone subproblem remains after the choiceseveral independent subproblems
Correctnessgreedy-choice property + optimal substructure must be proveddepends on the correctness of the combine step
Costusually O(n log n): sort + one passvia a recurrence and the master theorem
Exampleactivity selection, Kruskal, Huffmanmerge sort, Karatsuba
Both shrink the problem: divide and conquer solves all the pieces and combines them; greedy makes one choice and continues with the single remaining subproblem.

Activity Selection

Activity Selection

  • Each activity has a start time si and a finish time fi; only one activity can run at a time (one hall).
  • Goal: select the maximum number of mutually compatible activities. [si, fi) and [sj, fj) are compatible ⇔ sj ≥ fi or si ≥ fj.
  • Greedy criterion: select the activity that finishes earliest, discard those that overlap it, and repeat with the rest.
Activity12345678910
start12465803710
finish35687924912
Time: sorting O(n log n) + one pass O(n). If the activities are already sorted by finish time, O(n).

Activity Selection · Run the code step by step

select(e): Code Trace

024681012#7#1#8#2#3#5#4#6#9#10

The activities are sorted by finish time (top to bottom). Dashed line: finish of the last selected activity.

024681012#7#1#8#2#3#5#4#6#9#10

The earliest-finishing activity (#7, [0, 2)) is always selected.

024681012#7#1#8#2#3#5#4#6#9#10

#1 [1, 3): start 1 < 2 → overlaps, skipped.

024681012#7#1#8#2#3#5#4#6#9#10

#8 [3, 4): start 3 ≥ 2 → selected, i = 2.

024681012#7#1#8#2#3#5#4#6#9#10

#2 [2, 5): start 2 < 4 → overlaps, skipped.

024681012#7#1#8#2#3#5#4#6#9#10

#3 [4, 6): start 4 ≥ 4 → selected, i = 4.

024681012#7#1#8#2#3#5#4#6#9#10

#5 [5, 7): start 5 < 6 → overlaps, skipped.

024681012#7#1#8#2#3#5#4#6#9#10

#4 [6, 8): start 6 ≥ 6 → selected, i = 6.

024681012#7#1#8#2#3#5#4#6#9#10

#6 [8, 9): start 8 ≥ 8 → selected, i = 7.

024681012#7#1#8#2#3#5#4#6#9#10

#9 [7, 9): start 7 < 9 → overlaps, skipped.

024681012#7#1#8#2#3#5#4#6#9#10

#10 [10, 12): start 10 ≥ 9 → selected, i = 9. Result: 6 activities (7, 8, 3, 4, 6, 10).

static void select(Activity[] e) {
  Arrays.sort(e, (x, y) -> x.finish - y.finish);
  int i = 0;                 // last selected
  System.out.print(e[i].id + " ");
  for (int j = 1; j < e.length; j++) {
    if (e[j].start >= e[i].finish) {
      System.out.print(e[j].id + " ");
      i = j;
    }
  }
}

Activity Selection

Why Earliest Finish?

Other natural criteria do not guarantee an optimal result (orange: greedy's choice, green: optimal solution):

Earliest startgreedy 1 · optimal 3Shortest firstgreedy 1 · optimal 20246810

The "fewest overlaps" criterion has a counterexample too.

Exchange argument

Let a1 be the earliest-finishing activity and O an optimal solution. If the first activity of O is b ≠ a1, replace b with a1: since f(a1) ≤ f(b), it overlaps no other activity of O, and the count stays the same.

So some optimal solution contains a1. After discarding the activities that overlap a1, the remaining problem has the same structure (optimal substructure).

Job Sequencing

Job Sequencing with Deadlines

  • Each job takes 1 unit of time and has a deadline and a profit.
  • The profit is earned only if the job finishes by its deadline; one job runs at a time.
  • Goal: maximize the total profit.
  • Greedy criterion: take the jobs in decreasing order of profit; put each in the latest free slot before its deadline; if no slot is free, skip the job.
  • Choosing the latest slot leaves the early slots for jobs with closer deadlines.
Jobdeadlineprofit
1420
2110
3140
4130
5350
6230

Time: sorting O(n log n) + slot search O(n · dmax) → worst case O(n²). With Union-Find, the slot search drops to nearly O(n).

Sorting jobs by deadline is wrong: for (profit 10, deadline 1) and (profit 40, deadline 1), deadline order earns 10, profit order earns 40. Decreasing profit order is optimal: the feasible job sets form a matroid.

Job Sequencing · Run the code step by step

jobSequencing(jobs, 6): Code Trace

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1—2—3—4—5—6—

The jobs are sorted in decreasing order of profit (Job 4 and Job 6, with equal profit, keep their original order). All slots are empty (−1).

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1—2—3Job 54—5—6—

Job 5 (deadline 3, profit 50): the latest feasible slot 3 is free → placed. total = 50.

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1Job 32—3Job 54—5—6—

Job 3 (deadline 1, profit 40): the latest feasible slot 1 is free → placed. total = 90.

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1Job 32—3Job 54—5—6—

Job 4 (deadline 1, profit 30): slot 1 is taken (Job 3) and there is no earlier slot → the job is skipped.

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1Job 32Job 63Job 54—5—6—

Job 6 (deadline 2, profit 30): the latest feasible slot 2 is free → placed. total = 120.

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1Job 32Job 63Job 54Job 15—6—

Job 1 (deadline 4, profit 20): the latest feasible slot 4 is free → placed. total = 140.

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1Job 32Job 63Job 54Job 15—6—

Job 2 (deadline 1, profit 10): slot 1 is taken (Job 3) and there is no earlier slot → the job is skipped.

jobs sorted by profitJob 5t ≤ 3profit 50Job 3t ≤ 1profit 40Job 4t ≤ 1profit 30Job 6t ≤ 2profit 30Job 1t ≤ 4profit 20Job 2t ≤ 1profit 10slot (time slot)1Job 32Job 63Job 54Job 15—6—

Result: slots 1–4 = Job 3, Job 6, Job 5, Job 1 → total profit 140. Job 4 and Job 2 cannot be scheduled.

static int jobSequencing(Job[] jobs, int n) {
  Arrays.sort(jobs, (a, b) -> b.profit - a.profit);
  int[] slot = new int[n];
  Arrays.fill(slot, -1);
  int total = 0;
  for (int i = 0; i < n; i++) {
    int j = Math.min(n, jobs[i].deadline) - 1;
    for (; j >= 0; j--) {
      if (slot[j] == -1) {
        slot[j] = jobs[i].id;
        total += jobs[i].profit;
        break;
      }
    }
  }
  return total;
}

Knapsack

Fractional Knapsack

Capacity W = 50. If items can be split, the greedy criterion is: take the one with the largest profit/weight ratio.

Itemprofitweightratio
160106
2100205
3120304

Fractional: items 1 + 2 in full, 20/30 of item 3 → 60 + 100 + 80 = 240 (optimal). Time O(n log n).

Why optimal?

Each unit of the highest-ratio item earns at least as much as any other option of the same weight. If an optimal solution does not take as much of this item as it can, swap in an equal weight of it for another item; the profit does not drop (exchange argument).

If items cannot be split (0/1 knapsack), the ratio criterion does not guarantee the optimum; that problem is solved with dynamic programming.

fractional123 · 20/30profit 2400W = 50

Knapsack · Run the code step by step

fractional(s, 50): Code Trace

itemprofitweightratio16010621002053120304knapsack W = 50cap = 50profit = 0

Items sorted by decreasing profit/weight ratio: 1 (6), 2 (5), 3 (4). The knapsack is empty; remaining capacity 50.

itemprofitweightratio16010621002053120304knapsack W = 50cap = 50profit = 0

Item 1: weight 10 ≤ cap 50 → fits entirely.

itemprofitweightratio16010621002053120304knapsack W = 501cap = 40profit = 60

cap −= 10 → 40, profit += 60 → 60.

itemprofitweightratio16010621002053120304knapsack W = 501cap = 40profit = 60

Item 2: weight 20 ≤ cap 40 → fits entirely.

itemprofitweightratio16010621002053120304knapsack W = 5012cap = 20profit = 160

cap −= 20 → 20, profit += 100 → 160.

itemprofitweightratio16010621002053120304knapsack W = 5012cap = 20profit = 160

Item 3: weight 30 > cap 20 → does not fit entirely, else branch.

itemprofitweightratio16010621002053120304knapsack W = 5012320/30knapsack fullprofit = 240

For the remaining 20 units, 20/30 of item 3: 120 · 20/30 = 80 → profit = 240. The knapsack is full, break.

itemprofitweightratio16010621002053120304knapsack W = 5012320/30knapsack fullprofit = 240

Result 240: optimal for the fractional problem. The cost comes from sorting: O(n log n); the scan is O(n).

static double fractional(Item[] s, int W) {
  Arrays.sort(s, (a, b) -> Double.compare(b.ratio(), a.ratio()));
  double profit = 0;
  int cap = W;
  for (Item x : s) {
    if (x.weight <= cap) {           // fits entirely
      cap -= x.weight;
      profit += x.profit;
    } else {                         // fractional part
      profit += x.profit * (double) cap / x.weight;
      break;
    }
  }
  return profit;
}

Minimum Spanning Tree

Minimum Spanning Trees and the Cut Property

  • Minimum spanning tree (MST): a tree connecting all vertices of a connected, weighted, undirected graph with the minimum total weight; V − 1 edges.
  • Cut property: if the vertices are split into S and V − S, the lightest edge crossing this cut belongs to an MST.
  • Kruskal, Prim, and Borůvka are greedy algorithms that use this property with different cuts: every edge added is the lightest edge crossing some cut, so no choice is ever undone.
  • If the weights are distinct, the MST is unique.
43562794811pqrstuSV − S

Edges crossing the cut: 9, 4, 8, 11. The lightest, r–s (4), is in the MST.

Minimum Spanning Tree

Kruskal, Prim, and Borůvka

KruskalPrimBorůvka
Greedy choicelightest edge in the whole graph that forms no cyclelightest edge leaving the treelightest edge leaving each component
Growthforest → treea single tree, vertex by vertexa forest, merging in rounds
Cut usedthe two components separating the edge's endpointsthe tree vs. the remaining verticeseach component vs. the rest
Data structuresorting + Union-Findpriority queueUnion-Find
TimeO(E log V)O(E log V) (binary heap)O(E log V)
Best forsparse graphs, edge listdense graphs, adjacencyparallel / distributed
All three find the same MST on the same graph (it is unique when weights are distinct); they differ in the order they consider edges and in the data structure they need.

Huffman Coding

Huffman Coding

  • A lossless data compression method (David Huffman, 1952).
  • Uses variable-length codes: frequent characters get short codes, rare ones get long codes.
  • Produces prefix-free codes: no code is a prefix of another → decodes unambiguously without separators.
  • Greedy step: merge the two lowest-frequency trees.
Why prefix-free?

Let a = 0, b = 01, c = 1. Is "01" "ab" or "b"? Ambiguous: 0 is a prefix of 01.

Huffman codes sit at the leaves of a binary tree, so no code can be a prefix of another.

011acb✗ a is on b's path0101cab✓ all leaves: 00, 01, 1
Used in: the entropy-coding stage of ZIP/DEFLATE, gzip, PNG, JPEG, and MP3.

Huffman Coding

Huffman Coding: Steps

  1. Step 1: Count the frequency of each character.
  2. Step 2: Build a Huffman tree from the frequencies: each character is a leaf; repeatedly merge the two lowest-frequency nodes into a new internal node.
  3. Step 3: Read each character's code from the tree: left edge 0, right edge 1.
  4. Step 4: Encode the data with these Huffman codes.

Sample frequencies (25 characters in total):

characterFORGET
frequency234457

n − 1 merges for n characters; O(n log n) with a priority queue.

Huffman Coding · Run the code step by step

Building the Huffman Tree: Code Trace

F2O3E5T7G4R4

Create a leaf with its frequency for each character and add it to the priority queue (min-heap).

F2O3E5T7G4R4

poll() twice: the smallest frequencies are F2 (left) and O3 (right).

F2O3E5T7G4R45

New internal node 5 = 2 + 3 with children F2 and O3, added back to the queue.

F2O3E5T7G4R45

poll() twice: the smallest frequencies are G4 (left) and R4 (right). G4 and R4 tie; which comes out first depends on the heap.

F2O3E5T7G4R458

New internal node 8 = 4 + 4 with children G4 and R4, added back to the queue.

F2O3E5T7G4R458

poll() twice: the smallest frequencies are (5) (left) and E5 (right). (5) and E5 tie; which comes out first depends on the heap, and the code lengths do not change.

F2O3E5T7G4R45810

New internal node 10 = 5 + 5 with children (5) and E5, added back to the queue.

F2O3E5T7G4R45810

poll() twice: the smallest frequencies are T7 (left) and (8) (right).

F2O3E5T7G4R4581015

New internal node 15 = 7 + 8 with children T7 and (8), added back to the queue.

F2O3E5T7G4R4581015

poll() twice: the smallest frequencies are (10) (left) and (15) (right).

F2O3E5T7G4R458101525

New internal node 25 = 10 + 15 with children (10) and (15), added back to the queue. One node is left in the queue: the root.

0101010101F2O3E5T7G4R458101525

The loop ends and the root (25) is returned. Left edge 0, right edge 1: F=000, O=001, E=01, T=10, G=110, R=111.

static Node huffman(char[] ch, int[] f) {
  PriorityQueue<Node> pq =
    new PriorityQueue<>((a, b) -> a.f - b.f);
  for (int k = 0; k < ch.length; k++)
    pq.add(new Node(ch[k], f[k]));
  while (pq.size() > 1) {
    Node left = pq.poll();
    Node right = pq.poll();
    Node parent = new Node('\0', left.f + right.f);
    parent.left = left;
    parent.right = right;
    pq.add(parent);
  }
  return pq.poll();
}

Huffman Coding

Codes, Encoding, and Decoding

charfrequencycodelengthbits
F200036
O300139
E501210
T710214
G4110312
R4111312
total63
  • Fixed length: 6 characters → 3 bits, 25 × 3 = 75 bits. Huffman: 63 bits (16% savings, 2.52 bits on average).
  • Encoding: "FORTE" → 000 001 111 10 01 = 13 bits (15 with a fixed code).
  • Decoding: start at the root, 0 → left, 1 → right; at a leaf, output the character and return to the root: 1100110 → 110 | 01 | 10 → "GET".
  • Which child goes left or right is arbitrary (e.g., T could be the right child): the codes change, but the code lengths and the 63 bits do not.

Huffman Coding · Run the code step by step

Generating Codes from the Tree: Code Trace

static void genCodes(Node d, String code) {
  if (d.left == null) {     // leaf
    System.out.println(d.ch + " = " + code);
    return;
  }
  genCodes(d.left, code + "0"); // left: 0
  genCodes(d.right, code + "1"); // right: 1
}
01010101012510155ET8FOGR

genCodes(root, "") is called on the root (25) of the Huffman tree: the root is not a leaf, so we descend to the left child, appending "0".

01010101012510155ET8FOGR

Internal node (10), code = "0" (reached via a left edge) → not a leaf; descend left with "00".

01010101012510155ET8FOGR

Internal node (5), code = "00" (reached via a left edge) → not a leaf; descend left with "000".

01010101012510155ET8F000OGR

Leaf F (frequency 2): print the bits on the path → F = 000.

01010101012510155ET8F000O001GR

Leaf O (frequency 3): print the bits on the path → O = 001.

01010101012510155E01T8F000O001GR

Leaf E (frequency 5): print the bits on the path → E = 01.

01010101012510155E01T8F000O001GR

Internal node (15), code = "1" (reached via a right edge) → not a leaf; descend left with "10".

01010101012510155E01T108F000O001GR

Leaf T (frequency 7): print the bits on the path → T = 10.

01010101012510155E01T108F000O001GR

Internal node (8), code = "11" (reached via a right edge) → not a leaf; descend left with "110".

01010101012510155E01T108F000O001G110R

Leaf G (frequency 4): print the bits on the path → G = 110.

01010101012510155E01T108F000O001G110R111

Leaf R (frequency 4): print the bits on the path → R = 111. All leaves are printed: the frequent characters (E, T) get 2 bits, the rare ones 3. The tree is traversed once: O(n).

Huffman Coding

Why Is Huffman Optimal?

Greedy choice

There is an optimal code tree in which the two lowest-frequency characters x and y are siblings at the deepest level: swapping with a deeper character does not increase the total bits (exchange argument).

Optimal substructure

Merge x and y into a single character "z" with frequency f(x) + f(y); expanding z in an optimal tree of the smaller problem gives an optimal tree of the original problem.

Cost

Total bits = Σ freq(c) · depth(c). Among prefix-free codes, Huffman makes this minimal.

Complexity

Counting frequencies O(m) (m = text length); tree O(n log n); encoding O(m). The tree (or the code table) must also be sent to the decoder.

xbpq⇄pbxqx: least frequent, shallowx deepest: no extra bits

Applications

Other Greedy Algorithms

Dijkstra and PrimAt each step, permanently select the vertex with the smallest distance / connection cost. Dijkstra finds shortest paths in graphs without negative edges; Prim finds an MST for any weights.
Graph coloringColor the vertices in order, each with the smallest color its neighbors do not use. Fast, but the number of colors may not be minimal (NP-hard).
Bin packingFirst fit: put each item in the first bin it fits (best fit: the bin that leaves the least space). First fit decreasing uses at most ≈ 11/9 · OPT + 1 bins.
Maximum coverageAt each step, pick the set that covers the most new elements: guarantees at least (1 − 1/e) ≈ 63% of the optimum.
For NP-hard problems, greedy algorithms are valuable as approximation algorithms with proven ratios.

Summary

Summary

ProblemGreedy criterionTimeOptimal?
Making changelargest coin that fitsO(k + count)if canonical
Activity selectionearliest finishO(n log n)yes
Job sequencingmost profitable job, latest free slotO(n²)yes
Fractional knapsacklargest profit/weightO(n log n)yes (0/1: no)
MSTlightest edge crossing a cut (Kruskal, Prim, Borůvka)O(E log V)yes
Huffmantwo lowest-frequency treesO(n log n)yes

Summary

Common Mistakes

Assuming greedy is correct without proof6 with coins {1, 3, 4}: greedy uses 3 coins, the optimum 2. The greedy-choice property must be shown for each problem.
The wrong criterionIn activity selection, the earliest start or the shortest activity is wrong; the right criterion is the earliest finish.
Ratio in the 0/1 knapsackThe ratio criterion, optimal for the fractional problem, does not guarantee the optimum for indivisible items.
Earliest free slotIn job sequencing, placing a job in the earliest free slot before its deadline leaves no room for jobs with tight deadlines; choose the latest free slot.
Misreading the cut propertyThe safe edge is the lightest edge crossing a cut. Taking the lightest edges of the whole graph in order can create a cycle; that is why Kruskal discards edges that form one.
Decoding Huffman without the treeWithout the code table or the tree, the data cannot be decoded. For a one-character text the root is a leaf: its code still needs at least 1 bit.

Summary

Sample Problems

  1. How many coins does greedy use to pay 63 with {1, 5, 10, 25}? What are the greedy and optimal results for 14 with {1, 7, 10}?
  2. Activities (s, f): (1,4), (3,5), (0,6), (5,7), (3,9), (5,9), (6,10), (8,11), (8,12), (2,14), (12,16). What is the maximum number of activities that can be selected?
  3. Frequencies a:5, b:9, c:12, d:13, e:16, f:45. Find the Huffman code lengths and the total bits.
  4. The vertices are split into {A, B} and {C, D}; the edges crossing the cut are A–C 5, B–C 3, B–D 4. Which edge is in every MST?
  5. Can Kruskal, Prim, and Borůvka find different spanning trees on the same graph?
Show answers
  1. 63 = 25 + 25 + 10 + 1 + 1 + 1 → 6 coins. 14: greedy 10 + 1 + 1 + 1 + 1 = 5 coins; optimal 7 + 7 = 2 coins.
  2. Sort by finish time and take the earliest finishing: (1,4), (5,7), (8,11), (12,16) → 4 activities.
  3. f = 1, c = d = e = 3, a = b = 4 bits (e.g., f=0, c=100, d=101, a=1100, b=1101, e=111) → 45 + 36 + 39 + 48 + 20 + 36 = 224 bits (300 with a fixed 3-bit code).
  4. The lightest edge crossing the cut is B–C (3); since the weights are distinct, the cut property puts it in every MST.
  5. No if the weights are distinct: the MST is unique. With equal weights they can find different trees, but with the same total weight.

Summary

Exercises

  1. Jobs (id, deadline, profit): (a,2,100), (b,1,19), (c,2,27), (d,1,25), (e,3,15). Apply greedy job sequencing.
  2. W = 10, items (profit, weight): (10,5), (40,4), (30,6), (50,3). What is the fractional knapsack value?
  3. How many bits does the Huffman encoding of "ABRACADABRA" take? Compare with a fixed-length code.
  4. Explain what Kruskal produces when all edges have the same weight, and which step accounts for its O(E log V) cost.
  5. Find a coin system with the smallest amount for which greedy change is not optimal, and justify it.
Answers / hints
  1. Profit order a, c, d, b, e: a → slot 2, c → slot 1, d and b do not fit, e → slot 3 → c, a, e; profit 142.
  2. Ratios 2, 10, 5, 16.7: 50 (3 kg) + 40 (4 kg) + 3/6 of 30 (15) = 105.
  3. A5 B2 R2 C1 D1 → A 1 bit, the others 3 bits (2–4 bits depending on tie-breaking) → 23 bits; 33 bits with a fixed 3-bit code.
  4. Any spanning tree (every tree is minimum); the dominant cost is sorting the edges, and Union-Find is nearly linear.
  5. {1, 3, 4} and 6: greedy 4+1+1, optimal 3+3. In canonical systems (e.g., 1, 5, 10, 25), each larger coin "covers" the smaller ones optimally.

Summary

References

  • Cormen, Leiserson, Rivest, and Stein, Introduction to Algorithms, 4th ed., 2022 (Chapter 15: Greedy Algorithms; Chapter 21: MST).
  • Kleinberg and Tardos, Algorithm Design, 2006 (Chapter 4: Greedy Algorithms).
  • Sedgewick and Wayne, Algorithms, 4th ed., 2011 (4.3 MST, 5.5 Data Compression).
  • D. A. Huffman, "A Method for the Construction of Minimum-Redundancy Codes", Proceedings of the IRE 40(9), 1952.
  • J. B. Kruskal, "On the Shortest Spanning Subtree of a Graph and the Traveling Salesman Problem", Proc. AMS 7(1), 1956.

The End

Chapter 2: Greedy Algorithms
Previous chapter: Chapter 1: Divide and Conquer · Next chapter: Chapter 3: Dynamic Programming
1 / 1 Sercan KÜLCÜ. All rights reserved.