Chapter 2: Greedy Algorithms
Introduction
Learning Outcomes
Define the greedy strategy; explain when the greedy-choice property and optimal substructure guarantee a correct result.
Solve making change, activity selection, job sequencing, and fractional knapsack greedily; give counterexamples where greedy fails.
Explain why the cut property makes the greedy choices of Kruskal, Prim, and Borůvka correct; compare the three by selection rule, data structure, and running time.
Build a Huffman tree, derive prefix-free codes, encode and decode a text, and compute the savings in bits.
The Greedy Approach
Winning Step by Step
- Greedy algorithms proceed by choosing, at each step, the option that looks best right now.
- Like a treasure hunter: at every stage, take the option with the largest immediate gain.
- It works in stages: instead of looking at the problem as a whole, it focuses on the best option at hand.
- A decision is never undone; there is no backtracking.
- Like planning a trip by taking the road that looks shortest at every junction.
Fast and simple; the code is usually short and clear.
However, it does not guarantee an optimal solution; in many cases it yields a good approximate one.
The Greedy Approach
The Greedy Paradigm: General Template
Solution greedy(Candidate[] candidates) {
Solution s = new Solution(); // empty
sort(candidates); // by criterion
for (Candidate a : candidates) {
if (s.isFeasible(a)) // constraints hold
s.add(a); // never undone
}
return s;
}- Define the set of feasible solutions (which choices keep the constraints satisfied?).
- Pick a greedy criterion: cheapest, earliest finishing, most profitable, most frequent…
- Choose the locally best candidate by the criterion; add it to the solution if feasible.
- Repeat until the solution is complete.
The Greedy Approach
When Is Greedy Correct?
There is at least one optimal solution that contains the locally best choice. The first greedy choice never leads us "down the wrong path".
Once a choice is made, an optimal solution of the remaining subproblem, together with that choice, gives an optimal solution of the whole problem.
Take any optimal solution O. If O does not contain the greedy choice, exchange one of its elements for the greedy choice; the solution stays feasible and does not get worse. So an optimal solution containing the greedy choice exists; induction extends this to every step.
The Greedy Approach
Advantages and Disadvantages
Advantages
- Easy to understand and implement; short code.
- Fast: usually a sort plus one pass, O(n log n).
- Little memory: only the current solution is kept.
- Optimal on the right problems (MST, Huffman, activity selection).
- On hard problems it gives a good approximation or a starting point.
Disadvantages
- No guarantee of optimality (making change with {1, 3, 4}, 0/1 knapsack).
- Correctness must be proved separately for each problem.
- The criterion is critical: the wrong one gives the wrong result.
- Decisions are never undone; there is no chance to fix a mistake.
The Greedy Approach
The Change-Making Problem (Coin Change)
Pay a given amount with the fewest coins.
Greedy criterion: take the largest coin not exceeding the remaining amount; repeat until the amount is zero.
// coins[] sorted in decreasing order static int makeChange(int[] coins, int amount) { int count = 0; for (int p : coins) { while (amount >= p) { amount -= p; count++; } } return count; }
Yes for canonical coin systems such as {50, 25, 10, 5, 1}: 68 = 50 + 10 + 5 + 1 + 1 + 1.
With {4, 3, 1}, greedy pays 6 as 4 + 1 + 1 (3 coins); the optimum is 3 + 3 (2 coins). In general, dynamic programming is needed.
Time: O(k + count), where k is the number of coin types.
The Greedy Approach · Run the code step by step
makeChange({4, 3, 1}, 10): Code Trace
// coins[] sorted in decreasing order static int makeChange(int[] coins, int amount) { int count = 0; for (int p : coins) { while (amount >= p) { amount -= p; count++; } } return count; }
The coins {4, 3, 1} are sorted in decreasing order; the amount is 10. This system is not canonical, so the greedy result may not be optimal.
p = 4: 10 ≥ 4 → this coin can be used.
amount −= 4 → 6, count = 1. The loop continues.
amount −= 4 → 2, count = 2. 2 < 4: the while loop ends.
p = 3: 2 < 3 → the 3-coin is never used; move on to the next coin.
p = 1: 2 ≥ 1 → this coin can be used.
amount −= 1 → 1, count = 3. The loop continues.
amount −= 1 → 0, count = 4. 0 < 1: the while loop ends.
Greedy returns 4 + 4 + 1 + 1 = 4 coins, yet 4 + 3 + 3 = 3 coins suffice (found by DP). Taking two 4-coins first blocked the use of the 3-coins.
The Greedy Approach · Step by step
Making Change: Step by Step
Example 1: coins {50, 25, 10, 5, 1}, amount 68. Start with the largest coin.
The largest coin that fits is 50 → 68 − 50 = 18 left.
The largest coin that fits is 10 → 18 − 10 = 8 left.
The largest coin that fits is 5 → 8 − 5 = 3 left.
The largest coin that fits is 1 → 3 − 1 = 2 left.
The largest coin that fits is 1 → 2 − 1 = 1 left.
The largest coin that fits is 1 → 1 − 1 = 0 left. 6 coins in total — optimal for this coin system.
Example 2: coins {4, 3, 1}, amount 6. Start with the largest coin.
The largest coin that fits is 4 → 6 − 4 = 2 left.
The largest coin that fits is 1 → 2 − 1 = 1 left.
The largest coin that fits is 1 → 1 − 1 = 0 left.
Greedy gave 4 + 1 + 1 = 3 coins, yet 3 + 3 = 2 coins suffice. The locally best choice (4) missed the global optimum.
The Greedy Approach
Try It: Greedy vs. Optimal Change
The optimum is found by dynamic programming: minCoins[t] = 1 + min minCoins[t − p]. Coins are integers from 1 to 999, at most 8 kinds; the amount is at most 999. Enter = compute.
The Greedy Approach
The Greedy Approach vs. Divide and Conquer
| Greedy | Divide and conquer | |
|---|---|---|
| Decision | a single locally best choice per step; never undone | split by a fixed rule; no choice is made |
| Subproblems | one subproblem remains after the choice | several independent subproblems |
| Correctness | greedy-choice property + optimal substructure must be proved | depends on the correctness of the combine step |
| Cost | usually O(n log n): sort + one pass | via a recurrence and the master theorem |
| Example | activity selection, Kruskal, Huffman | merge sort, Karatsuba |
Activity Selection
Activity Selection
- Each activity has a start time si and a finish time fi; only one activity can run at a time (one hall).
- Goal: select the maximum number of mutually compatible activities. [si, fi) and [sj, fj) are compatible ⇔ sj ≥ fi or si ≥ fj.
- Greedy criterion: select the activity that finishes earliest, discard those that overlap it, and repeat with the rest.
| Activity | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| start | 1 | 2 | 4 | 6 | 5 | 8 | 0 | 3 | 7 | 10 |
| finish | 3 | 5 | 6 | 8 | 7 | 9 | 2 | 4 | 9 | 12 |
Activity Selection · Run the code step by step
select(e): Code Trace
The activities are sorted by finish time (top to bottom). Dashed line: finish of the last selected activity.
The earliest-finishing activity (#7, [0, 2)) is always selected.
#1 [1, 3): start 1 < 2 → overlaps, skipped.
#8 [3, 4): start 3 ≥ 2 → selected, i = 2.
#2 [2, 5): start 2 < 4 → overlaps, skipped.
#3 [4, 6): start 4 ≥ 4 → selected, i = 4.
#5 [5, 7): start 5 < 6 → overlaps, skipped.
#4 [6, 8): start 6 ≥ 6 → selected, i = 6.
#6 [8, 9): start 8 ≥ 8 → selected, i = 7.
#9 [7, 9): start 7 < 9 → overlaps, skipped.
#10 [10, 12): start 10 ≥ 9 → selected, i = 9. Result: 6 activities (7, 8, 3, 4, 6, 10).
static void select(Activity[] e) { Arrays.sort(e, (x, y) -> x.finish - y.finish); int i = 0; // last selected System.out.print(e[i].id + " "); for (int j = 1; j < e.length; j++) { if (e[j].start >= e[i].finish) { System.out.print(e[j].id + " "); i = j; } } }
Activity Selection
Why Earliest Finish?
Other natural criteria do not guarantee an optimal result (orange: greedy's choice, green: optimal solution):
The "fewest overlaps" criterion has a counterexample too.
Let a1 be the earliest-finishing activity and O an optimal solution. If the first activity of O is b ≠ a1, replace b with a1: since f(a1) ≤ f(b), it overlaps no other activity of O, and the count stays the same.
So some optimal solution contains a1. After discarding the activities that overlap a1, the remaining problem has the same structure (optimal substructure).
Job Sequencing
Job Sequencing with Deadlines
- Each job takes 1 unit of time and has a deadline and a profit.
- The profit is earned only if the job finishes by its deadline; one job runs at a time.
- Goal: maximize the total profit.
- Greedy criterion: take the jobs in decreasing order of profit; put each in the latest free slot before its deadline; if no slot is free, skip the job.
- Choosing the latest slot leaves the early slots for jobs with closer deadlines.
| Job | deadline | profit |
|---|---|---|
| 1 | 4 | 20 |
| 2 | 1 | 10 |
| 3 | 1 | 40 |
| 4 | 1 | 30 |
| 5 | 3 | 50 |
| 6 | 2 | 30 |
Time: sorting O(n log n) + slot search O(n · dmax) → worst case O(n²). With Union-Find, the slot search drops to nearly O(n).
Job Sequencing · Run the code step by step
jobSequencing(jobs, 6): Code Trace
The jobs are sorted in decreasing order of profit (Job 4 and Job 6, with equal profit, keep their original order). All slots are empty (−1).
Job 5 (deadline 3, profit 50): the latest feasible slot 3 is free → placed. total = 50.
Job 3 (deadline 1, profit 40): the latest feasible slot 1 is free → placed. total = 90.
Job 4 (deadline 1, profit 30): slot 1 is taken (Job 3) and there is no earlier slot → the job is skipped.
Job 6 (deadline 2, profit 30): the latest feasible slot 2 is free → placed. total = 120.
Job 1 (deadline 4, profit 20): the latest feasible slot 4 is free → placed. total = 140.
Job 2 (deadline 1, profit 10): slot 1 is taken (Job 3) and there is no earlier slot → the job is skipped.
Result: slots 1–4 = Job 3, Job 6, Job 5, Job 1 → total profit 140. Job 4 and Job 2 cannot be scheduled.
static int jobSequencing(Job[] jobs, int n) { Arrays.sort(jobs, (a, b) -> b.profit - a.profit); int[] slot = new int[n]; Arrays.fill(slot, -1); int total = 0; for (int i = 0; i < n; i++) { int j = Math.min(n, jobs[i].deadline) - 1; for (; j >= 0; j--) { if (slot[j] == -1) { slot[j] = jobs[i].id; total += jobs[i].profit; break; } } } return total; }
Knapsack
Fractional Knapsack
Capacity W = 50. If items can be split, the greedy criterion is: take the one with the largest profit/weight ratio.
| Item | profit | weight | ratio |
|---|---|---|---|
| 1 | 60 | 10 | 6 |
| 2 | 100 | 20 | 5 |
| 3 | 120 | 30 | 4 |
Fractional: items 1 + 2 in full, 20/30 of item 3 → 60 + 100 + 80 = 240 (optimal). Time O(n log n).
Each unit of the highest-ratio item earns at least as much as any other option of the same weight. If an optimal solution does not take as much of this item as it can, swap in an equal weight of it for another item; the profit does not drop (exchange argument).
If items cannot be split (0/1 knapsack), the ratio criterion does not guarantee the optimum; that problem is solved with dynamic programming.
Knapsack · Run the code step by step
fractional(s, 50): Code Trace
Items sorted by decreasing profit/weight ratio: 1 (6), 2 (5), 3 (4). The knapsack is empty; remaining capacity 50.
Item 1: weight 10 ≤ cap 50 → fits entirely.
cap −= 10 → 40, profit += 60 → 60.
Item 2: weight 20 ≤ cap 40 → fits entirely.
cap −= 20 → 20, profit += 100 → 160.
Item 3: weight 30 > cap 20 → does not fit entirely, else branch.
For the remaining 20 units, 20/30 of item 3: 120 · 20/30 = 80 → profit = 240. The knapsack is full, break.
Result 240: optimal for the fractional problem. The cost comes from sorting: O(n log n); the scan is O(n).
static double fractional(Item[] s, int W) { Arrays.sort(s, (a, b) -> Double.compare(b.ratio(), a.ratio())); double profit = 0; int cap = W; for (Item x : s) { if (x.weight <= cap) { // fits entirely cap -= x.weight; profit += x.profit; } else { // fractional part profit += x.profit * (double) cap / x.weight; break; } } return profit; }
Minimum Spanning Tree
Minimum Spanning Trees and the Cut Property
- Minimum spanning tree (MST): a tree connecting all vertices of a connected, weighted, undirected graph with the minimum total weight; V − 1 edges.
- Cut property: if the vertices are split into S and V − S, the lightest edge crossing this cut belongs to an MST.
- Kruskal, Prim, and Borůvka are greedy algorithms that use this property with different cuts: every edge added is the lightest edge crossing some cut, so no choice is ever undone.
- If the weights are distinct, the MST is unique.
Edges crossing the cut: 9, 4, 8, 11. The lightest, r–s (4), is in the MST.
Minimum Spanning Tree
Kruskal, Prim, and Borůvka
| Kruskal | Prim | Borůvka | |
|---|---|---|---|
| Greedy choice | lightest edge in the whole graph that forms no cycle | lightest edge leaving the tree | lightest edge leaving each component |
| Growth | forest → tree | a single tree, vertex by vertex | a forest, merging in rounds |
| Cut used | the two components separating the edge's endpoints | the tree vs. the remaining vertices | each component vs. the rest |
| Data structure | sorting + Union-Find | priority queue | Union-Find |
| Time | O(E log V) | O(E log V) (binary heap) | O(E log V) |
| Best for | sparse graphs, edge list | dense graphs, adjacency | parallel / distributed |
Huffman Coding
Huffman Coding
- A lossless data compression method (David Huffman, 1952).
- Uses variable-length codes: frequent characters get short codes, rare ones get long codes.
- Produces prefix-free codes: no code is a prefix of another → decodes unambiguously without separators.
- Greedy step: merge the two lowest-frequency trees.
Let a = 0, b = 01, c = 1. Is "01" "ab" or "b"? Ambiguous: 0 is a prefix of 01.
Huffman codes sit at the leaves of a binary tree, so no code can be a prefix of another.
Huffman Coding
Huffman Coding: Steps
- Step 1: Count the frequency of each character.
- Step 2: Build a Huffman tree from the frequencies: each character is a leaf; repeatedly merge the two lowest-frequency nodes into a new internal node.
- Step 3: Read each character's code from the tree: left edge 0, right edge 1.
- Step 4: Encode the data with these Huffman codes.
Sample frequencies (25 characters in total):
| character | F | O | R | G | E | T |
|---|---|---|---|---|---|---|
| frequency | 2 | 3 | 4 | 4 | 5 | 7 |
n − 1 merges for n characters; O(n log n) with a priority queue.
Huffman Coding · Run the code step by step
Building the Huffman Tree: Code Trace
Create a leaf with its frequency for each character and add it to the priority queue (min-heap).
poll() twice: the smallest frequencies are F2 (left) and O3 (right).
New internal node 5 = 2 + 3 with children F2 and O3, added back to the queue.
poll() twice: the smallest frequencies are G4 (left) and R4 (right). G4 and R4 tie; which comes out first depends on the heap.
New internal node 8 = 4 + 4 with children G4 and R4, added back to the queue.
poll() twice: the smallest frequencies are (5) (left) and E5 (right). (5) and E5 tie; which comes out first depends on the heap, and the code lengths do not change.
New internal node 10 = 5 + 5 with children (5) and E5, added back to the queue.
poll() twice: the smallest frequencies are T7 (left) and (8) (right).
New internal node 15 = 7 + 8 with children T7 and (8), added back to the queue.
poll() twice: the smallest frequencies are (10) (left) and (15) (right).
New internal node 25 = 10 + 15 with children (10) and (15), added back to the queue. One node is left in the queue: the root.
The loop ends and the root (25) is returned. Left edge 0, right edge 1: F=000, O=001, E=01, T=10, G=110, R=111.
static Node huffman(char[] ch, int[] f) { PriorityQueue<Node> pq = new PriorityQueue<>((a, b) -> a.f - b.f); for (int k = 0; k < ch.length; k++) pq.add(new Node(ch[k], f[k])); while (pq.size() > 1) { Node left = pq.poll(); Node right = pq.poll(); Node parent = new Node('\0', left.f + right.f); parent.left = left; parent.right = right; pq.add(parent); } return pq.poll(); }
Huffman Coding
Codes, Encoding, and Decoding
| char | frequency | code | length | bits |
|---|---|---|---|---|
| F | 2 | 000 | 3 | 6 |
| O | 3 | 001 | 3 | 9 |
| E | 5 | 01 | 2 | 10 |
| T | 7 | 10 | 2 | 14 |
| G | 4 | 110 | 3 | 12 |
| R | 4 | 111 | 3 | 12 |
| total | 63 | |||
- Fixed length: 6 characters → 3 bits, 25 × 3 = 75 bits. Huffman: 63 bits (16% savings, 2.52 bits on average).
- Encoding: "FORTE" → 000 001 111 10 01 = 13 bits (15 with a fixed code).
- Decoding: start at the root, 0 → left, 1 → right; at a leaf, output the character and return to the root:
1100110→ 110 | 01 | 10 → "GET". - Which child goes left or right is arbitrary (e.g., T could be the right child): the codes change, but the code lengths and the 63 bits do not.
Huffman Coding · Run the code step by step
Generating Codes from the Tree: Code Trace
static void genCodes(Node d, String code) { if (d.left == null) { // leaf System.out.println(d.ch + " = " + code); return; } genCodes(d.left, code + "0"); // left: 0 genCodes(d.right, code + "1"); // right: 1 }
genCodes(root, "") is called on the root (25) of the Huffman tree: the root is not a leaf, so we descend to the left child, appending "0".
Internal node (10), code = "0" (reached via a left edge) → not a leaf; descend left with "00".
Internal node (5), code = "00" (reached via a left edge) → not a leaf; descend left with "000".
Leaf F (frequency 2): print the bits on the path → F = 000.
Leaf O (frequency 3): print the bits on the path → O = 001.
Leaf E (frequency 5): print the bits on the path → E = 01.
Internal node (15), code = "1" (reached via a right edge) → not a leaf; descend left with "10".
Leaf T (frequency 7): print the bits on the path → T = 10.
Internal node (8), code = "11" (reached via a right edge) → not a leaf; descend left with "110".
Leaf G (frequency 4): print the bits on the path → G = 110.
Leaf R (frequency 4): print the bits on the path → R = 111. All leaves are printed: the frequent characters (E, T) get 2 bits, the rare ones 3. The tree is traversed once: O(n).
Huffman Coding
Why Is Huffman Optimal?
There is an optimal code tree in which the two lowest-frequency characters x and y are siblings at the deepest level: swapping with a deeper character does not increase the total bits (exchange argument).
Merge x and y into a single character "z" with frequency f(x) + f(y); expanding z in an optimal tree of the smaller problem gives an optimal tree of the original problem.
Total bits = Σ freq(c) · depth(c). Among prefix-free codes, Huffman makes this minimal.
Counting frequencies O(m) (m = text length); tree O(n log n); encoding O(m). The tree (or the code table) must also be sent to the decoder.
Applications
Other Greedy Algorithms
Summary
Summary
| Problem | Greedy criterion | Time | Optimal? |
|---|---|---|---|
| Making change | largest coin that fits | O(k + count) | if canonical |
| Activity selection | earliest finish | O(n log n) | yes |
| Job sequencing | most profitable job, latest free slot | O(n²) | yes |
| Fractional knapsack | largest profit/weight | O(n log n) | yes (0/1: no) |
| MST | lightest edge crossing a cut (Kruskal, Prim, Borůvka) | O(E log V) | yes |
| Huffman | two lowest-frequency trees | O(n log n) | yes |
Summary
Common Mistakes
Summary
Sample Problems
- How many coins does greedy use to pay 63 with {1, 5, 10, 25}? What are the greedy and optimal results for 14 with {1, 7, 10}?
- Activities (s, f): (1,4), (3,5), (0,6), (5,7), (3,9), (5,9), (6,10), (8,11), (8,12), (2,14), (12,16). What is the maximum number of activities that can be selected?
- Frequencies a:5, b:9, c:12, d:13, e:16, f:45. Find the Huffman code lengths and the total bits.
- The vertices are split into {A, B} and {C, D}; the edges crossing the cut are A–C 5, B–C 3, B–D 4. Which edge is in every MST?
- Can Kruskal, Prim, and Borůvka find different spanning trees on the same graph?
Show answers
- 63 = 25 + 25 + 10 + 1 + 1 + 1 → 6 coins. 14: greedy 10 + 1 + 1 + 1 + 1 = 5 coins; optimal 7 + 7 = 2 coins.
- Sort by finish time and take the earliest finishing: (1,4), (5,7), (8,11), (12,16) → 4 activities.
- f = 1, c = d = e = 3, a = b = 4 bits (e.g., f=0, c=100, d=101, a=1100, b=1101, e=111) → 45 + 36 + 39 + 48 + 20 + 36 = 224 bits (300 with a fixed 3-bit code).
- The lightest edge crossing the cut is B–C (3); since the weights are distinct, the cut property puts it in every MST.
- No if the weights are distinct: the MST is unique. With equal weights they can find different trees, but with the same total weight.
Summary
Exercises
- Jobs (id, deadline, profit): (a,2,100), (b,1,19), (c,2,27), (d,1,25), (e,3,15). Apply greedy job sequencing.
- W = 10, items (profit, weight): (10,5), (40,4), (30,6), (50,3). What is the fractional knapsack value?
- How many bits does the Huffman encoding of "ABRACADABRA" take? Compare with a fixed-length code.
- Explain what Kruskal produces when all edges have the same weight, and which step accounts for its O(E log V) cost.
- Find a coin system with the smallest amount for which greedy change is not optimal, and justify it.
Answers / hints
- Profit order a, c, d, b, e: a → slot 2, c → slot 1, d and b do not fit, e → slot 3 → c, a, e; profit 142.
- Ratios 2, 10, 5, 16.7: 50 (3 kg) + 40 (4 kg) + 3/6 of 30 (15) = 105.
- A5 B2 R2 C1 D1 → A 1 bit, the others 3 bits (2–4 bits depending on tie-breaking) → 23 bits; 33 bits with a fixed 3-bit code.
- Any spanning tree (every tree is minimum); the dominant cost is sorting the edges, and Union-Find is nearly linear.
- {1, 3, 4} and 6: greedy 4+1+1, optimal 3+3. In canonical systems (e.g., 1, 5, 10, 25), each larger coin "covers" the smaller ones optimally.
Summary
References
- Cormen, Leiserson, Rivest, and Stein, Introduction to Algorithms, 4th ed., 2022 (Chapter 15: Greedy Algorithms; Chapter 21: MST).
- Kleinberg and Tardos, Algorithm Design, 2006 (Chapter 4: Greedy Algorithms).
- Sedgewick and Wayne, Algorithms, 4th ed., 2011 (4.3 MST, 5.5 Data Compression).
- D. A. Huffman, "A Method for the Construction of Minimum-Redundancy Codes", Proceedings of the IRE 40(9), 1952.
- J. B. Kruskal, "On the Shortest Spanning Subtree of a Graph and the Traveling Salesman Problem", Proc. AMS 7(1), 1956.