Chapter 6: Approximation Algorithms
Introduction
Learning Outcomes
Explain what an approximation algorithm is, the approximation ratio, and how it differs from a heuristic.
Apply nearest neighbor, the MST-based 2-approximation and 2-opt local search to the traveling salesman problem step by step.
Write and trace machine scheduling (list scheduling, LPT) and the vertex cover 2-approximation in Java; prove their ratios.
Compute how power iteration converges to the largest eigenvalue and its eigenvector.
Introduction
Approximation Algorithms
- For some problems finding the exact (optimal) solution is very hard or impossible in practice: TSP, vertex cover, machine scheduling and knapsack are NP-hard; if P ≠ NP they have no polynomial-time exact algorithms.
- Instead of looking for the perfect solution, approximation algorithms produce solutions as close as possible to the optimum.
- They run in polynomial time: hard problems can be solved in reasonable time.
- It can often be proven that the solution is within a fixed ratio or percentage of the optimum.
- Preferred when computing resources (time, memory) are limited.
Introduction
The Balance Scale Analogy
- When a weight is placed on one pan of a scale, it can be hard to put exactly the same weight on the other pan.
- Instead, we use the small weights at hand to balance the large one as closely as possible.
- If we only have 20, 10, 5 and 1 kg, each usable once, the closest balance to 37 kg is 36 kg.
Introduction
Approximation Ratio
C: cost of the solution the algorithm finds, C*: cost of the optimal solution. If for every input the algorithm satisfies
it is a ρ(n)-approximation algorithm (ρ ≥ 1; ρ = 1 means the algorithm is optimal).
The ratio is a worst-case guarantee; in practice solutions are often much closer to the optimum.
Introduction
Approximation Algorithms, Heuristics and Approximation Schemes
| Approach | Guarantee | Example |
|---|---|---|
| Exact algorithm | Optimal, but exponential time on NP-hard problems | Brute-force TSP: (n−1)!/2 tours |
| Approximation algorithm | Polynomial time + a proven ratio ρ | Vertex cover: ρ = 2 |
| PTAS | Ratio (1 + ε) for every fixed ε > 0, polynomial in n (may be exponential in ε) | Euclidean TSP (Arora, 1998) |
| FPTAS | Ratio (1 + ε), polynomial in both n and 1/ε | Knapsack: O(n³/ε) |
| Heuristic | No guarantee; usually good in practice | 2-opt, nearest neighbor |
| Metaheuristic | No guarantee; a problem-independent search framework | Genetic algorithms, ant colony |
There are also randomized approximation algorithms such as randomized rounding; they guarantee a ratio in expectation.
Introduction
Applications of Approximation Algorithms
Choosing the route that visits every customer and returns with the shortest total length: logistics, parcel delivery, circuit-board drilling.
Choosing the items that fit and give the most value: budget, cargo and portfolio planning.
Running several jobs or processes on machines/processors as efficiently as possible.
Placing the fewest monitoring points in a network; converging iteratively to an eigenvalue of a huge matrix (PageRank).
Introduction
Kinds of Approximation Algorithms
Solve a simpler version of the problem (e.g. relax integrality: LP relaxation, an MST lower bound for TSP).
At each step take the option that looks best right now (nearest neighbor, LPT, ratio-sorted knapsack).
Improve the current solution with small changes; stop when no better neighbor remains (2-opt).
Traveling Salesman
The Traveling Salesman Problem (TSP)
- Optimization problem: a salesman wants to visit each of a set of cities once with the shortest total distance and return to the starting city.
- Combinatorial problem: a solution is the order in which the cities are visited (a permutation).
- NP-hard: finding the best route for many cities is hard in practice. The shortest path between two vertices (Dijkstra), however, is solvable in polynomial time; do not confuse the two problems.
- Input: n cities (0, 1, 2, … or A, B, C, …) and dij: the distance between cities i and j.
- Symmetric TSP: dij = dji; in graph terms: the lightest Hamiltonian cycle in a complete graph.
| d | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| 0 | 0 | 10 | 15 | 20 | 25 |
| 1 | 10 | 0 | 35 | 25 | 30 |
| 2 | 15 | 35 | 0 | 30 | 35 |
| 3 | 20 | 25 | 30 | 0 | 40 |
| 4 | 25 | 30 | 35 | 40 | 0 |
Example: the distance matrix of 5 cities.
Traveling Salesman
Solving the Problem
| n | (n−1)!/2 distinct tours |
|---|---|
| 5 | 12 |
| 10 | 181,440 |
| 15 | ≈ 4.4 · 1010 |
| 20 | ≈ 6.1 · 1016 |
| 25 | ≈ 3.1 · 1023 |
The start city is fixed and both directions count once. Even at 10⁹ tours per second, n = 25 would take ≈ 10⁷ years.
Traveling Salesman
The Nearest Neighbor Heuristic
- Pick a start city and mark it visited.
- From the current city go to the nearest unvisited city (greedy choice).
- Repeat step 2 until every city is visited.
- Return to the start city from the last one to close the tour.
Pros
- Very simple; O(n²) time.
- Usually gives a tour about 25% above the optimum.
- A good initial tour for local search.
Cons
- There is no constant approximation ratio; even for metric TSP the ratio can be Θ(log n).
- The final return edge can be very long.
Traveling Salesman · Run the code step by step
Nearest Neighbor: Code Trace
Start city 0: visited[0] = true, tour = [0], total = 0.
Scan row 0, unvisited: d[0][1] = 10, d[0][2] = 15, d[0][3] = 20, d[0][4] = 25 → the smallest is 1 (10).
tour[1] = 1, total += 10 → 10; now cur = 1.
Scan row 1, unvisited: d[1][2] = 35, d[1][3] = 25, d[1][4] = 30 → the smallest is 3 (25).
tour[2] = 3, total += 25 → 35; now cur = 3.
Scan row 3, unvisited: d[3][2] = 30, d[3][4] = 40 → the smallest is 2 (30).
tour[3] = 2, total += 30 → 65; now cur = 2.
Scan row 2, unvisited: d[2][4] = 35 → the smallest is 4 (35).
tour[4] = 4, total += 35 → 100; now cur = 4.
Every city is visited; return to the start: total += d[4][0] = 25 → 125.
Tour 0 → 1 → 3 → 2 → 4 → 0, length 125. (Checked by brute force: optimal for this example.)
int[] nearestNeighbor(int[][] d, int start) { int n = d.length, total = 0, cur = start; boolean[] visited = new boolean[n]; int[] tour = new int[n + 1]; tour[0] = start; visited[start] = true; for (int k = 1; k < n; k++) { int nearest = -1, minD = Integer.MAX_VALUE; for (int j = 0; j < n; j++) if (!visited[j] && d[cur][j] < minD) { minD = d[cur][j]; nearest = j; } // go to the nearest city tour[k] = nearest; visited[nearest] = true; total += minD; cur = nearest; } // return to the start tour[n] = start; total += d[cur][start]; return tour; }
Traveling Salesman
Comparison with Brute Force: 12 Distinct Tours
| Tour | Length |
|---|---|
| 0 → 1 → 3 → 2 → 4 → 0 | 125 |
| 0 → 1 → 3 → 4 → 2 → 0 | 125 |
| 0 → 1 → 4 → 2 → 3 → 0 | 125 |
| 0 → 1 → 4 → 3 → 2 → 0 | 125 |
| 0 → 2 → 3 → 1 → 4 → 0 | 125 |
| 0 → 2 → 4 → 1 → 3 → 0 | 125 |
| Tour | Length |
|---|---|
| 0 → 1 → 2 → 3 → 4 → 0 | 140 |
| 0 → 1 → 2 → 4 → 3 → 0 | 140 |
| 0 → 2 → 1 → 3 → 4 → 0 | 140 |
| 0 → 2 → 1 → 4 → 3 → 0 | 140 |
| 0 → 3 → 1 → 2 → 4 → 0 | 140 |
| 0 → 3 → 2 → 1 → 4 → 0 | 140 |
- (5−1)!/2 = 12 tours tried: the optimum is 125; nearest neighbor also found 125 (a lucky case for this example).
- This matrix violates the triangle inequality: d12 = 35 > d10 + d02 = 10 + 15. So the example is not metric.
Traveling Salesman
The Triangle Inequality and Approximation Limits
| Case | Ratio | Description |
|---|---|---|
| General TSP | none | If P ≠ NP, no polynomial-time ρ-approximation exists for any constant ρ: such an algorithm would solve the Hamiltonian cycle problem. |
| Metric TSP, MST-based | 2 | MST + preorder walk + shortcuts; O(n²). |
| Metric TSP, Christofides | 1.5 | MST + minimum perfect matching on odd-degree vertices + Euler tour (1976). |
| Euclidean TSP | 1 + ε | A PTAS exists for points in the plane (Arora, Mitchell). |
| Nearest neighbor | Θ(log n) | No constant ratio is guaranteed, even in the metric case. |
Traveling Salesman
MST-Based 2-Approximation for Metric TSP
A metric example: 7 cities in the plane; distances are Euclidean (the triangle inequality holds).
1) Build a minimum spanning tree (MST) with Prim: total weight w(T) = 228.8. Deleting one edge of a tour leaves a spanning tree → w(T) ≤ OPT.
2) Walk the tree depth-first from A, twice (each edge once down, once back): A → D → B → C → E → F → E → C → B → D → G → D → A. Length 2 · w(T) = 457.6.
3) Skip repeated cities (shortcut): preorder A → D → B → C → E → F → G → A. Thanks to the triangle inequality, shortcuts never lengthen the path: tour = 287.7.
Result: tour ≤ 2 · w(T) ≤ 2 · OPT. Here tour = 287.7, optimum (brute force, 360 tours) = 282.5 → ratio 1.02 ≤ 2.
Local Search
Local Search: The 2-opt Move
Before: edges (a, b) and (c, d) cross.
After: (a, c) and (b, d); the path between them is reversed.
- Remove two edges of the tour and reconnect the ends crosswise: the subpath from b to c is reversed.
- Make the move if the gain Δ = d(a,c) + d(b,d) − d(a,b) − d(c,d) < 0. In the Euclidean plane every crossing can be removed this way.
- When no improving move remains, the tour is a 2-opt local optimum (not necessarily the global optimum).
Local Search · Step by step
Improving a Tour with 2-opt
28 cities, a random initial tour: length 1269.8. Edges keep crossing each other.
After improving move 1: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −36.9 → length 1232.9.
After improving move 2: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −8.9 → length 1224.0.
After improving move 4: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −27.6 → length 1187.3.
After improving move 8: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −32.6 → length 1111.7.
After improving move 14: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −14.9 → length 1005.2.
After improving move 22: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −37.0 → length 852.8.
After improving move 32: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −11.8 → length 595.9.
Move 48, the last: no 2-opt move shortens the tour any more → local optimum, length 430.8 (34% of the start). No crossing edges remain.
Local Search
2-opt Local Search: Java
void twoOpt(int[] tour, double[][] d) { int n = tour.length; boolean improved = true; while (improved) { improved = false; for (int i = 0; i < n - 1; i++) for (int j = i + 2; j < n; j++) { int a = tour[i], b = tour[i + 1]; int c = tour[j], e = tour[(j + 1) % n]; double delta = d[a][c] + d[b][e] - d[a][b] - d[c][e]; if (delta < -1e-9) { // shorter reverse(tour, i + 1, j); improved = true; } } } }
Local Search · Run the code step by step
2-opt: Code Trace
Initial tour 0 → 1 → 2 → 3 → 0, length 18. Edges (0, 1) and (2, 3) cross.
Pass 1 starts: improved = false. If no move is made in this pass, the loop ends.
i = 0, j = 2: replace (0,1) and (2,3) with (0,2) and (1,3). delta = 3 + 3 − 5 − 5 = −4 < 0 → the tour gets shorter.
reverse(tour, 1, 2): the path in between is reversed → [0, 2, 1, 3], length 14. improved = true.
i = 0, j = 3: delta = 4 + 3 − 3 − 4 = 0 (the two edges are adjacent: e = a) → no move.
i = 1, j = 3: delta = 5 + 5 − 4 − 4 = 2 → no move.
A move was made in the previous pass (improved = true), so the pass repeats from the start.
i = 0, j = 2: delta = 5 + 5 − 3 − 3 = 4 → no move.
i = 0, j = 3: delta = 4 + 3 − 3 − 4 = 0 (the two edges are adjacent: e = a) → no move.
i = 1, j = 3: delta = 5 + 5 − 4 − 4 = 2 → no move.
No shortening found in the whole pass: the loop ends. Tour [0, 2, 1, 3], length 14: a 2-opt local optimum (and for these 4 points also the optimum).
void twoOpt(int[] tour, double[][] d) { int n = tour.length; boolean improved = true; while (improved) { improved = false; for (int i = 0; i < n - 1; i++) for (int j = i + 2; j < n; j++) { int a = tour[i], b = tour[i + 1]; int c = tour[j], e = tour[(j + 1) % n]; double delta = d[a][c] + d[b][e] - d[a][b] - d[c][e]; if (delta < -1e-9) { // shorter reverse(tour, i + 1, j); improved = true; } } } }
Local Search
Escaping Local Optima: Metaheuristics
A worsening move is also accepted with probability e−Δ/T; the "temperature" T is lowered slowly. It can escape local optima.
A population of tours improved over generations by selection, crossover and mutation.
Ants leave more pheromone on the edges of short tours; later ants prefer edges with dense pheromone.
- All three are common in TSP, vehicle routing, job scheduling and assignment problems.
- None of them guarantees an approximation ratio; the result depends on the initial solution and the parameters. Typical settings: T₀ = 100, T ← 0.997 · T at each step; a population of 100, crossover at a random cut point, mutation probability 0.05.
Local search that accepts only improving moves stays in the left valley. Simulated annealing can climb over the hill at high T; as T drops, moves become pickier.
Job Scheduling
Job Scheduling
- Optimization problem: planning jobs on given resources (workers, machines, processors) as efficiently as possible.
- Goal: minimize the completion time of the jobs, or optimize other criteria (profit, lateness, deadlines).
n jobs, m identical machines; job j takes pj. Minimize the time the last job finishes, Cmax (the makespan). NP-hard for m ≥ 2.
Each job takes 1 unit and has a profit and a deadline. Maximize the total profit of the jobs finished by their deadlines.
The first problem has greedy approximation algorithms (list scheduling, LPT); the second is solved optimally by a greedy algorithm.
Job Scheduling · Step by step
Assigning Jobs to Machines: List Scheduling
Three identical machines M1–M3. Jobs arrive in order: J1 (2), J2 (8), J3 (6) ready at t = 0; J4 (3) arrives at t = 3. List scheduling (Graham): the next job goes to the machine that becomes free first.
J1 (time 2) → M1, runs in [0, 2] (all machines are free; the first one is chosen).
J2 (time 8) → M2, runs in [0, 8] (M1 is busy until t = 2).
J3 (time 6) → M3, runs in [0, 6] (M1 is busy until t = 2; M2 until t = 8).
t = 2: M1 is free but no job is waiting; the machine stays idle until t = 3 (hatched area).
J4 (time 3) → M1, runs in [3, 6] (M2 is busy until t = 8; M3 until t = 6).
All jobs assigned: Cmax = 8 (M2). Lower bound: OPT ≥ the longest job = 8 → this schedule is optimal. In general, though, list scheduling only guarantees a ratio of (2 − 1/m).
Job Scheduling
List Scheduling: A (2 − 1/m)-Approximation
Two lower bounds: OPT ≥ Σpj / m (the work must be shared) and OPT ≥ max pj.
- Let job j, the last to finish, start at time sj.
- Before sj all machines were busy (otherwise j would have started earlier) → sj ≤ (Σp − pj) / m.
- Cmax = sj + pj ≤ Σp / m + (1 − 1/m) · pj ≤ OPT + (1 − 1/m) · OPT.
• List: M1 = 1 + 2 = 3, M2 = 1 → Cmax = 3
• Optimum: {1, 1} and {2} → 2
Ratio 3/2 = 2 − 1/2.In general, m(m − 1) jobs of 1 unit followed by a single job of m units reach the ratio 2 − 1/m.
Job Scheduling
Longest Processing Time First (LPT): A 4/3-Approximation
LPT (Longest Processing Time first): sort the jobs by decreasing time, then apply list scheduling. Jobs: 2, 3, 4, 6, 2, 2 and m = 3.
Given order: Cmax = 8
LPT order 6, 4, 3, 2, 2, 2: Cmax = 7 = OPT (≥ ⌈19/3⌉ = 7)
- With long jobs first, the short jobs left at the end even out the imbalance: Cmax ≤ (4/3 − 1/(3m)) · OPT (Graham, 1969). Time O(n log n).
- Tight example (m = 3): 5, 5, 4, 4, 3, 3, 3 → LPT 11, optimum 9 ({5, 4}, {5, 4}, {3, 3, 3}).
Job Scheduling · Run the code step by step
LPT Scheduling: Code Trace
Jobs 2, 3, 4, 6, 2, 2 and m = 3 machines. List scheduling in this order gives Cmax = 8.
LPT: sort the jobs by time in decreasing order → 6, 4, 3, 2, 2, 2.
All three machines have load 0. Each job goes to the least loaded machine at that moment.
p = 6: the least loaded machine is k = 0 (load 0) (on ties, < keeps the first machine) → load = [6, 0, 0].
p = 4: the least loaded machine is k = 1 (load 0) → load = [6, 4, 0].
p = 3: the least loaded machine is k = 2 (load 0) → load = [6, 4, 3].
p = 2: the least loaded machine is k = 2 (load 3) → load = [6, 4, 5].
p = 2: the least loaded machine is k = 1 (load 4) → load = [6, 6, 5].
p = 2: the least loaded machine is k = 2 (load 5) → load = [6, 6, 7].
Makespan Cmax = 7. Since the lower bound is ⌈19/3⌉ = 7, this schedule is optimal.
int lpt(Integer[] jobs, int m) { Arrays.sort(jobs, Comparator.reverseOrder()); int[] load = new int[m]; for (int p : jobs) { int k = 0; // least loaded for (int i = 1; i < m; i++) if (load[i] < load[k]) k = i; load[k] += p; } return Arrays.stream(load).max().getAsInt(); }
Vertex Cover
Vertex Cover
- Covering problem: choose the fewest vertices that cover every edge of a graph. An edge is covered if at least one of its endpoints is chosen.
- Goal: find a set C ⊆ V covering all edges with as few vertices as possible. The decision version ("is there a cover of k vertices?") is NP-complete.
A valid cover of size 6 (every edge is watched by a "guard")
A minimum cover of the same graph, size 3
Vertex Cover
Example: A Four-Vertex Graph
- Vertices: A, B, C, D
- Edges: (A, B), (A, C), (B, C), (B, D), (C, D)
- {A, B, C} is a vertex cover: every edge has at least one endpoint in the set.
- But it is not the smallest: {B, C} also covers all five edges.
- A single vertex is not enough (B does not cover (A, C)) → the optimum is 2.
{A, B, C}: valid, size 3
{B, C}: minimum, size 2
Vertex Cover
Vertex Cover: Solution Approaches
Minimum cover {A, B, H, I, J} (5 vertices)
Vertex Cover
Vertex Cover Algorithm
Naive idea:
- Start with an empty cover.
- Pick an uncovered edge and add one of its endpoints to the set.
- Repeat until every edge is covered.
Adding only one endpoint gives no guarantee. The correct 2-approximation:
- Pick an uncovered edge (u, v) and add both u and v.
- Every edge touching u or v is now covered; repeat until no edges remain.
If the leaf end of every edge is chosen: 5 vertices; the optimum is {m}, 1 vertex. On a star with n leaves the ratio is n.
Vertex Cover · Run the code step by step
Vertex Cover 2-Approximation: Code Trace
The cover is empty. The edges are visited in order: (A,B), (A,C), …, (J,L).
(A, B): neither endpoint is in the cover → add A and B. These edges form a matching.
(A, C): A is already in the cover → the edge is covered, skip.
(A, D): A is already in the cover → the edge is covered, skip.
(A, E): A is already in the cover → the edge is covered, skip.
(A, F): A is already in the cover → the edge is covered, skip.
(B, C): B is already in the cover → the edge is covered, skip.
(G, H): neither endpoint is in the cover → add G and H. These edges form a matching.
(G, I): G is already in the cover → the edge is covered, skip.
(H, I): H is already in the cover → the edge is covered, skip.
(H, K): H is already in the cover → the edge is covered, skip.
(I, J): neither endpoint is in the cover → add I and J. These edges form a matching.
(J, K): J is already in the cover → the edge is covered, skip.
(J, L): J is already in the cover → the edge is covered, skip.
Result {A, B, G, H, I, J}: 6 vertices. The optimum is 5 ({A, B, H, I, J}) → ratio 6/5 = 1.2 ≤ 2.
Set<Character> vertexCover(char[][] edges) { Set<Character> cover = new HashSet<>(); for (char[] e : edges) { char u = e[0], v = e[1]; if (!cover.contains(u) && !cover.contains(v)) { cover.add(u); cover.add(v); } // both endpoints } return cover; }
Vertex Cover
Why At Most 2 × Optimum?
- Let M be the set of edges the algorithm picks. Both endpoints of each picked edge are added, so the edges of M share no endpoints (M is a matching).
- Any cover must contain a distinct vertex for each edge of M → OPT ≥ |M|.
- The algorithm picks exactly |C| = 2 |M| vertices → |C| ≤ 2 · OPT.
No constant ratio noticeably better than 2 is known; a ratio of 2 − ε has been shown impossible (under the Unique Games Conjecture).
Vertex Cover · Step by step
NP-Completeness: Reducing 3-SAT to Vertex Cover
φ = (x₁ ∨ x₁ ∨ x₂) ∧ (¬x₁ ∨ ¬x₂ ∨ ¬x₂) ∧ (¬x₁ ∨ x₂ ∨ x₂). One edge per variable (top boxes), one triangle per clause; each literal in a clause is joined to the variable vertex with the same name. l = 2 variables, m = 3 clauses → k = l + 2m = 8.
Assignment x₁ = false: pick ¬x₁ from the variable edge. Covered edges fade.
Assignment x₂ = true: pick x₂. Each variable edge is covered by exactly one vertex (l = 2 vertices).
In C₁ the true literal x₂ (its link edge is already covered) is left out; the other two vertices (x₁, x₁) are added.
In C₂ the true literal ¬x₁ is left out; the vertices ¬x₂ and ¬x₂ are added.
In C₃ the true literal ¬x₁ is left out; x₂ and x₂ are added. 8 vertices, all 20 edges covered: φ is satisfiable ⇔ a cover of size k = l + 2m exists.
Knapsack
Knapsack: Greedy by Ratio
| Item | Value | Weight | Ratio | Capacity 10 |
|---|---|---|---|---|
| 6 | 20 | 7 | 2.86 | ✓ taken (weight 7) |
| 3 | 8 | 4 | 2.00 | ✗ does not fit |
| 7 | 15 | 8 | 1.88 | ✗ does not fit |
| 2 | 5 | 3 | 1.67 | ✓ taken (weight 10) |
| 5 | 10 | 6 | 1.67 | ✗ does not fit |
| 1 | 3 | 2 | 1.50 | ✗ does not fit |
| 4 | 6 | 5 | 1.20 | ✗ does not fit |
Greedy total value 25; the DP optimum is also 25 (items 2 + 6).
- Sort the items by value/weight ratio in decreasing order; take each one that fits. O(n log n).
- For the fractional knapsack this rule is optimal; for the 0/1 version there is no guarantee.
- Bad example: (value 2, weight 1) and (value W, weight W). Greedy takes 2, the optimum is W: ratio 5 for W = 10, W/2 in general; it grows without bound as W grows.
- Improved rule: max(greedy value, the most valuable single item that fits) → a 2-approximation (at least half the optimum).
- Better still: an FPTAS that scales the values and runs DP, (1 − ε) · OPT, O(n³/ε).
Power Iteration
The Power Iteration Algorithm
- An iterative method that approximates a matrix's largest (dominant, largest in absolute value) eigenvalue λ1 and a corresponding eigenvector v1.
- Instead of computing all eigenvalues exactly (O(n³)), it gives only an estimate of the dominant eigenvalue and eigenvector.
- Each step is just one matrix–vector product (O(n²), or O(nonzeros) for a sparse matrix): effective on very large matrices.
A = [[2, 0], [0, 1]], x₀ = (1, 1): xk ∝ (2k, 1). Each product doubles the v₁ component and leaves the other unchanged; the normalized vector turns step by step toward v₁ = (1, 0) (error ∝ (1/2)k).
Power Iteration
Power Iteration: Steps and Why It Works
- Fix a matrix A and choose a random (nonzero) start vector x0.
- Multiply the vector by the matrix to get a new vector: y = A · xk.
- Normalize the new vector to norm (length) 1: xk+1 = y / ‖y‖.
- Repeat for the desired number of steps or until the change is small.
- xk → v1; the eigenvalue estimate is the Rayleigh quotient λ ≈ xkT A xk.
Why? If x0 = c1v1 + c2v2 + … then
- Since |λ2/λ1| < 1, the other components die out geometrically.
- The error shrinks by a factor of ≈ |λ2/λ1| per step: if the ratio is close to 1, convergence is slow.
- Condition: c1 ≠ 0 (x0 must not be orthogonal to v1) and |λ1| > |λ2|.
Power Iteration · Step by step
Power Iteration Step by Step
A = [[2, 1], [1, 3]], eigenvalues λ₁ = (5 + √5)/2 ≈ 3.618 and λ₂ ≈ 1.382. Start x₀ = (1, 0); error ‖x₀ − v₁‖ = 0.974.
k = 1: A·x = (2.000, 1.000), normalized x = (0.894, 0.447); λ ≈ 3.000, error 0.5465.
k = 2: A·x = (2.236, 2.236), normalized x = (0.707, 0.707); λ ≈ 3.500, error 0.2313.
k = 3: A·x = (2.121, 2.828), normalized x = (0.600, 0.800); λ ≈ 3.600, error 0.0899.
k = 4: x = (0.5547, 0.8321); λ ≈ 3.61538; error 3.4 · 10⁻². Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.
k = 5: x = (0.5369, 0.8437); λ ≈ 3.61765; error 1.3 · 10⁻². Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.
k = 6: x = (0.5300, 0.8480); λ ≈ 3.61798; error 5.0 · 10⁻³. Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.
k = 7: x = (0.5274, 0.8496); λ ≈ 3.61803; error 1.9 · 10⁻³. Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.
k = 8: x = (0.5264, 0.8503); λ ≈ 3.61803; error 7.3 · 10⁻⁴. Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.
k = 9: x ≈ (0.5260, 0.8505) ≈ v₁ = (0.5257, 0.8507), λ ≈ 3.618034; error ≈ 2.8 · 10⁻⁴. On a log scale the error plot is a straight line: linear (geometric) convergence.
Power Iteration
Power Iteration: Java and a 3 × 3 Example
double[] power(double[][] a, int steps) { int n = a.length; double[] x = new double[n]; Arrays.fill(x, 1.0); // x₀ ≠ 0, not ⊥ v₁ for (int k = 0; k < steps; k++) { double[] y = new double[n]; for (int i = 0; i < n; i++) // y = A·x for (int j = 0; j < n; j++) y[i] += a[i][j] * x[j]; double norm = 0; for (double yi : y) norm += yi * yi; norm = Math.sqrt(norm); if (norm == 0) break; // A·x = 0 for (int i = 0; i < n; i++) // ‖x‖ = 1 x[i] = y[i] / norm; } return x; }
Example matrix (rows):
(0.511 0.32 0.12) · (0.211 0.62 0.12) · (0.311 0.12 0.16); x₀ = (1, 1, 1)
| k | x₁ | x₂ | x₃ | ‖A·x‖ |
|---|---|---|---|---|
| 1 | 0.6474 | 0.6474 | 0.4023 | 1.4690 |
| 2 | 0.6533 | 0.6533 | 0.3827 | 0.8974 |
| 3 | 0.6539 | 0.6539 | 0.3807 | 0.9005 |
| 4 | 0.6539 | 0.6539 | 0.3805 | 0.9008 |
| 5 | 0.6539 | 0.6539 | 0.3805 | 0.9008 |
| 6 | 0.6539 | 0.6539 | 0.3805 | 0.9008 |
| 8 | 0.6539 | 0.6539 | 0.3805 | 0.9008 |
‖A·x‖ → λ₁ = 0.9008; v₁ ∝ (0.6539, 0.6539, 0.3805). λ₂/λ₁ ≈ 0.33 → fast convergence.
Power Iteration · Run the code step by step
Power Iteration: Code Trace
Start vector x₀ = (1, 1, 1). Any nonzero vector not orthogonal to v₁ will do.
y[0] = row 0 of A · x = 0.511·1 + 0.32·1 + 0.12·1 = 0.9510.
y[1] = row 1 of A · x = 0.211·1 + 0.62·1 + 0.12·1 = 0.9510.
y[2] = row 2 of A · x = 0.311·1 + 0.12·1 + 0.16·1 = 0.5910.
‖y‖ = √(0.9510² + 0.9510² + 0.5910²) = 1.4690.
x = y / ‖y‖ = (0.6474, 0.6474, 0.4023): a new direction of length 1; the numbers neither blow up nor vanish.
k = 1: y = A·x = (0.5862, 0.5862, 0.3434). The ratios of the components barely changed.
‖y‖ = 0.8974 → x = (0.6533, 0.6533, 0.3827). The direction changed very little.
k = 2: y = A·x = (0.5888, 0.5888, 0.3428). The ratios of the components barely changed.
‖y‖ = 0.9005 → x = (0.6539, 0.6539, 0.3807). ‖y‖ is approaching λ₁ = 0.9008.
steps = 3 iterations done: x = (0.6539, 0.6539, 0.3807) ≈ v₁. In the next iteration the last component becomes 0.3805; after that the four digits no longer change.
double[] power(double[][] a, int steps) { int n = a.length; double[] x = new double[n]; Arrays.fill(x, 1.0); // x₀ ≠ 0, not orthogonal to v₁ for (int k = 0; k < steps; k++) { double[] y = new double[n]; for (int i = 0; i < n; i++) // y = A·x for (int j = 0; j < n; j++) y[i] += a[i][j] * x[j]; double norm = 0; for (double yi : y) norm += yi * yi; norm = Math.sqrt(norm); if (norm == 0) break; // A·x = 0 for (int i = 0; i < n; i++) // ‖x‖ = 1 x[i] = y[i] / norm; } return x; }
Power Iteration
Power Iteration: Limits and Applications
Strengths
- Very simple; only matrix–vector products.
- Works on sparse, huge matrices (billions of rows).
- Memory O(n): the matrix is not modified.
Limits
- Finds only the dominant eigenvalue.
- Very slow if |λ2| ≈ |λ1|; may not converge if |λ1| = |λ2|.
- Fails in theory if x0 is orthogonal to v1 (c1 = 0).
- PageRank: the dominant eigenvector of the web link matrix gives the importance of pages.
- Variants: inverse iteration (smallest eigenvalue), shifted iteration, Lanczos/Arnoldi; the first principal component in PCA.
Summary
Summary
| Problem | Algorithm | Ratio | Time |
|---|---|---|---|
| TSP (general) | nearest neighbor (heuristic) | no guarantee | O(n²) |
| Metric TSP | MST + preorder + shortcuts / Christofides | 2 / 1.5 | O(n²) / O(n³) |
| TSP | 2-opt local search | no guarantee | O(n²) / pass |
| Machine scheduling | list scheduling / LPT | 2 − 1/m / 4/3 − 1/(3m) | O(n log m) / O(n log n) |
| Vertex cover | pick an edge, add both endpoints | 2 | O(V + E) |
| 0/1 knapsack | max(ratio-sorted, best single item) / FPTAS | 2 / 1 + ε | O(n log n) / O(n³/ε) |
| Dominant eigenvalue | power iteration | error ∝ |λ₂/λ₁|k | O(n²) / step |
Summary
Common Mistakes
Summary
Sample Problems
- A minimization algorithm finds 30; the optimum is 20. What is the ratio on this input?
- Nearest neighbor from city 2 on the 5-city TSP example: tour and length?
- Path 1–2–3–4–5–6, edges taken in order (1,2), …, (5,6): how many vertices does the 2-approximation pick? The optimum?
- m = 2, jobs 3, 3, 2, 2, 2: what are Cmax for list scheduling and LPT, and the optimum?
- Simulated annealing: probability of accepting a move with Δ = 5 at T = 10 and at T = 1?
- W = 10, items (value, weight) (2, 1) and (10, 10): what do ratio greedy and the improved rule find?
Show answers
- C/C* = 30/20 = 1.5.
- 2 → 0 (15) → 1 (10) → 3 (25) → 4 (40) → 2 (35): 125.
- (1,2), (3,4), (5,6) are picked → 6 vertices; optimum {2, 4, 5} → 3; the ratio is exactly 2.
- List: 3 | 3, then 2 → M1 (5), 2 → M2 (5), 2 → M1 (7) → 7; LPT has the same order → 7; optimum {3, 3} | {2, 2, 2} = 6; 7/6 = 4/3 − 1/6 (the bound is tight).
- e−5/10 ≈ 0.61; e−5/1 ≈ 0.0067. As the temperature drops, worsening moves are almost never accepted.
- Greedy takes (2, 1) first, (10, 10) does not fit → 2; improved rule: max(2, 10) = 10 = optimum.
Summary
Exercises
- a(0,0), b(4,3), c(4,0), d(0,3), and the tour has edges (a,b), (c,d). What is the 2-opt gain Δ?
- Build a job list for which list scheduling reaches the ratio 5/3 with m = 3.
- How many vertices does the 2-approximation pick on the complete graph K5? What is the optimum?
- A = [[2, 0], [0, 1]], x0 = (1, 1). After how many steps is the second component of xk below 10⁻³ times the first?
- In simulated annealing with T₀ = 100 and T ← 0.997 · T at each step, after how many steps does T drop below 1?
- Write the 2-approximation in Java so that it also returns the chosen matching.
Answers / hints
- d(a,c) + d(b,d) − d(a,b) − d(c,d) = 4 + 4 − 5 − 5 = −2 → make the move.
- Six jobs of 1 unit, then one job of 3 units: list 2 + 3 = 5; optimum {3} | {1,1,1} | {1,1,1} = 3 → 5/3.
- Two disjoint edges are picked → 4 vertices; optimum n − 1 = 4 (ratio 1).
- xk ∝ (2k, 1): 2−k < 10⁻³ → k = 10 (2¹⁰ = 1024).
- 100 · 0.997k < 1 → k > ln 100 / (−ln 0.997) ≈ 1532.8 → 1533 steps.
- Add every edge with both endpoints unmarked to the matching and mark both endpoints; the cover = the endpoints of the matching.
Summary
References
- Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., 2022 (Chapter 35: Approximation Algorithms).
- V. V. Vazirani, Approximation Algorithms, Springer, 2001.
- D. P. Williamson and D. B. Shmoys, The Design of Approximation Algorithms, Cambridge University Press, 2011.
- R. L. Graham, "Bounds on Multiprocessing Timing Anomalies", SIAM Journal on Applied Mathematics 17(2), 1969.
- N. Christofides, "Worst-Case Analysis of a New Heuristic for the Travelling Salesman Problem", Report 388, Carnegie Mellon University, 1976.
- M. Sipser, Introduction to the Theory of Computation, 3rd ed., 2012 (the 3-SAT → vertex cover reduction).
- G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., 2013 (power iteration).