Chapter 6: Approximation Algorithms

Advanced Algorithms
Approximation ratioTraveling salesman (TSP)Nearest neighbor and MST2-opt local search Job schedulingVertex coverKnapsackPower iteration
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Concepts

Explain what an approximation algorithm is, the approximation ratio, and how it differs from a heuristic.

TSP

Apply nearest neighbor, the MST-based 2-approximation and 2-opt local search to the traveling salesman problem step by step.

Scheduling and cover

Write and trace machine scheduling (list scheduling, LPT) and the vertex cover 2-approximation in Java; prove their ratios.

Numerical convergence

Compute how power iteration converges to the largest eigenvalue and its eigenvector.

Introduction

Approximation Algorithms

  • For some problems finding the exact (optimal) solution is very hard or impossible in practice: TSP, vertex cover, machine scheduling and knapsack are NP-hard; if P ≠ NP they have no polynomial-time exact algorithms.
  • Instead of looking for the perfect solution, approximation algorithms produce solutions as close as possible to the optimum.
  • They run in polynomial time: hard problems can be solved in reasonable time.
  • It can often be proven that the solution is within a fixed ratio or percentage of the optimum.
  • Preferred when computing resources (time, memory) are limited.
The trade-off: give up a little optimality, gain efficiency and a guarantee on solution quality.

Introduction

The Balance Scale Analogy

  • When a weight is placed on one pan of a scale, it can be hard to put exactly the same weight on the other pan.
  • Instead, we use the small weights at hand to balance the large one as closely as possible.
  • If we only have 20, 10, 5 and 1 kg, each usable once, the closest balance to 37 kg is 36 kg.
37 kg201051target: 3720 + 10 + 5 + 1 = 36gap: 1 kg (2.7%)
An approximation algorithm works the same way: instead of searching for the exact balance (the optimum), it settles for one that is found quickly and whose error is bounded.

Introduction

Approximation Ratio

C: cost of the solution the algorithm finds, C*: cost of the optimal solution. If for every input the algorithm satisfies

max( C / C*, C* / C ) ≤ ρ(n)

it is a ρ(n)-approximation algorithm (ρ ≥ 1; ρ = 1 means the algorithm is optimal).

MinimizationC ≤ ρ · C*. Example: a 2-approximation for vertex cover picks at most twice as many vertices as the optimum.
MaximizationC ≥ C* / ρ. Example: a 2-approximation for knapsack guarantees at least half the optimal value.
Minimization (ρ = 2): C* ≤ C ≤ ρ · C*0C*20C30ρ · C*40Maximization (ρ = 2): C* / ρ ≤ C ≤ C*0C* / ρ10C15C*20

The ratio is a worst-case guarantee; in practice solutions are often much closer to the optimum.

Introduction

Approximation Algorithms, Heuristics and Approximation Schemes

ApproachGuaranteeExample
Exact algorithmOptimal, but exponential time on NP-hard problemsBrute-force TSP: (n−1)!/2 tours
Approximation algorithmPolynomial time + a proven ratio ρVertex cover: ρ = 2
PTASRatio (1 + ε) for every fixed ε > 0, polynomial in n (may be exponential in ε)Euclidean TSP (Arora, 1998)
FPTASRatio (1 + ε), polynomial in both n and 1/εKnapsack: O(n³/ε)
HeuristicNo guarantee; usually good in practice2-opt, nearest neighbor
MetaheuristicNo guarantee; a problem-independent search frameworkGenetic algorithms, ant colony

There are also randomized approximation algorithms such as randomized rounding; they guarantee a ratio in expectation.

Introduction

Applications of Approximation Algorithms

Traveling salesman (TSP)

Choosing the route that visits every customer and returns with the shortest total length: logistics, parcel delivery, circuit-board drilling.

Knapsack

Choosing the items that fit and give the most value: budget, cargo and portfolio planning.

Job scheduling

Running several jobs or processes on machines/processors as efficiently as possible.

Vertex cover and numerical methods

Placing the fewest monitoring points in a network; converging iteratively to an eigenvalue of a huge matrix (PageRank).

Introduction

Kinds of Approximation Algorithms

Simplification

Solve a simpler version of the problem (e.g. relax integrality: LP relaxation, an MST lower bound for TSP).

Greedy

At each step take the option that looks best right now (nearest neighbor, LPT, ratio-sorted knapsack).

Local search

Improve the current solution with small changes; stop when no better neighbor remains (2-opt).

Primal–dualGrow primal and dual LP solutions together; common for covering problems.
RandomizedRandomized rounding or sampling; a guarantee on the expected ratio.
MetaheuristicSimulated annealing, genetic algorithms, ant colony.

Traveling Salesman

The Traveling Salesman Problem (TSP)

  • Optimization problem: a salesman wants to visit each of a set of cities once with the shortest total distance and return to the starting city.
  • Combinatorial problem: a solution is the order in which the cities are visited (a permutation).
  • NP-hard: finding the best route for many cities is hard in practice. The shortest path between two vertices (Dijkstra), however, is solvable in polynomial time; do not confuse the two problems.
  • Input: n cities (0, 1, 2, … or A, B, C, …) and dij: the distance between cities i and j.
  • Symmetric TSP: dij = dji; in graph terms: the lightest Hamiltonian cycle in a complete graph.
d01234
0010152025
1100352530
2153503035
3202530040
4253035400

Example: the distance matrix of 5 cities.

Traveling Salesman

Solving the Problem

Exact solutionCheck every possible route (brute force) or use dynamic programming (Held–Karp, O(n² · 2n)).
Approximate solutionsHeuristic algorithms: nearest neighbor, the MST-based 2-approximation, Christofides, genetic algorithms.
Optimization toolsLocal search (2-opt), simulated annealing, ant colony optimization.
n(n−1)!/2 distinct tours
512
10181,440
15≈ 4.4 · 1010
20≈ 6.1 · 1016
25≈ 3.1 · 1023

The start city is fixed and both directions count once. Even at 10⁹ tours per second, n = 25 would take ≈ 10⁷ years.

Traveling Salesman

The Nearest Neighbor Heuristic

  1. Pick a start city and mark it visited.
  2. From the current city go to the nearest unvisited city (greedy choice).
  3. Repeat step 2 until every city is visited.
  4. Return to the start city from the last one to close the tour.

Pros

  • Very simple; O(n²) time.
  • Usually gives a tour about 25% above the optimum.
  • A good initial tour for local search.

Cons

  • There is no constant approximation ratio; even for metric TSP the ratio can be Θ(log n).
  • The final return edge can be very long.

Traveling Salesman · Run the code step by step

Nearest Neighbor: Code Trace

0011223344010152025100352530153503035202530040253035400tour: 0

Start city 0: visited[0] = true, tour = [0], total = 0.

0011223344010152025100352530153503035202530040253035400tour: 0

Scan row 0, unvisited: d[0][1] = 10, d[0][2] = 15, d[0][3] = 20, d[0][4] = 25 → the smallest is 1 (10).

0011223344010152025100352530153503035202530040253035400tour: 0 → 1

tour[1] = 1, total += 10 → 10; now cur = 1.

0011223344010152025100352530153503035202530040253035400tour: 0 → 1

Scan row 1, unvisited: d[1][2] = 35, d[1][3] = 25, d[1][4] = 30 → the smallest is 3 (25).

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3

tour[2] = 3, total += 25 → 35; now cur = 3.

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3

Scan row 3, unvisited: d[3][2] = 30, d[3][4] = 40 → the smallest is 2 (30).

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3 → 2

tour[3] = 2, total += 30 → 65; now cur = 2.

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3 → 2

Scan row 2, unvisited: d[2][4] = 35 → the smallest is 4 (35).

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3 → 2 → 4

tour[4] = 4, total += 35 → 100; now cur = 4.

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3 → 2 → 4 → 0

Every city is visited; return to the start: total += d[4][0] = 25 → 125.

0011223344010152025100352530153503035202530040253035400tour: 0 → 1 → 3 → 2 → 4 → 0

Tour 0 → 1 → 3 → 2 → 4 → 0, length 125. (Checked by brute force: optimal for this example.)

int[] nearestNeighbor(int[][] d, int start) {
  int n = d.length, total = 0, cur = start;
  boolean[] visited = new boolean[n];
  int[] tour = new int[n + 1];
  tour[0] = start; visited[start] = true;
  for (int k = 1; k < n; k++) {
    int nearest = -1, minD = Integer.MAX_VALUE;
    for (int j = 0; j < n; j++)
      if (!visited[j] && d[cur][j] < minD) {
        minD = d[cur][j]; nearest = j; }
    // go to the nearest city
    tour[k] = nearest; visited[nearest] = true;
    total += minD; cur = nearest;
  }
  // return to the start
  tour[n] = start; total += d[cur][start];
  return tour;
}

Traveling Salesman

Comparison with Brute Force: 12 Distinct Tours

TourLength
0 → 1 → 3 → 2 → 4 → 0125
0 → 1 → 3 → 4 → 2 → 0125
0 → 1 → 4 → 2 → 3 → 0125
0 → 1 → 4 → 3 → 2 → 0125
0 → 2 → 3 → 1 → 4 → 0125
0 → 2 → 4 → 1 → 3 → 0125
TourLength
0 → 1 → 2 → 3 → 4 → 0140
0 → 1 → 2 → 4 → 3 → 0140
0 → 2 → 1 → 3 → 4 → 0140
0 → 2 → 1 → 4 → 3 → 0140
0 → 3 → 1 → 2 → 4 → 0140
0 → 3 → 2 → 1 → 4 → 0140
  • (5−1)!/2 = 12 tours tried: the optimum is 125; nearest neighbor also found 125 (a lucky case for this example).
  • This matrix violates the triangle inequality: d12 = 35 > d10 + d02 = 10 + 15. So the example is not metric.

Traveling Salesman

The Triangle Inequality and Approximation Limits

Metric TSP: dij ≤ dik + dkj for all i, j, k (going directly is never longer)
CaseRatioDescription
General TSPnoneIf P ≠ NP, no polynomial-time ρ-approximation exists for any constant ρ: such an algorithm would solve the Hamiltonian cycle problem.
Metric TSP, MST-based2MST + preorder walk + shortcuts; O(n²).
Metric TSP, Christofides1.5MST + minimum perfect matching on odd-degree vertices + Euler tour (1976).
Euclidean TSP1 + εA PTAS exists for points in the plane (Arora, Mitchell).
Nearest neighborΘ(log n)No constant ratio is guaranteed, even in the metric case.

Traveling Salesman

MST-Based 2-Approximation for Metric TSP

ABCDEFG

A metric example: 7 cities in the plane; distances are Euclidean (the triangle inequality holds).

47.737.938.830.034.939.4ABCDEFG

1) Build a minimum spanning tree (MST) with Prim: total weight w(T) = 228.8. Deleting one edge of a tour leaves a spanning tree → w(T) ≤ OPT.

ABCDEFG

2) Walk the tree depth-first from A, twice (each edge once down, once back): A → D → B → C → E → F → E → C → B → D → G → D → A. Length 2 · w(T) = 457.6.

ABCDEFG

3) Skip repeated cities (shortcut): preorder A → D → B → C → E → F → G → A. Thanks to the triangle inequality, shortcuts never lengthen the path: tour = 287.7.

ABCDEFG

Result: tour ≤ 2 · w(T) ≤ 2 · OPT. Here tour = 287.7, optimum (brute force, 360 tours) = 282.5 → ratio 1.02 ≤ 2.

Local Search

Local Search: The 2-opt Move

abcd

Before: edges (a, b) and (c, d) cross.

abcd

After: (a, c) and (b, d); the path between them is reversed.

  • Remove two edges of the tour and reconnect the ends crosswise: the subpath from b to c is reversed.
  • Make the move if the gain Δ = d(a,c) + d(b,d) − d(a,b) − d(c,d) < 0. In the Euclidean plane every crossing can be removed this way.
  • When no improving move remains, the tour is a 2-opt local optimum (not necessarily the global optimum).

Local Search · Step by step

Improving a Tour with 2-opt

002020404060608080

28 cities, a random initial tour: length 1269.8. Edges keep crossing each other.

002020404060608080

After improving move 1: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −36.9 → length 1232.9.

002020404060608080

After improving move 2: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −8.9 → length 1224.0.

002020404060608080

After improving move 4: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −27.6 → length 1187.3.

002020404060608080

After improving move 8: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −32.6 → length 1111.7.

002020404060608080

After improving move 14: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −14.9 → length 1005.2.

002020404060608080

After improving move 22: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −37.0 → length 852.8.

002020404060608080

After improving move 32: (a,b), (c,d) removed (dashed), (a,c), (b,d) added (bold). Δ = −11.8 → length 595.9.

002020404060608080

Move 48, the last: no 2-opt move shortens the tour any more → local optimum, length 430.8 (34% of the start). No crossing edges remain.

Local Search

2-opt Local Search: Java

void twoOpt(int[] tour, double[][] d) {
  int n = tour.length;
  boolean improved = true;
  while (improved) {
    improved = false;
    for (int i = 0; i < n - 1; i++)
      for (int j = i + 2; j < n; j++) {
        int a = tour[i], b = tour[i + 1];
        int c = tour[j], e = tour[(j + 1) % n];
        double delta = d[a][c] + d[b][e]
                     - d[a][b] - d[c][e];
        if (delta < -1e-9) {  // shorter
          reverse(tour, i + 1, j);
          improved = true;
        }
      }
  }
}
One pass over the tourO(n²) candidate moves; each is evaluated in O(1), a reversal costs O(n).
Local optimumThe loop ends when no move gives a gain. The number of passes can be exponential in the worst case but is small in practice.
A good startStarting from a nearest-neighbor tour instead of a random one needs fewer moves. Generalizations: 3-opt, Lin–Kernighan.

Local Search · Run the code step by step

2-opt: Code Trace

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→1→2→3→0length = 18

Initial tour 0 → 1 → 2 → 3 → 0, length 18. Edges (0, 1) and (2, 3) cross.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→1→2→3→0length = 18pass 1

Pass 1 starts: improved = false. If no move is made in this pass, the loop ends.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→1→2→3→0length = 18out: 5 + 5in: 3 + 3delta = −4

i = 0, j = 2: replace (0,1) and (2,3) with (0,2) and (1,3). delta = 3 + 3 − 5 − 5 = −4 < 0 → the tour gets shorter.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14pass 1

reverse(tour, 1, 2): the path in between is reversed → [0, 2, 1, 3], length 14. improved = true.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14out: 3 + 4in: 4 + 3delta = 0

i = 0, j = 3: delta = 4 + 3 − 3 − 4 = 0 (the two edges are adjacent: e = a) → no move.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14out: 4 + 4in: 5 + 5delta = 2

i = 1, j = 3: delta = 5 + 5 − 4 − 4 = 2 → no move.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14pass 2

A move was made in the previous pass (improved = true), so the pass repeats from the start.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14out: 3 + 3in: 5 + 5delta = 4

i = 0, j = 2: delta = 5 + 5 − 3 − 3 = 4 → no move.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14out: 3 + 4in: 4 + 3delta = 0

i = 0, j = 3: delta = 4 + 3 − 3 − 4 = 0 (the two edges are adjacent: e = a) → no move.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14out: 4 + 4in: 5 + 5delta = 2

i = 1, j = 3: delta = 5 + 5 − 4 − 4 = 2 → no move.

0(0,0)1(3,4)2(3,0)3(0,4)tour: 0→2→1→3→0length = 14local optimum

No shortening found in the whole pass: the loop ends. Tour [0, 2, 1, 3], length 14: a 2-opt local optimum (and for these 4 points also the optimum).

void twoOpt(int[] tour, double[][] d) {
  int n = tour.length;
  boolean improved = true;
  while (improved) {
    improved = false;
    for (int i = 0; i < n - 1; i++)
      for (int j = i + 2; j < n; j++) {
        int a = tour[i], b = tour[i + 1];
        int c = tour[j], e = tour[(j + 1) % n];
        double delta = d[a][c] + d[b][e]
                     - d[a][b] - d[c][e];
        if (delta < -1e-9) {  // shorter
          reverse(tour, i + 1, j);
          improved = true;
        }
      }
  }
}

Local Search

Escaping Local Optima: Metaheuristics

Simulated annealing

A worsening move is also accepted with probability e−Δ/T; the "temperature" T is lowered slowly. It can escape local optima.

Genetic algorithm

A population of tours improved over generations by selection, crossover and mutation.

Ant colony

Ants leave more pheromone on the edges of short tours; later ants prefer edges with dense pheromone.

  • All three are common in TSP, vehicle routing, job scheduling and assignment problems.
  • None of them guarantees an approximation ratio; the result depends on the initial solution and the parameters. Typical settings: T₀ = 100, T ← 0.997 · T at each step; a population of 100, crossover at a random cut point, mutation probability 0.05.
local optimumglobal optimumworsening move, probability e^(−Δ/T)cost

Local search that accepts only improving moves stays in the left valley. Simulated annealing can climb over the hill at high T; as T drops, moves become pickier.

Job Scheduling

Job Scheduling

  • Optimization problem: planning jobs on given resources (workers, machines, processors) as efficiently as possible.
  • Goal: minimize the completion time of the jobs, or optimize other criteria (profit, lateness, deadlines).
Machine scheduling (makespan)

n jobs, m identical machines; job j takes pj. Minimize the time the last job finishes, Cmax (the makespan). NP-hard for m ≥ 2.

Job sequencing with deadlines

Each job takes 1 unit and has a profit and a deadline. Maximize the total profit of the jobs finished by their deadlines.

Machine scheduling: CM132M24M322Cmax = 5Job sequencing with deadlines: time slotsjob 4kâr 30t = 1job 3kâr 40t = 2t = 3job 1kâr 20t = 4job 2 (deadline 1) does not fit → total profit 90

The first problem has greedy approximation algorithms (list scheduling, LPT); the second is solved optimally by a greedy algorithm.

Job Scheduling · Step by step

Assigning Jobs to Machines: List Scheduling

M1J1J4M2J2M3J30123456789t

Three identical machines M1–M3. Jobs arrive in order: J1 (2), J2 (8), J3 (6) ready at t = 0; J4 (3) arrives at t = 3. List scheduling (Graham): the next job goes to the machine that becomes free first.

M1J1M2M30123456789t

J1 (time 2) → M1, runs in [0, 2] (all machines are free; the first one is chosen).

M1J1M2J2M30123456789t

J2 (time 8) → M2, runs in [0, 8] (M1 is busy until t = 2).

M1J1M2J2M3J30123456789t

J3 (time 6) → M3, runs in [0, 6] (M1 is busy until t = 2; M2 until t = 8).

M1J1M2J2M3J30123456789t

t = 2: M1 is free but no job is waiting; the machine stays idle until t = 3 (hatched area).

M1J1J4M2J2M3J30123456789t

J4 (time 3) → M1, runs in [3, 6] (M2 is busy until t = 8; M3 until t = 6).

M1J1J4M2J2M3J30123456789tCmax = 8

All jobs assigned: Cmax = 8 (M2). Lower bound: OPT ≥ the longest job = 8 → this schedule is optimal. In general, though, list scheduling only guarantees a ratio of (2 − 1/m).

Job Scheduling

List Scheduling: A (2 − 1/m)-Approximation

Two lower bounds: OPT ≥ Σpj / m (the work must be shared) and OPT ≥ max pj.

  1. Let job j, the last to finish, start at time sj.
  2. Before sj all machines were busy (otherwise j would have started earlier) → sj ≤ (Σp − pj) / m.
  3. Cmax = sj + pj ≤ Σp / m + (1 − 1/m) · pj ≤ OPT + (1 − 1/m) · OPT.
Cmax ≤ (2 − 1/m) · OPT; time O(n log m) (the earliest-free machine is kept in a heap).
The bound is tightm = 2, jobs in the order 1, 1, 2:
• List: M1 = 1 + 2 = 3, M2 = 1 → Cmax = 3
• Optimum: {1, 1} and {2} → 2
Ratio 3/2 = 2 − 1/2.ListM112M21C = 3OptimumM111M22C = 2In general, m(m − 1) jobs of 1 unit followed by a single job of m units reach the ratio 2 − 1/m.

Job Scheduling

Longest Processing Time First (LPT): A 4/3-Approximation

LPT (Longest Processing Time first): sort the jobs by decreasing time, then apply list scheduling. Jobs: 2, 3, 4, 6, 2, 2 and m = 3.

M126M232M3420123456789tCmax = 8

Given order: Cmax = 8

M16M242M33220123456789tCmax = 7

LPT order 6, 4, 3, 2, 2, 2: Cmax = 7 = OPT (≥ ⌈19/3⌉ = 7)

  • With long jobs first, the short jobs left at the end even out the imbalance: Cmax ≤ (4/3 − 1/(3m)) · OPT (Graham, 1969). Time O(n log n).
  • Tight example (m = 3): 5, 5, 4, 4, 3, 3, 3 → LPT 11, optimum 9 ({5, 4}, {5, 4}, {3, 3, 3}).

Job Scheduling · Run the code step by step

LPT Scheduling: Code Trace

jobs234622M0M1M2

Jobs 2, 3, 4, 6, 2, 2 and m = 3 machines. List scheduling in this order gives Cmax = 8.

jobs643222M0M1M2

LPT: sort the jobs by time in decreasing order → 6, 4, 3, 2, 2, 2.

jobs643222M00M10M20load

All three machines have load 0. Each job goes to the least loaded machine at that moment.

jobs643222M066M10M20load

p = 6: the least loaded machine is k = 0 (load 0) (on ties, < keeps the first machine) → load = [6, 0, 0].

jobs643222M066M144M20load

p = 4: the least loaded machine is k = 1 (load 0) → load = [6, 4, 0].

jobs643222M066M144M233load

p = 3: the least loaded machine is k = 2 (load 0) → load = [6, 4, 3].

jobs643222M066M144M2325load

p = 2: the least loaded machine is k = 2 (load 3) → load = [6, 4, 5].

jobs643222M066M1426M2325load

p = 2: the least loaded machine is k = 1 (load 4) → load = [6, 6, 5].

jobs643222M066M1426M23227load

p = 2: the least loaded machine is k = 2 (load 5) → load = [6, 6, 7].

jobs643222M066M1426M23227loadCmax = 7

Makespan Cmax = 7. Since the lower bound is ⌈19/3⌉ = 7, this schedule is optimal.

int lpt(Integer[] jobs, int m) {
  Arrays.sort(jobs, Comparator.reverseOrder());
  int[] load = new int[m];
  for (int p : jobs) {
    int k = 0;              // least loaded
    for (int i = 1; i < m; i++)
      if (load[i] < load[k]) k = i;
    load[k] += p;
  }
  return Arrays.stream(load).max().getAsInt();
}

Vertex Cover

Vertex Cover

  • Covering problem: choose the fewest vertices that cover every edge of a graph. An edge is covered if at least one of its endpoints is chosen.
  • Goal: find a set C ⊆ V covering all edges with as few vertices as possible. The decision version ("is there a cover of k vertices?") is NP-complete.

A valid cover of size 6 (every edge is watched by a "guard")

A minimum cover of the same graph, size 3

Vertex Cover

Example: A Four-Vertex Graph

  • Vertices: A, B, C, D
  • Edges: (A, B), (A, C), (B, C), (B, D), (C, D)
  • {A, B, C} is a vertex cover: every edge has at least one endpoint in the set.
  • But it is not the smallest: {B, C} also covers all five edges.
  • A single vertex is not enough (B does not cover (A, C)) → the optimum is 2.
ABCD

{A, B, C}: valid, size 3

ABCD

{B, C}: minimum, size 2

Vertex Cover

Vertex Cover: Solution Approaches

Exact solutionCheck every possible vertex set: 2n subsets × O(m) checks.
Approximate solutionsGreedy algorithms (highest-degree vertex), the edge-picking 2-approximation.
Optimization toolsLinear programming (LP relaxation + rounding), branch and bound.
AEDFCBGIJLHK

Minimum cover {A, B, H, I, J} (5 vertices)

Vertex Cover

Vertex Cover Algorithm

Naive idea:

  1. Start with an empty cover.
  2. Pick an uncovered edge and add one of its endpoints to the set.
  3. Repeat until every edge is covered.

Adding only one endpoint gives no guarantee. The correct 2-approximation:

  1. Pick an uncovered edge (u, v) and add both u and v.
  2. Every edge touching u or v is now covered; repeat until no edges remain.
m12345

If the leaf end of every edge is chosen: 5 vertices; the optimum is {m}, 1 vertex. On a star with n leaves the ratio is n.

Vertex Cover · Run the code step by step

Vertex Cover 2-Approximation: Code Trace

AEDFCBGIJLHK

The cover is empty. The edges are visited in order: (A,B), (A,C), …, (J,L).

AEDFCBGIJLHK

(A, B): neither endpoint is in the cover → add A and B. These edges form a matching.

AEDFCBGIJLHK

(A, C): A is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(A, D): A is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(A, E): A is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(A, F): A is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(B, C): B is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(G, H): neither endpoint is in the cover → add G and H. These edges form a matching.

AEDFCBGIJLHK

(G, I): G is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(H, I): H is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(H, K): H is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(I, J): neither endpoint is in the cover → add I and J. These edges form a matching.

AEDFCBGIJLHK

(J, K): J is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

(J, L): J is already in the cover → the edge is covered, skip.

AEDFCBGIJLHK

Result {A, B, G, H, I, J}: 6 vertices. The optimum is 5 ({A, B, H, I, J}) → ratio 6/5 = 1.2 ≤ 2.

Set<Character> vertexCover(char[][] edges) {
  Set<Character> cover = new HashSet<>();
  for (char[] e : edges) {
    char u = e[0], v = e[1];
    if (!cover.contains(u) && !cover.contains(v)) {
      cover.add(u); cover.add(v); }  // both endpoints
  }
  return cover;
}

Vertex Cover

Why At Most 2 × Optimum?

  1. Let M be the set of edges the algorithm picks. Both endpoints of each picked edge are added, so the edges of M share no endpoints (M is a matching).
  2. Any cover must contain a distinct vertex for each edge of M → OPT ≥ |M|.
  3. The algorithm picks exactly |C| = 2 |M| vertices → |C| ≤ 2 · OPT.
TimeO(V + E): each edge is examined once.
Greedy by degreePicking the highest-degree vertex is intuitive, but its ratio is only O(log n).
LP roundingSolve the relaxation xu + xv ≥ 1 and pick the vertices with xv ≥ 1/2 → again a 2-approximation.

No constant ratio noticeably better than 2 is known; a ratio of 2 − ε has been shown impossible (under the Unique Games Conjecture).

Vertex Cover · Step by step

NP-Completeness: Reducing 3-SAT to Vertex Cover

x₁¬x₁x₂¬x₂x₁x₁x₂¬x₁¬x₂¬x₂¬x₁x₂x₂C₁C₂C₃

φ = (x₁ ∨ x₁ ∨ x₂) ∧ (¬x₁ ∨ ¬x₂ ∨ ¬x₂) ∧ (¬x₁ ∨ x₂ ∨ x₂). One edge per variable (top boxes), one triangle per clause; each literal in a clause is joined to the variable vertex with the same name. l = 2 variables, m = 3 clauses → k = l + 2m = 8.

x₁¬x₁x₂¬x₂x₁x₁x₂¬x₁¬x₂¬x₂¬x₁x₂x₂C₁C₂C₃

Assignment x₁ = false: pick ¬x₁ from the variable edge. Covered edges fade.

x₁¬x₁x₂¬x₂x₁x₁x₂¬x₁¬x₂¬x₂¬x₁x₂x₂C₁C₂C₃

Assignment x₂ = true: pick x₂. Each variable edge is covered by exactly one vertex (l = 2 vertices).

x₁¬x₁x₂¬x₂x₁x₁x₂¬x₁¬x₂¬x₂¬x₁x₂x₂C₁C₂C₃

In C₁ the true literal x₂ (its link edge is already covered) is left out; the other two vertices (x₁, x₁) are added.

x₁¬x₁x₂¬x₂x₁x₁x₂¬x₁¬x₂¬x₂¬x₁x₂x₂C₁C₂C₃

In C₂ the true literal ¬x₁ is left out; the vertices ¬x₂ and ¬x₂ are added.

x₁¬x₁x₂¬x₂x₁x₁x₂¬x₁¬x₂¬x₂¬x₁x₂x₂C₁C₂C₃

In C₃ the true literal ¬x₁ is left out; x₂ and x₂ are added. 8 vertices, all 20 edges covered: φ is satisfiable ⇔ a cover of size k = l + 2m exists.

Knapsack

Knapsack: Greedy by Ratio

ItemValueWeightRatioCapacity 10
62072.86✓ taken (weight 7)
3842.00✗ does not fit
71581.88✗ does not fit
2531.67✓ taken (weight 10)
51061.67✗ does not fit
1321.50✗ does not fit
4651.20✗ does not fit

Greedy total value 25; the DP optimum is also 25 (items 2 + 6).

  • Sort the items by value/weight ratio in decreasing order; take each one that fits. O(n log n).
  • For the fractional knapsack this rule is optimal; for the 0/1 version there is no guarantee.
  • Bad example: (value 2, weight 1) and (value W, weight W). Greedy takes 2, the optimum is W: ratio 5 for W = 10, W/2 in general; it grows without bound as W grows.
  • Improved rule: max(greedy value, the most valuable single item that fits) → a 2-approximation (at least half the optimum).
  • Better still: an FPTAS that scales the values and runs DP, (1 − ε) · OPT, O(n³/ε).

Power Iteration

The Power Iteration Algorithm

  • An iterative method that approximates a matrix's largest (dominant, largest in absolute value) eigenvalue λ1 and a corresponding eigenvector v1.
  • Instead of computing all eigenvalues exactly (O(n³)), it gives only an estimate of the dominant eigenvalue and eigenvector.
  • Each step is just one matrix–vector product (O(n²), or O(nonzeros) for a sparse matrix): effective on very large matrices.
x₀x₁x₂x₃v₁

A = [[2, 0], [0, 1]], x₀ = (1, 1): xk ∝ (2k, 1). Each product doubles the v₁ component and leaves the other unchanged; the normalized vector turns step by step toward v₁ = (1, 0) (error ∝ (1/2)k).

Two ideas meet here: the result is an approximation (an estimate) rather than an exact value, and the iterates converge to the true value. Google's PageRank is the most famous application of the method.

Power Iteration

Power Iteration: Steps and Why It Works

  1. Fix a matrix A and choose a random (nonzero) start vector x0.
  2. Multiply the vector by the matrix to get a new vector: y = A · xk.
  3. Normalize the new vector to norm (length) 1: xk+1 = y / ‖y‖.
  4. Repeat for the desired number of steps or until the change is small.
  5. xk → v1; the eigenvalue estimate is the Rayleigh quotient λ ≈ xkT A xk.

Why? If x0 = c1v1 + c2v2 + … then

Akx0 = λ1k [ c1v1 + c2(λ2/λ1)k v2 + … ]
  • Since |λ2/λ1| < 1, the other components die out geometrically.
  • The error shrinks by a factor of ≈ |λ2/λ1| per step: if the ratio is close to 1, convergence is slow.
  • Condition: c1 ≠ 0 (x0 must not be orthogonal to v1) and |λ1| > |λ2|.

Power Iteration · Step by step

Power Iteration Step by Step

v₁v₂x₀110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

A = [[2, 1], [1, 3]], eigenvalues λ₁ = (5 + √5)/2 ≈ 3.618 and λ₂ ≈ 1.382. Start x₀ = (1, 0); error ‖x₀ − v₁‖ = 0.974.

v₁v₂x₁110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 1: A·x = (2.000, 1.000), normalized x = (0.894, 0.447); λ ≈ 3.000, error 0.5465.

v₁v₂x₂110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 2: A·x = (2.236, 2.236), normalized x = (0.707, 0.707); λ ≈ 3.500, error 0.2313.

v₁v₂x₃110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 3: A·x = (2.121, 2.828), normalized x = (0.600, 0.800); λ ≈ 3.600, error 0.0899.

v₁v₂x₄110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 4: x = (0.5547, 0.8321); λ ≈ 3.61538; error 3.4 · 10⁻². Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.

v₁v₂x₅110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 5: x = (0.5369, 0.8437); λ ≈ 3.61765; error 1.3 · 10⁻². Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.

v₁v₂x₆110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 6: x = (0.5300, 0.8480); λ ≈ 3.61798; error 5.0 · 10⁻³. Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.

v₁v₂x₇110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 7: x = (0.5274, 0.8496); λ ≈ 3.61803; error 1.9 · 10⁻³. Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.

v₁v₂x₈110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 8: x = (0.5264, 0.8503); λ ≈ 3.61803; error 7.3 · 10⁻⁴. Each step shrinks it by ≈ λ₂/λ₁ ≈ 0.38.

v₁v₂x₉110⁻¹10⁻²10⁻³10⁻⁴02468iteration kerror ‖xₖ − v₁‖

k = 9: x ≈ (0.5260, 0.8505) ≈ v₁ = (0.5257, 0.8507), λ ≈ 3.618034; error ≈ 2.8 · 10⁻⁴. On a log scale the error plot is a straight line: linear (geometric) convergence.

Power Iteration

Power Iteration: Java and a 3 × 3 Example

double[] power(double[][] a, int steps) {
  int n = a.length;
  double[] x = new double[n];
  Arrays.fill(x, 1.0); // x₀ ≠ 0, not ⊥ v₁
  for (int k = 0; k < steps; k++) {
    double[] y = new double[n];
    for (int i = 0; i < n; i++)  // y = A·x
      for (int j = 0; j < n; j++)
        y[i] += a[i][j] * x[j];
    double norm = 0;
    for (double yi : y) norm += yi * yi;
    norm = Math.sqrt(norm);
    if (norm == 0) break; // A·x = 0
    for (int i = 0; i < n; i++)  // ‖x‖ = 1
      x[i] = y[i] / norm;
  }
  return x;
}

Example matrix (rows):
(0.511 0.32 0.12) · (0.211 0.62 0.12) · (0.311 0.12 0.16); x₀ = (1, 1, 1)

kx₁x₂x₃‖A·x‖
10.64740.64740.40231.4690
20.65330.65330.38270.8974
30.65390.65390.38070.9005
40.65390.65390.38050.9008
50.65390.65390.38050.9008
60.65390.65390.38050.9008
80.65390.65390.38050.9008

‖A·x‖ → λ₁ = 0.9008; v₁ ∝ (0.6539, 0.6539, 0.3805). λ₂/λ₁ ≈ 0.33 → fast convergence.

Power Iteration · Run the code step by step

Power Iteration: Code Trace

A0.5110.320.120.2110.620.120.3110.120.16·x1.00001.00001.0000=ystart

Start vector x₀ = (1, 1, 1). Any nonzero vector not orthogonal to v₁ will do.

A0.5110.320.120.2110.620.120.3110.120.16·x1.00001.00001.0000=y0.9510k = 0

y[0] = row 0 of A · x = 0.511·1 + 0.32·1 + 0.12·1 = 0.9510.

A0.5110.320.120.2110.620.120.3110.120.16·x1.00001.00001.0000=y0.95100.9510k = 0

y[1] = row 1 of A · x = 0.211·1 + 0.62·1 + 0.12·1 = 0.9510.

A0.5110.320.120.2110.620.120.3110.120.16·x1.00001.00001.0000=y0.95100.95100.5910k = 0

y[2] = row 2 of A · x = 0.311·1 + 0.12·1 + 0.16·1 = 0.5910.

A0.5110.320.120.2110.620.120.3110.120.16·x1.00001.00001.0000=y0.95100.95100.5910‖y‖ = 1.4690k = 0

‖y‖ = √(0.9510² + 0.9510² + 0.5910²) = 1.4690.

A0.5110.320.120.2110.620.120.3110.120.16·x0.64740.64740.4023=y0.95100.95100.5910‖y‖ = 1.4690x ← y / ‖y‖k = 0

x = y / ‖y‖ = (0.6474, 0.6474, 0.4023): a new direction of length 1; the numbers neither blow up nor vanish.

A0.5110.320.120.2110.620.120.3110.120.16·x0.64740.64740.4023=y0.58620.58620.3434k = 1

k = 1: y = A·x = (0.5862, 0.5862, 0.3434). The ratios of the components barely changed.

A0.5110.320.120.2110.620.120.3110.120.16·x0.65330.65330.3827=y0.58620.58620.3434‖y‖ = 0.8974x ← y / ‖y‖k = 1

‖y‖ = 0.8974 → x = (0.6533, 0.6533, 0.3827). The direction changed very little.

A0.5110.320.120.2110.620.120.3110.120.16·x0.65330.65330.3827=y0.58880.58880.3428k = 2

k = 2: y = A·x = (0.5888, 0.5888, 0.3428). The ratios of the components barely changed.

A0.5110.320.120.2110.620.120.3110.120.16·x0.65390.65390.3807=y0.58880.58880.3428‖y‖ = 0.9005x ← y / ‖y‖k = 2

‖y‖ = 0.9005 → x = (0.6539, 0.6539, 0.3807). ‖y‖ is approaching λ₁ = 0.9008.

A0.5110.320.120.2110.620.120.3110.120.16·x0.65390.65390.3807=yresult

steps = 3 iterations done: x = (0.6539, 0.6539, 0.3807) ≈ v₁. In the next iteration the last component becomes 0.3805; after that the four digits no longer change.

double[] power(double[][] a, int steps) {
  int n = a.length;
  double[] x = new double[n];
  Arrays.fill(x, 1.0); // x₀ ≠ 0, not orthogonal to v₁
  for (int k = 0; k < steps; k++) {
    double[] y = new double[n];
    for (int i = 0; i < n; i++)  // y = A·x
      for (int j = 0; j < n; j++)
        y[i] += a[i][j] * x[j];
    double norm = 0;
    for (double yi : y) norm += yi * yi;
    norm = Math.sqrt(norm);
    if (norm == 0) break; // A·x = 0
    for (int i = 0; i < n; i++)  // ‖x‖ = 1
      x[i] = y[i] / norm;
  }
  return x;
}

Power Iteration

Power Iteration: Limits and Applications

Strengths

  • Very simple; only matrix–vector products.
  • Works on sparse, huge matrices (billions of rows).
  • Memory O(n): the matrix is not modified.

Limits

  • Finds only the dominant eigenvalue.
  • Very slow if |λ2| ≈ |λ1|; may not converge if |λ1| = |λ2|.
  • Fails in theory if x0 is orthogonal to v1 (c1 = 0).
  • PageRank: the dominant eigenvector of the web link matrix gives the importance of pages.
  • Variants: inverse iteration (smallest eigenvalue), shifted iteration, Lanczos/Arnoldi; the first principal component in PCA.

Summary

Summary

ProblemAlgorithmRatioTime
TSP (general)nearest neighbor (heuristic)no guaranteeO(n²)
Metric TSPMST + preorder + shortcuts / Christofides2 / 1.5O(n²) / O(n³)
TSP2-opt local searchno guaranteeO(n²) / pass
Machine schedulinglist scheduling / LPT2 − 1/m / 4/3 − 1/(3m)O(n log m) / O(n log n)
Vertex coverpick an edge, add both endpoints2O(V + E)
0/1 knapsackmax(ratio-sorted, best single item) / FPTAS2 / 1 + εO(n log n) / O(n³/ε)
Dominant eigenvaluepower iterationerror ∝ |λ₂/λ₁|kO(n²) / step

Summary

Common Mistakes

Taking a heuristic for an approximation algorithmNearest neighbor and 2-opt give good results but have no ratio guarantee; an approximation algorithm needs a proven ρ.
Writing the ratio backwardsFor minimization C ≤ ρ · C*, for maximization C ≥ C*/ρ. ρ is always ≥ 1.
Adding one endpoint in vertex coverAdding only one endpoint of an edge can be n times worse on a star graph; the 2-approximation adds both endpoints.
Forgetting the triangle inequalityThe shortcut step of the MST-based 2-approximation keeps the path from growing only for metric TSP.
Treating metaheuristics as exactSimulated annealing and genetic algorithms do not guarantee convergence to the optimum; run them several times with different seeds and parameters and keep the best result.
Skipping normalizationIf x is not normalized at every step of power iteration, the numbers overflow (|λ₁| > 1) or shrink to zero.

Summary

Sample Problems

  1. A minimization algorithm finds 30; the optimum is 20. What is the ratio on this input?
  2. Nearest neighbor from city 2 on the 5-city TSP example: tour and length?
  3. Path 1–2–3–4–5–6, edges taken in order (1,2), …, (5,6): how many vertices does the 2-approximation pick? The optimum?
  4. m = 2, jobs 3, 3, 2, 2, 2: what are Cmax for list scheduling and LPT, and the optimum?
  5. Simulated annealing: probability of accepting a move with Δ = 5 at T = 10 and at T = 1?
  6. W = 10, items (value, weight) (2, 1) and (10, 10): what do ratio greedy and the improved rule find?
Show answers
  1. C/C* = 30/20 = 1.5.
  2. 2 → 0 (15) → 1 (10) → 3 (25) → 4 (40) → 2 (35): 125.
  3. (1,2), (3,4), (5,6) are picked → 6 vertices; optimum {2, 4, 5} → 3; the ratio is exactly 2.
  4. List: 3 | 3, then 2 → M1 (5), 2 → M2 (5), 2 → M1 (7) → 7; LPT has the same order → 7; optimum {3, 3} | {2, 2, 2} = 6; 7/6 = 4/3 − 1/6 (the bound is tight).
  5. e−5/10 ≈ 0.61; e−5/1 ≈ 0.0067. As the temperature drops, worsening moves are almost never accepted.
  6. Greedy takes (2, 1) first, (10, 10) does not fit → 2; improved rule: max(2, 10) = 10 = optimum.

Summary

Exercises

  1. a(0,0), b(4,3), c(4,0), d(0,3), and the tour has edges (a,b), (c,d). What is the 2-opt gain Δ?
  2. Build a job list for which list scheduling reaches the ratio 5/3 with m = 3.
  3. How many vertices does the 2-approximation pick on the complete graph K5? What is the optimum?
  4. A = [[2, 0], [0, 1]], x0 = (1, 1). After how many steps is the second component of xk below 10⁻³ times the first?
  5. In simulated annealing with T₀ = 100 and T ← 0.997 · T at each step, after how many steps does T drop below 1?
  6. Write the 2-approximation in Java so that it also returns the chosen matching.
Answers / hints
  1. d(a,c) + d(b,d) − d(a,b) − d(c,d) = 4 + 4 − 5 − 5 = −2 → make the move.
  2. Six jobs of 1 unit, then one job of 3 units: list 2 + 3 = 5; optimum {3} | {1,1,1} | {1,1,1} = 3 → 5/3.
  3. Two disjoint edges are picked → 4 vertices; optimum n − 1 = 4 (ratio 1).
  4. xk ∝ (2k, 1): 2−k < 10⁻³ → k = 10 (2¹⁰ = 1024).
  5. 100 · 0.997k < 1 → k > ln 100 / (−ln 0.997) ≈ 1532.8 → 1533 steps.
  6. Add every edge with both endpoints unmarked to the matching and mark both endpoints; the cover = the endpoints of the matching.

Summary

References

  • Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., 2022 (Chapter 35: Approximation Algorithms).
  • V. V. Vazirani, Approximation Algorithms, Springer, 2001.
  • D. P. Williamson and D. B. Shmoys, The Design of Approximation Algorithms, Cambridge University Press, 2011.
  • R. L. Graham, "Bounds on Multiprocessing Timing Anomalies", SIAM Journal on Applied Mathematics 17(2), 1969.
  • N. Christofides, "Worst-Case Analysis of a New Heuristic for the Travelling Salesman Problem", Report 388, Carnegie Mellon University, 1976.
  • M. Sipser, Introduction to the Theory of Computation, 3rd ed., 2012 (the 3-SAT → vertex cover reduction).
  • G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., 2013 (power iteration).

The End

Chapter 6: Approximation Algorithms
Previous: Chapter 5: Branch and Bound · Next: Chapter 7: Randomized Algorithms
1 / 1 Sercan KÜLCÜ. All rights reserved.