Chapter 2: Algorithm Complexity
Introduction
Learning Outcomes
Distinguish time and space complexity, and best-, worst- and average-case analysis.
State the definitions of Big O, Ω and Θ; prove a bound by finding the constants c and n₀.
Count the steps of single, nested and multiplicative loops with summation formulas (f1, f2, f6).
Set up a recurrence for recursive code and solve it by expansion, a recursion tree or the Master Theorem (f3, f4, f5).
f1–f6), each representing a different growth class (n, n³, 2ⁿ, n, n log n, log* n).The Notion of Complexity
Algorithm Complexity
- Complexity theory measures an algorithm's use of resources: time and memory.
- Resource use is written as a function of the input size n: T(n), S(n).
- The goal is not to measure seconds but to see how the resource needs grow as the input grows.
The Notion of Complexity
Why Timing Is Not Enough
long start = System.currentTimeMillis(); linearSearch(arr, 10001); // 10^8 elements long end = System.currentTimeMillis(); System.out.println("Elapsed: " + (end - start) + " ms");
- This code times a linear search over an array of 10⁸ elements.
- The result depends on the machine, the processor, the JIT compiler, the cache and the current load.
- A single measurement does not tell us what happens as the input grows.
The Notion of Complexity
Counting Steps: linearSearch
for (int i = 0; i < a.length; i++) { if (a[i] == target) return i; } return -1;
| Operation | Worst-case count |
|---|---|
i = 0 | 1 |
i < a.length | n + 1 |
a[i] == target | n |
i++ | n |
return -1 | 1 |
| Total T(n) | 3n + 3 |
- Each basic operation counts as 1 unit (the RAM model). Worst case: the target is not in the array.
- The coefficient 3 and the +3 in 3n + 3 depend on the machine; what does not change is the linear growth in n → T(n) = Θ(n).
The Notion of Complexity · Run the code step by step
linearSearch: Tracing the Step Count
a = [4, 7, 2], target = 5 (not in the array: worst case). i = 0 is assigned and 0 < 3 is checked: 2 steps.
a[0] == target? 4 ≠ 5. One comparison so far.
i++ → i = 1, then 1 < 3 is checked: the loop continues.
a[1] == target? 7 ≠ 5. Two comparisons so far.
i++ → i = 2, then 2 < 3 is checked: the loop continues.
a[2] == target? 2 ≠ 5. Three comparisons so far.
i++ → i = 3. 3 < 3 is false: the condition was checked n + 1 = 4 times and the loop ends.
return -1. 12 steps in total = 3n + 3 (n = 3). A constant number (3) of operations per element: Θ(n).
for (int i = 0; i < a.length; i++) { if (a[i] == target) return i; } return -1;
The Notion of Complexity
Best, Worst and Average Case
Best case: the target is the first element; one comparison suffices → Θ(1).
Worst case: the target is absent (or last); n = 6 comparisons → Θ(n).
Average case: if the target is equally likely to be in any position, the average is (1 + 2 + … + n)/n = (n + 1)/2 = 3.5 comparisons → still Θ(n).
The Notion of Complexity
Space Complexity
Space complexity usually refers to auxiliary space: the input itself is not counted. In recursion every active call keeps a stack frame.
| Code | Extra space | Space | Time |
|---|---|---|---|
f1: single loop | only x and i | Θ(1) | Θ(n) |
creating new int[n] | auxiliary array of n elements | Θ(n) | Θ(n) |
f3: f3(n−1) + f3(n−1) | stack depth n (at most n frames at once) | Θ(n) | Θ(2ⁿ) |
f4: f4(n/2) + f4(n/2) | stack depth ≈ log₂ n | Θ(log n) | Θ(n) |
| Merge sort | auxiliary array for merging | Θ(n) | Θ(n log n) |
Asymptotic Notation
Big-O Notation
0 ≤ f(n) ≤ c · g(n) for all n ≥ n₀
- For large n, f never exceeds a constant multiple of g: g is an upper bound.
- Small n (n < n₀) do not matter; constant factors are absorbed into c.
- Example: 3n² + 5n + 2 ≤ 4n² for all n ≥ 6 → O(n²).
Asymptotic Notation
Big Omega (Ω) and Big Theta (Θ)
| Notation | Meaning | Definition (∃ c, n₀ > 0, for all n ≥ n₀) | Analogy |
|---|---|---|---|
| f = O(g) | upper bound | f(n) ≤ c · g(n) | f ≤ g |
| f = Ω(g) | lower bound | f(n) ≥ c · g(n) | f ≥ g |
| f = Θ(g) | tight bound | c₁ · g(n) ≤ f(n) ≤ c₂ · g(n) | f = g |
- f = Θ(g) ⇔ f = O(g) and f = Ω(g).
- 3n² + 5n + 2 is both O(n²) and O(n³), but it is Θ only of n².
- Linear search: worst case Θ(n), best case Θ(1); O(n) and Ω(1) over all inputs.
- The case (which input) and the bound (O, Ω, Θ) are separate ideas. Insertion sort is Θ(n²) in the worst case and Θ(n) in the best, so no single Θ covers all inputs. Merge sort, by contrast, is Θ(n log n) on every input.
Asymptotic Notation
Example: 3n² + 5n + 2 = Θ(n²)
Try c₂ = 4: 3n² + 5n + 2 ≤ 4n² ⇔ n² − 5n − 2 ≥ 0 ⇔ n ≥ 5.37. So n₀ = 6.
n = 6 → 140 ≤ 144 ✓
The inequality fails at n = 5 and holds from n = 6 on (the left side grows by 6n + 8, the right side by 8n + 4).
With c₁ = 3 the inequality holds for every n ≥ 1, since 5n + 2 > 0.
c₁ = 3, c₂ = 4, n₀ = 6 satisfy the definition: f(n) = Θ(n²). The constants are not unique; c₂ = 10, n₀ = 1 also works.
Asymptotic Notation
Rules of Asymptotic Arithmetic
Asymptotic Notation
Common Growth Classes
| Class | Name | Typical example | When n doubles | In this chapter |
|---|---|---|---|---|
| O(1) | Constant | printing the first element, array access | unchanged | — |
| O(log n) | Logarithmic | binary search, fast exponentiation, Euclid | +1 step | f6 (upper bound) |
| O(√n) | Square root | primality test (trial division) | ≈ 1,41× | — |
| O(n) | Linear | linear search, maximum, array sum | 2× | f1, f4 |
| O(n log n) | Linearithmic | merge sort, heapsort, quicksort (average) | a bit more than 2× | f5 |
| O(n²) | Quadratic | bubble, selection, insertion sort | 4× | — |
| O(n³) | Cubic | standard matrix multiplication | 8× | f2 |
| O(2ⁿ) | Exponential | all subsets, naive recursive Fibonacci | squared | f3 |
| O(n!) | Factorial | all permutations | blows up even faster | — |
The classes get slower from top to bottom. Polynomial classes (nk) are usually considered "efficient", exponential ones "inefficient". The true class of f6 is log* n, which grows even more slowly than log n.
Asymptotic Notation
Growth Curves
- For small n the curves are close; at n = 4, n² and 2ⁿ are equal (16).
- 2ⁿ passes 100 steps at n ≈ 6.6, n² after n = 10.
- n log n stays close to linear; log n is almost flat.
- The extremes drift further apart: at n = 10, n! reaches 3.6 million, while √n is only 10 at n = 100.
- Asymptotic analysis cares about the order of the curves for large n.
Asymptotic Notation
Growth in Numbers: Calculator
| f(n) | operations | time at 10⁹ ops/s | relative |
|---|
With "n × 2", the n column doubles, n² grows 4×, n³ 8×; log₂ n grows by only 1, while 2ⁿ is squared. A faster computer buys a constant factor; a better algorithm changes the growth class.
Loop Analysis
f1: Single Loop → O(n)
public int f1(int n) { int x = 0; for (int i = 0; i < n; i++) { x++; } return x; }
- Initialization:
int x = 0;sets x to 0; constant time, O(1). - Loop:
for (int i = 0; i < n; i++)runs i from 0 to n − 1: n times. - Increment:
x++;runs once per iteration; O(1).
x = n, so the return value is the step count itself: f1(16) = 16, f1(32) = 32.Loop Analysis
f1(4): Tracing the Code
x is set to zero. The loop has not started yet.
i = 0: condition 0 < 4 is true → enter the body.
x++ → x = 1. This is iteration 1.
i = 1: condition 1 < 4 is true → enter the body.
x++ → x = 2. This is iteration 2.
i = 2: condition 2 < 4 is true → enter the body.
x++ → x = 3. This is iteration 3.
i = 3: condition 3 < 4 is true → enter the body.
x++ → x = 4. This is iteration 4.
i = 4: condition 4 < 4 is false → the loop ends. The condition was checked n + 1 = 5 times.
return x → 4. The body ran exactly n = 4 times: Θ(n).
public int f1(int n) { int x = 0; for (int i = 0; i < n; i++) { x++; } return x; }
Loop Analysis
f2: Nested Loop → O(n³)
public int f2(int n) { int x = 0; for (int i = 0; i < n; i++) { for (int j = 0; j < i * i; j++) { x++; } } return x; }
- Initialization:
int x = 0;constant time, O(1). - Outer loop: i from 0 to n − 1: n times.
- Inner loop: j from 0 to i · i − 1: i² times.
- The number of inner iterations depends on the current value of i, so we cannot just say "n · n"; we have to add up the iteration counts.
Loop Analysis · Run the code step by step
f2(4): Tracing the Code
f2(4) is called, x = 0. The gray boxes are the inner-loop iterations to come: i² boxes in row i.
i = 0: the inner condition 0 < 0 · 0 is false at once; the inner loop runs zero times.
i = 1: the inner loop runs while j < 1·1 = 1. First iteration (j = 0): x = 1.
i = 2: the inner loop runs while j < 2·2 = 4. First iteration (j = 0): x = 2.
j = 3 is the last iteration: row i = 2 finished in 4 iterations, x = 5.
i = 3: the inner loop runs while j < 3·3 = 9. First iteration (j = 0): x = 6.
j = 8 is the last iteration: row i = 3 finished in 9 iterations, x = 14.
i = 4: 4 < 4 is false and the outer loop ends. Iterations: 0 + 1 + 4 + 9.
return 14 = 0² + 1² + 2² + 3² = Σ i². Since the rows grow as i², the total grows like n³/3: Θ(n³).
public int f2(int n) { int x = 0; for (int i = 0; i < n; i++) { for (int j = 0; j < i * i; j++) { x++; } } return x; }
Loop Analysis
f2(5): How Often Does the Inner Loop Run?
Each bar shows how many times the inner loop runs in one outer iteration (fixed i). The total area of the bars is the number of x++ operations f2 performs.
For i = 0 the inner loop runs 0 times (0 × 0 = 0).
For i = 1 the inner loop runs 1 time (1 × 1 = 1).
For i = 2 the inner loop runs 4 times (2 × 2 = 4).
For i = 3 the inner loop runs 9 times (3 × 3 = 9).
For i = 4 the inner loop runs 16 times. In general it runs i² times for each i.
Total: 0 + 1 + 4 + 9 + 16 = 30 = (n − 1) · n · (2n − 1) / 6 = 4 · 5 · 9 / 6. f2(5) returns 30.
Loop Analysis
f2: Evaluating the Sum
Total number of inner iterations (i = 0 to n − 1):
The dominant term is n³/3; dropping the constant 1/3 and the lower-order terms gives T(n) = Θ(n³).
| n | f2(n) | n³/3 |
|---|---|---|
| 4 | 14 | 21 |
| 5 | 30 | 42 |
| 16 | 1,240 | 1,365 |
| 32 | 10,416 | 10,923 |
Loop Analysis
Loop Patterns and Summation Formulas
| Pattern | Iterations | Result |
|---|---|---|
for (i = 0; i < n; i++) | Σ 1 = n | Θ(n) |
for (j = 0; j < n; j++) inside for i < n | n · n | Θ(n²) |
for (j = 0; j < i; j++) inside for i < n | Σ i = n(n − 1)/2 | Θ(n²) |
for (j = 0; j < i * i; j++) inside for i < n | Σ i² = (n − 1)n(2n − 1)/6 | Θ(n³) |
for (i = 1; i < n; i *= 2) | ⌈log₂ n⌉ | Θ(log n) |
for j < n inside for (i = n; i > 0; i /= 2) | n · (⌊log₂ n⌋ + 1) | Θ(n log n) |
for j < i inside for (i = 1; i < n; i *= 2) | 1 + 2 + 4 + … < 2n | Θ(n) |
for (i = 0; i < n; i = 1 << i) (f6) | about log* n + 1 | Θ(log* n)* |
Method: write down the inner iteration count for each outer iteration and add them up; the sum reduces to a known series. * Mathematical model; in Java n ≤ 65536.
Loop Analysis
f6: A Shifting Loop
public static int f6(int n) { int x = 0; // 1 << i equals 2^i (overflow ignored) for (int i = 0; i < n; i = 1 << i) { x++; } return x; }
- Initialization:
int i = 0;sets i to 0; constant time, O(1). - Condition:
i < nchecks on every iteration that i is less than n. - Update:
i = 1 << ishifts 1 left by i bits; the result is 2i, computed in constant time. - The update is not
i = 2 * ibuti = 2^i. i itself becomes the exponent, so i grows much faster.
Loop Analysis
f6(20): Tracing the Code
x is set to zero.
Start with i = 0; the condition 0 < 20 is true.
x++ → x = 1.
i = 1 << 0 = 1; the condition 1 < 20 is true.
x++ → x = 2.
i = 1 << 1 = 2; the condition 2 < 20 is true.
x++ → x = 3.
i = 1 << 2 = 4; the condition 4 < 20 is true.
x++ → x = 4.
i = 1 << 4 = 16; the condition 16 < 20 is true.
x++ → x = 5.
i = 1 << 16 = 65536; the condition 65536 < 20 is false → the loop ends.
return x → 5. The values of i are 0, 1, 2, 4, 16: only 5 iterations.
public static int f6(int n) { int x = 0; // 1 << i equals 2^i (overflow ignored) for (int i = 0; i < n; i = 1 << i) { x++; } return x; }
Loop Analysis
f6: How Many Iterations Really?
Values of i: 0 → 2⁰ = 1 → 2¹ = 2 → 2² = 4 → 2⁴ = 16 → 2¹⁶ = 65536. Each iteration turns i into an exponent (a tower of powers of 2).
A common line of reasoning: "after k steps 2ᵏ ≥ n, so k ≥ log₂ n" — true for an i = 2 * i loop. In f6, after k iterations i is a tower of height k − 1; the number of iterations is about log* n (the iterated logarithm).
O(log n) is correct as an upper bound but not tight: Θ(log* n).
| n | f6(n) | ⌈log₂ n⌉ |
|---|---|---|
| 4 | 3 | 2 |
| 16 | 4 | 4 |
| 32 | 5 | 5 |
| 1000 | 5 | 10 |
| 65536 | 5 | 16 |
int is taken mod 32: 1 << 65536 = 1. So for n > 65536, i cycles through 1, 2, 4, 16, 65536 and f6 never terminates. The "ignore overflow" assumption matters here.Analyzing Recursion
Recurrence Relations
The running time of a recursive function is written in terms of its own running time on smaller inputs:
Analyzing Recursion
f3: Two Calls on n − 1 → O(2ⁿ)
public long f3(int n) { if (n <= 1) { return 1; } return f3(n - 1) + f3(n - 1); }
- Base case: for n ≤ 1 it returns 1; constant time, O(1).
- Recursive case: for n > 1 it calls
f3(n − 1)twice and adds the results (constant work c).
Analyzing Recursion
f3(3): Tracing the Calls
f3(3) is called: 3 > 1, not the base case.
Line 5: the first f3(2) call. f3(3) waits on the stack.
f3(1): n ≤ 1 → base case, returns 1.
f3(2) makes its second call: again f3(1) = 1.
f3(2) returns 1 + 1 = 2. The left subtree is done.
f3(3) calls the second f3(2): the same work is done all over again.
f3(1) = 1.
f3(1) = 1.
The second f3(2) returns 2.
f3(3) returns 2 + 2 = 4. 7 = 2³ − 1 calls in total; the stack depth is at most 3.
public long f3(int n) { if (n <= 1) { return 1; } return f3(n - 1) + f3(n - 1); }
Analyzing Recursion
f3: Expanding the Recurrence
The recurrence itself: two subcalls, constant extra work.
Substitute the definition of T(n − 1).
Expanding once more reveals the pattern: coefficients 2, 4, 8 …
The general form after k steps. Ignoring the extra work c gives T(n) = 2ᵏ T(n − k); the growth class is the same.
The base case T(1) = c is reached when n − k = 1. Result: T(n) = Θ(2ⁿ).
Counted in Java: f3(16) makes 65,535 calls and f3(32) 4,294,967,295 (2ⁿ − 1).
Master Theorem
Master Theorem: General Form
Solves divide-and-conquer recurrences directly:
a: number of subproblems (calls) · b: shrink factor, subproblem size n/b · f(n): cost of dividing and combining. Depth logb n, number of leaves alogb n = nlogb a.
Master Theorem
Master Theorem: Three Cases
Compare f(n) with nlogb a, the total work at the leaves. Let c be the polynomial degree of f(n):
| Case | Condition | Result | Interpretation |
|---|---|---|---|
| 1 | f(n) = O(nc), c < logb a | T(n) = Θ(nlogb a) | leaves dominate |
| 2 | f(n) = Θ(nc), c = logb a | T(n) = Θ(nlogb a · log n) | levels balanced |
| 3 | f(n) = Ω(nc), c > logb a and a · f(n/b) ≤ k · f(n), k < 1 | T(n) = Θ(f(n)) | root dominates |
The regularity condition holds automatically for polynomials f(n) = nᶜ: a(n/b)ᶜ = (a/bᶜ) nᶜ and a/bᶜ < 1.
Master Theorem
Why Three Cases? Level Sums
If f(n) = nᶜ, level j of the tree has aʲ nodes, each doing (n/bʲ)ᶜ work: level sum = nᶜ · (a/bᶜ)ʲ. The total is a geometric series with ratio r = a/bᶜ.
Case 1: last level (leaves) dominates → Θ(n²)
Case 2: log n levels × n work → Θ(n log n)
Case 3: root dominates → Θ(n²)
Master Theorem
Example: T(n) = 2T(n/2) + Θ(n)
Read off the parameters: two subproblems, each of size n/2, linear combining work.
The leaf-work exponent is 1: nlog₂ 2 = n.
The degree of f(n) is c = 1.
Equal exponents: every level does the same (n) work.
Case 2: T(n) = Θ(nlogb a · log n) = Θ(n log n).
- Merge sort: split the array in half, sort both halves, merge in Θ(n).
- The chapter's
f5:f1(n)plus two calls tof5(n/2). - Quicksort with balanced partitions.
Master Theorem
Master Theorem: Examples
| Recurrence | a, b | logb a | c | Case | Result | Algorithm |
|---|---|---|---|---|---|---|
| T(n/2) + 1 | 1, 2 | 0 | 0 | 2 | Θ(log n) | binary search |
| 2T(n/2) + 1 | 2, 2 | 1 | 0 | 1 | Θ(n) | f4, tree traversal |
| 2T(n/2) + n | 2, 2 | 1 | 1 | 2 | Θ(n log n) | f5, merge sort |
| 3T(n/2) + n | 3, 2 | ≈ 1.585 | 1 | 1 | Θ(n1.585) | Karatsuba multiplication |
| 4T(n/2) + n | 4, 2 | 2 | 1 | 1 | Θ(n²) | — |
| 7T(n/2) + n² | 7, 2 | ≈ 2.807 | 2 | 1 | Θ(n2.807) | Strassen |
| 2T(n/2) + n² | 2, 2 | 1 | 2 | 3 | Θ(n²) | — |
| T(n/2) + n | 1, 2 | 0 | 1 | 3 | Θ(n) | quickselect (average) |
Regularity in the Case 3 rows: 2(n/2)² = n²/2 ≤ ½ n² and 1 · (n/2) ≤ ½ n hold with k = ½.
Master Theorem
When Does the Master Theorem Not Apply?
Application: f4 and f5
f4: Two Calls on n/2 → O(n)
public int f4(int n) { if (n <= 1) { return 1; } return f4(n / 2) + f4(n / 2); }
- Base case: for n ≤ 1 it returns 1; O(1).
- Recursive case: for n > 1 it calls
f4(n/2)twice and adds the results (constant work).
Application: f4 and f5
f4(8): Recursion Tree
Root: f4(8) does only constant work (c) and starts two calls.
Level 1: two f4(4) calls, 2c in total.
Level 2: four f4(2) calls, 4c in total. The work per level doubles at each step.
Leaves: eight f4(1), 8c = n · c in total. Overall c(1 + 2 + 4 + 8) = 15c = (2n − 1)c → Θ(n).
The level sums grow geometrically, so the last term (the leaves) dominates. In Java: f4(16) makes 31 calls, f4(32) 63 (2n − 1).
Application: f4 and f5
f4: Solving with the Master Theorem
Match it against the general form T(n) = aT(n/b) + f(n).
Here a = 2, b = 2, f(n) = O(1).
Leaf work: nlog₂ 2 = n¹ = n.
f(n) = O(1) corresponds to c = 0, which is less than logb a = 1.
By the Master Theorem, if f(n) = O(nᶜ) with c < logb a then T(n) = Θ(nlogb a). Result: Θ(n).
Note: calling f4 "O(log n)" is a common mistake; the right answer is Θ(n). log n is only the stack depth (memory).
Application: f4 and f5
f5: Linear Work + Two Calls → O(n log n)
public int f5(int n) { if (n <= 1) { return 1; } return f1(n) + f5(n / 2) + f5(n / 2); }
- Base case: for n ≤ 1 it returns 1; O(1).
- Recursive case: for n > 1 it first calls
f1(n)(Θ(n) work), thenf5(n/2)twice.
Application: f4 and f5 · Run the code step by step
f5(4): Tracing the Calls
f5(4) is called: 4 > 1, not the base case.
Line 5 first computes f1(4): a 4-iteration loop, 4 units of work. Then two f5(2) calls.
f5(2) is called: 2 > 1, not the base case.
Line 5 first computes f1(2): a 2-iteration loop, 2 units of work. Then two f5(1) calls.
f5(1): n ≤ 1 → base case, returns 1.
f5(1): n ≤ 1 → base case, returns 1.
f5(2) = f1(2) + f5(1) + f5(1) = 2 + 1 + 1 = 4 is returned.
The second f5(2) call repeats the same work as the first.
Line 5 first computes f1(2): a 2-iteration loop, 2 units of work. Then two f5(1) calls.
f5(1): n ≤ 1 → base case, returns 1.
f5(1): n ≤ 1 → base case, returns 1.
f5(2) = f1(2) + f5(1) + f5(1) = 2 + 1 + 1 = 4 is returned.
f5(4) = 4 + 4 + 4 = 12. Each level does n = 4 work: log₂ 4 = 2 levels · 4 + 4 leaves → n log₂ n + n. The recurrence is 2T(n/2) + n: Θ(n log n).
public int f5(int n) { if (n <= 1) { return 1; } return f1(n) + f5(n / 2) + f5(n / 2); }
Application: f4 and f5
f5(8): Recursion Tree
Root: inside f5(8), f1(8) takes 8 steps.
Level 1: two f5(4), each with f1(4) = 4 steps → 8 in total.
Level 2: four f5(2), 2 steps each → again 8. Every level does n work.
Leaves: 8 × 1. Total 3 · 8 + 8 = 32 = n log₂ n + n → Θ(n log n). Java: f5(8) = 32.
Each of the log₂ n levels does n work. In Java: f5(16) = 80 = 16 · 4 + 16, f5(32) = 192 = 32 · 5 + 32.
Application: f4 and f5
f5: Solving with the Master Theorem
Of the form T(n) = aT(n/b) + f(n).
Same parameters as f4; only f(n) has changed.
f(n) = Θ(n) → c = 1 = logb a.
Equal exponents: Θ(n log n). The same recurrence as the example earlier in the chapter.
| f4 | f5 | |
|---|---|---|
| Calls | 2 × n/2 | 2 × n/2 |
| Extra work f(n) | O(1) | Θ(n) |
| c vs logb a | 0 < 1 | 1 = 1 |
| Case | 1 | 2 |
| Result | Θ(n) | Θ(n log n) |
Comparison
f1–f6: Step Counts Measured in Java
| Function | Recurrence / sum | n = 16 | n = 32 | Ratio | Result |
|---|---|---|---|---|---|
f1 return | Σ 1 = n | 16 | 32 | × 2 | Θ(n) |
f2 return | Σ i² | 1,240 | 10,416 | × 8.4 | Θ(n³) |
f3 calls | 2T(n − 1) + c | 65,535 | 4,294,967,295 | × 65,537 | Θ(2ⁿ) |
f4 calls | 2T(n/2) + c | 31 | 63 | × 2 | Θ(n) |
f5 return | 2T(n/2) + n | 80 | 192 | × 2.4 | Θ(n log n) |
f6 return | i ← 2i | 4 | 5 | + 1 | Θ(log* n) (mathematical model; in Java n ≤ 65536) |
(long counter; f3(32) ≈ 4.3·10⁹ calls)
Comparison
Interactive f1–f6 Step Counter
| Function | Step count | Relative size (log scale) |
|---|
Steps: number of x++ for f1, f2, f6; number of calls for f3, f4; calls + f1 loop steps for f5. Values follow the logic of the code (BigInt for f3).
Comparison
A Quick Look: Amortized Analysis
- Amortized analysis divides the total cost of a sequence of operations by the number of operations.
- Dynamic array: when full, its capacity doubles and every element is copied (expensive); other insertions cost 1.
- Over n insertions the copies total 1 + 2 + 4 + … < 2n → total < 3n → amortized Θ(1) per insertion.
- Unlike the average case, there is no probability: the guarantee holds for every sequence of operations.
9 insertions starting from capacity 1; orange bars include a resize (copy).
Summary
Summary
| Topic | Key points |
|---|---|
| Complexity | time = number of basic operations T(n); space = extra memory (including the stack); best / worst / average case |
| Notation | O: upper, Ω: lower, Θ: tight bound; proved with c and n₀; drop constants and lower-order terms |
| Loops | add up iteration counts: f1 Σ1 = n → Θ(n); f2 Σi² → Θ(n³); f6 i ← 2i → Θ(log* n) (mathematical model; in Java n ≤ 65536; O(log n) is not tight) |
| Recursion | set up a recurrence: number of calls, how the input shrinks, extra work; solve by expansion or a recursion tree |
| f3 | 2T(n − 1) + c = (2ⁿ − 1)c → Θ(2ⁿ); the Master Theorem does not apply |
| Master Theorem | aT(n/b) + f(n): compare f(n) with nlogb a; leaves / balanced / root dominates |
| f4, f5 | 2T(n/2) + 1 → Case 1, Θ(n); 2T(n/2) + n → Case 2, Θ(n log n) |
Summary
Common Mistakes
i *= 2 loop runs only log n times.Summary
Sample Problems
- Show that 2n + 10 = O(n) by finding c and n₀.
- Solve T(n) = 3T(n/3) + n.
- Solve T(n) = 9T(n/3) + n.
- Solve T(n) = T(n/2) + n (check regularity).
- How many times does
for (i = 1; i < n; i *= 2) for (j = 0; j < i; j++) x++;executex++? - What is the complexity of
for (i = n; i > 0; i /= 2) for (j = 0; j < n; j++) x++;?
Show answers
- 2n + 10 ≤ 3n ⇔ n ≥ 10: c = 3, n₀ = 10 (or c = 12, n₀ = 1).
- a = 3, b = 3, log₃ 3 = 1 = c → Case 2 → Θ(n log n).
- log₃ 9 = 2 > c = 1 → Case 1 → Θ(n²).
- log₂ 1 = 0 < c = 1; 1 · (n/2) ≤ ½ · n (k = ½) → Case 3 → Θ(n).
- i = 1, 2, 4, …, 2ᵏ < n: total 1 + 2 + … + 2ᵏ = 2ᵏ⁺¹ − 1 < 2n → Θ(n) (n = 16: 15).
- The outer loop runs ⌊log₂ n⌋ + 1 times with n work each → Θ(n log n).
Summary
Exercises
- If the inner condition in f2 were
j < i, what would the complexity be? What would f2(5) return? - Write and solve the recurrence for
return g(n/2) + g(n/2) + g(n/2) + g(n/2);(base case n ≤ 1). - If the last line of f3 were
return 2 * f3(n - 1);, how would its return value and running time change? - Solve T(n) = 8T(n/2) + n² and T(n) = 7T(n/2) + n². Which algorithms do they describe?
- Prove by contradiction that n² ≠ O(n).
- If f6's loop were
for (i = 1; i < n; i *= 2), how many iterations would it make for n = 1000? Compare with f6(1000).
Answers / hints
- Σ i = n(n − 1)/2 → Θ(n²); f2(5) = 0 + 1 + 2 + 3 + 4 = 10.
- T(n) = 4T(n/2) + c; log₂ 4 = 2 > 0 → Case 1 → Θ(n²).
- Same return value (2ⁿ⁻¹), but a single call: T(n) = T(n − 1) + c → Θ(n).
- log₂ 8 = 3 > 2 → Θ(n³) (divide-and-conquer matrix multiplication); log₂ 7 ≈ 2.807 → Θ(n2.807) (Strassen).
- Suppose n² ≤ c·n for all n ≥ n₀ ⇒ n ≤ c. Choosing n = max(n₀, ⌈c⌉ + 1) gives a contradiction.
- i = 1, 2, 4, …, 512 → 10 iterations (⌈log₂ 1000⌉); f6(1000) makes only 5.
Summary
References
- Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., MIT Press, 2022 (Chapters 3 and 4).
- Sedgewick and Wayne, Algorithms, 4th ed., Addison-Wesley, 2011 (Section 1.4: Analysis of Algorithms).
- Kleinberg and Tardos, Algorithm Design, Pearson, 2005 (Chapters 2 and 5).
- D. E. Knuth, "Big Omicron and Big Omega and Big Theta", SIGACT News 8(2), 1976.
- J. L. Bentley, D. Haken and J. B. Saxe, "A General Method for Solving Divide-and-Conquer Recurrences", SIGACT News 12(3), 1980.