Chapter 2: Algorithm Complexity

Algorithms
Time and spaceBest / worst / averageBig O, Ω, ΘGrowth classes Loop analysisRecurrencesMaster TheoremExamples f1–f6
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Concepts

Distinguish time and space complexity, and best-, worst- and average-case analysis.

Notation

State the definitions of Big O, Ω and Θ; prove a bound by finding the constants c and n₀.

Loop analysis

Count the steps of single, nested and multiplicative loops with summation formulas (f1, f2, f6).

Recursion

Set up a recurrence for recursive code and solve it by expansion, a recursion tree or the Master Theorem (f3, f4, f5).

Throughout the chapter we work with six short functions (f1–f6), each representing a different growth class (n, n³, 2ⁿ, n, n log n, log* n).

The Notion of Complexity

Algorithm Complexity

  • Complexity theory measures an algorithm's use of resources: time and memory.
  • Resource use is written as a function of the input size n: T(n), S(n).
  • The goal is not to measure seconds but to see how the resource needs grow as the input grows.
Time complexityThe number of basic operations the algorithm performs: assignments, comparisons, arithmetic, array accesses.
Space complexityMemory used beyond the input: variables, auxiliary arrays, the recursion stack.
×1O(1)+1 stepO(log n)×2O(n)×4O(n²)×8O(n³)
The key question: what happens to the running time when the input size doubles? Does it stay the same, double, quadruple, or get squared?

The Notion of Complexity

Why Timing Is Not Enough

long start = System.currentTimeMillis();
linearSearch(arr, 10001);  // 10^8 elements
long end = System.currentTimeMillis();
System.out.println("Elapsed: "
    + (end - start) + " ms");
  • This code times a linear search over an array of 10⁸ elements.
  • The result depends on the machine, the processor, the JIT compiler, the cache and the current load.
  • A single measurement does not tell us what happens as the input grows.
machine A: 2nmachine B: 5nn²/4ntime
Experimental measurementGives real times but depends on the machine; the code must already be written.
Asymptotic analysisGives the step count as a function of n; machine-independent and possible before writing code.

The Notion of Complexity

Counting Steps: linearSearch

for (int i = 0; i < a.length; i++) {
  if (a[i] == target)
    return i;
}
return -1;
OperationWorst-case count
i = 01
i < a.lengthn + 1
a[i] == targetn
i++n
return -11
Total T(n)3n + 3
  • Each basic operation counts as 1 unit (the RAM model). Worst case: the target is not in the array.
  • The coefficient 3 and the +3 in 3n + 3 depend on the machine; what does not change is the linear growth in n → T(n) = Θ(n).

The Notion of Complexity · Run the code step by step

linearSearch: Tracing the Step Count

a407122itarget = 5i = 01i < a.length1a[i] == target0i++0return -10total steps2

a = [4, 7, 2], target = 5 (not in the array: worst case). i = 0 is assigned and 0 < 3 is checked: 2 steps.

a407122itarget = 5i = 01i < a.length1a[i] == target1i++0return -10total steps3

a[0] == target? 4 ≠ 5. One comparison so far.

a407122itarget = 5i = 01i < a.length2a[i] == target1i++1return -10total steps5

i++ → i = 1, then 1 < 3 is checked: the loop continues.

a407122itarget = 5i = 01i < a.length2a[i] == target2i++1return -10total steps6

a[1] == target? 7 ≠ 5. Two comparisons so far.

a407122itarget = 5i = 01i < a.length3a[i] == target2i++2return -10total steps8

i++ → i = 2, then 2 < 3 is checked: the loop continues.

a407122itarget = 5i = 01i < a.length3a[i] == target3i++2return -10total steps9

a[2] == target? 2 ≠ 5. Three comparisons so far.

a407122i = 3target = 5i = 01i < a.length4a[i] == target3i++3return -10total steps11

i++ → i = 3. 3 < 3 is false: the condition was checked n + 1 = 4 times and the loop ends.

a407122i = 3target = 5i = 01i < a.length4a[i] == target3i++3return -11total steps12

return -1. 12 steps in total = 3n + 3 (n = 3). A constant number (3) of operations per element: Θ(n).

for (int i = 0; i < a.length; i++) {
  if (a[i] == target)
    return i;
}
return -1;

The Notion of Complexity

Best, Worst and Average Case

Best caseThe fewest steps, on the most favorable input. A lower bound; on its own it says little.
Worst caseA guarantee that holds for every input: the algorithm never takes longer.
Average caseThe expected number of steps under a probability distribution over inputs.
703192438415target 7 → 1 comparison

Best case: the target is the first element; one comparison suffices → Θ(1).

703192438415target 5 → 6 comparisons, not found

Worst case: the target is absent (or last); n = 6 comparisons → Θ(n).

703192438415target 4 → 4 comparisons

Average case: if the target is equally likely to be in any position, the average is (1 + 2 + … + n)/n = (n + 1)/2 = 3.5 comparisons → still Θ(n).

The Notion of Complexity

Space Complexity

Space complexity usually refers to auxiliary space: the input itself is not counted. In recursion every active call keeps a stack frame.

CodeExtra spaceSpaceTime
f1: single looponly x and iΘ(1)Θ(n)
creating new int[n]auxiliary array of n elementsΘ(n)Θ(n)
f3: f3(n−1) + f3(n−1)stack depth n (at most n frames at once)Θ(n)Θ(2ⁿ)
f4: f4(n/2) + f4(n/2)stack depth ≈ log₂ nΘ(log n)Θ(n)
Merge sortauxiliary array for mergingΘ(n)Θ(n log n)
Time and space can grow differently. f3 makes an exponential number of calls, but since the calls finish one after another, at most n frames are on the stack at any time.

Asymptotic Notation

Big-O Notation

f(n) = O(g(n)) ⇔ ∃ c > 0, n₀ :
0 ≤ f(n) ≤ c · g(n) for all n ≥ n₀
  • For large n, f never exceeds a constant multiple of g: g is an upper bound.
  • Small n (n < n₀) do not matter; constant factors are absorbed into c.
  • Example: 3n² + 5n + 2 ≤ 4n² for all n ≥ 6 → O(n²).
02468100100200300400nn₀ = 6f(n)4n²
"Big O means the worst case" is not quite right. Big O is a kind of upper bound; it can be applied to the worst, best or average case alike. The two get confused because it is usually quoted for worst-case time.

Asymptotic Notation

Big Omega (Ω) and Big Theta (Θ)

NotationMeaningDefinition (∃ c, n₀ > 0, for all n ≥ n₀)Analogy
f = O(g)upper boundf(n) ≤ c · g(n)f ≤ g
f = Ω(g)lower boundf(n) ≥ c · g(n)f ≥ g
f = Θ(g)tight boundc₁ · g(n) ≤ f(n) ≤ c₂ · g(n)f = g
  • f = Θ(g) ⇔ f = O(g) and f = Ω(g).
  • 3n² + 5n + 2 is both O(n²) and O(n³), but it is Θ only of n².
  • Linear search: worst case Θ(n), best case Θ(1); O(n) and Ω(1) over all inputs.
  • The case (which input) and the bound (O, Ω, Θ) are separate ideas. Insertion sort is Θ(n²) in the worst case and Θ(n) in the best, so no single Θ covers all inputs. Merge sort, by contrast, is Θ(n log n) on every input.
02468100100200300400nn₀ = 6f(n)4n²3n²

Asymptotic Notation

Example: 3n² + 5n + 2 = Θ(n²)

Upper bound: 3n² + 5n + 2 ≤ c₂ n²

Try c₂ = 4: 3n² + 5n + 2 ≤ 4n² ⇔ n² − 5n − 2 ≥ 0 ⇔ n ≥ 5.37. So n₀ = 6.

Check: n = 5 → 102 > 100 ✗
n = 6 → 140 ≤ 144 ✓

The inequality fails at n = 5 and holds from n = 6 on (the left side grows by 6n + 8, the right side by 8n + 4).

Lower bound: 3n² + 5n + 2 ≥ 3n²

With c₁ = 3 the inequality holds for every n ≥ 1, since 5n + 2 > 0.

3n² ≤ f(n) ≤ 4n², n ≥ 6

c₁ = 3, c₂ = 4, n₀ = 6 satisfy the definition: f(n) = Θ(n²). The constants are not unique; c₂ = 10, n₀ = 1 also works.

02468100100200300400nn₀ = 6f(n)4n²3n²

Asymptotic Notation

Rules of Asymptotic Arithmetic

Drop constant factorsc · f(n) = Θ(f(n)): 5n = Θ(n), n²/3 = Θ(n²).
Drop lower-order termsA polynomial keeps its highest degree: 4n³ + 100n² + 7 = Θ(n³).
Sum rule (sequential code)O(f) + O(g) = O(max(f, g)): a Θ(n) loop + a Θ(n²) loop = Θ(n²).
Product rule (nested code)O(g) work inside a loop that runs n times: O(n · g). Check whether the inner work is the same on every iteration.
The log base does not matterloga n = logb n / logb a, a constant factor: O(log₂ n) = O(log₁₀ n) = O(log n).
The exponent base does matter3ⁿ = O(2ⁿ) is false, since 3ⁿ/2ⁿ = 1.5ⁿ → ∞. Unlike logarithms, changing the base of an exponential is not a constant factor.
Order of growth: 1 < log n < √n < n < n log n < n² < n³ < 2ⁿ < n!

Asymptotic Notation

Common Growth Classes

ClassNameTypical exampleWhen n doublesIn this chapter
O(1)Constantprinting the first element, array accessunchanged—
O(log n)Logarithmicbinary search, fast exponentiation, Euclid+1 stepf6 (upper bound)
O(√n)Square rootprimality test (trial division)≈ 1,41×—
O(n)Linearlinear search, maximum, array sum2×f1, f4
O(n log n)Linearithmicmerge sort, heapsort, quicksort (average)a bit more than 2×f5
O(n²)Quadraticbubble, selection, insertion sort4×—
O(n³)Cubicstandard matrix multiplication8×f2
O(2ⁿ)Exponentialall subsets, naive recursive Fibonaccisquaredf3
O(n!)Factorialall permutationsblows up even faster—

The classes get slower from top to bottom. Polynomial classes (nk) are usually considered "efficient", exponential ones "inefficient". The true class of f6 is log* n, which grows even more slowly than log n.

Asymptotic Notation

Growth Curves

051015200255075100nstepslog₂ nnn log₂ nn²2ⁿ
  • For small n the curves are close; at n = 4, n² and 2ⁿ are equal (16).
  • 2ⁿ passes 100 steps at n ≈ 6.6, n² after n = 10.
  • n log n stays close to linear; log n is almost flat.
  • The extremes drift further apart: at n = 10, n! reaches 3.6 million, while √n is only 10 at n = 100.
  • Asymptotic analysis cares about the order of the curves for large n.

Asymptotic Notation

Growth in Numbers: Calculator

f(n)operationstime at 10⁹ ops/srelative

With "n × 2", the n column doubles, n² grows 4×, n³ 8×; log₂ n grows by only 1, while 2ⁿ is squared. A faster computer buys a constant factor; a better algorithm changes the growth class.

Loop Analysis

f1: Single Loop → O(n)

public int f1(int n) {
  int x = 0;
  for (int i = 0; i < n; i++) {
    x++;
  }
  return x;
}
  • Initialization: int x = 0; sets x to 0; constant time, O(1).
  • Loop: for (int i = 0; i < n; i++) runs i from 0 to n − 1: n times.
  • Increment: x++; runs once per iteration; O(1).
T(n) = O(1) + n · O(1) = O(n)
The loop runs n times and its body takes constant time, so the total time complexity is Θ(n). The function returns x = n, so the return value is the step count itself: f1(16) = 16, f1(32) = 32.

Loop Analysis

f1(4): Tracing the Code

i=0i=1i=2i=3x++ count: 0

x is set to zero. The loop has not started yet.

i=0i=1i=2i=3x++ count: 0

i = 0: condition 0 < 4 is true → enter the body.

0i=0i=1i=2i=3x++ count: 1

x++ → x = 1. This is iteration 1.

0i=0i=1i=2i=3x++ count: 1

i = 1: condition 1 < 4 is true → enter the body.

0i=01i=1i=2i=3x++ count: 2

x++ → x = 2. This is iteration 2.

0i=01i=1i=2i=3x++ count: 2

i = 2: condition 2 < 4 is true → enter the body.

0i=01i=12i=2i=3x++ count: 3

x++ → x = 3. This is iteration 3.

0i=01i=12i=2i=3x++ count: 3

i = 3: condition 3 < 4 is true → enter the body.

0i=01i=12i=23i=3x++ count: 4

x++ → x = 4. This is iteration 4.

0i=01i=12i=23i=3x++ count: 4

i = 4: condition 4 < 4 is false → the loop ends. The condition was checked n + 1 = 5 times.

0i=01i=12i=23i=3x++ count: 4

return x → 4. The body ran exactly n = 4 times: Θ(n).

public int f1(int n) {
  int x = 0;
  for (int i = 0; i < n; i++) {
    x++;
  }
  return x;
}

Loop Analysis

f2: Nested Loop → O(n³)

public int f2(int n) {
  int x = 0;
  for (int i = 0; i < n; i++) {
    for (int j = 0; j < i * i; j++) {
      x++;
    }
  }
  return x;
}
  • Initialization: int x = 0; constant time, O(1).
  • Outer loop: i from 0 to n − 1: n times.
  • Inner loop: j from 0 to i · i − 1: i² times.
  • The number of inner iterations depends on the current value of i, so we cannot just say "n · n"; we have to add up the iteration counts.
Pitfall: seeing two nested loops and automatically saying O(n²) is wrong. Here the inner bound is i², so the result is of order n³.

Loop Analysis · Run the code step by step

f2(4): Tracing the Code

i=00 itersi=11 iteri=24 itersi=39 itersx = 0

f2(4) is called, x = 0. The gray boxes are the inner-loop iterations to come: i² boxes in row i.

i=00 itersi=11 iteri=24 itersi=39 itersx = 0

i = 0: the inner condition 0 < 0 · 0 is false at once; the inner loop runs zero times.

i=00 itersi=11 iteri=24 itersi=39 itersx = 1

i = 1: the inner loop runs while j < 1·1 = 1. First iteration (j = 0): x = 1.

i=00 itersi=11 iteri=24 itersi=39 itersx = 2

i = 2: the inner loop runs while j < 2·2 = 4. First iteration (j = 0): x = 2.

i=00 itersi=11 iteri=24 itersi=39 itersx = 5

j = 3 is the last iteration: row i = 2 finished in 4 iterations, x = 5.

i=00 itersi=11 iteri=24 itersi=39 itersx = 6

i = 3: the inner loop runs while j < 3·3 = 9. First iteration (j = 0): x = 6.

i=00 itersi=11 iteri=24 itersi=39 itersx = 14

j = 8 is the last iteration: row i = 3 finished in 9 iterations, x = 14.

i=00 itersi=11 iteri=24 itersi=39 itersx = 14

i = 4: 4 < 4 is false and the outer loop ends. Iterations: 0 + 1 + 4 + 9.

i=00 itersi=11 iteri=24 itersi=39 itersx = 14

return 14 = 0² + 1² + 2² + 3² = Σ i². Since the rows grow as i², the total grows like n³/3: Θ(n³).

public int f2(int n) {
  int x = 0;
  for (int i = 0; i < n; i++) {
    for (int j = 0; j < i * i; j++) {
      x++;
    }
  }
  return x;
}

Loop Analysis

f2(5): How Often Does the Inner Loop Run?

Each bar shows how many times the inner loop runs in one outer iteration (fixed i). The total area of the bars is the number of x++ operations f2 performs.

0² = 0i=0i=1i=2i=3i=4total x++ = 0

For i = 0 the inner loop runs 0 times (0 × 0 = 0).

0² = 0i=01² = 1i=1i=2i=3i=4total x++ = 1

For i = 1 the inner loop runs 1 time (1 × 1 = 1).

0² = 0i=01² = 1i=12² = 4i=2i=3i=4total x++ = 5

For i = 2 the inner loop runs 4 times (2 × 2 = 4).

0² = 0i=01² = 1i=12² = 4i=23² = 9i=3i=4total x++ = 14

For i = 3 the inner loop runs 9 times (3 × 3 = 9).

0² = 0i=01² = 1i=12² = 4i=23² = 9i=34² = 16i=4total x++ = 30

For i = 4 the inner loop runs 16 times. In general it runs i² times for each i.

0² = 0i=01² = 1i=12² = 4i=23² = 9i=34² = 16i=4total x++ = 30

Total: 0 + 1 + 4 + 9 + 16 = 30 = (n − 1) · n · (2n − 1) / 6 = 4 · 5 · 9 / 6. f2(5) returns 30.

Loop Analysis

f2: Evaluating the Sum

Total number of inner iterations (i = 0 to n − 1):

Σi=0n−1 i² = (n − 1) · n · (2n − 1) / 6
= (2n³ − 3n² + n) / 6 ≈ n³ / 3

The dominant term is n³/3; dropping the constant 1/3 and the lower-order terms gives T(n) = Θ(n³).

nf2(n)n³/3
41421
53042
161,2401,365
3210,41610,923
Check: when n doubles (16 → 32), the step count grows by 10,416 / 1,240 ≈ 8.4, close to 2³ = 8. That is the signature of cubic growth.

Loop Analysis

Loop Patterns and Summation Formulas

PatternIterationsResult
for (i = 0; i < n; i++)Σ 1 = nΘ(n)
for (j = 0; j < n; j++) inside for i < nn · nΘ(n²)
for (j = 0; j < i; j++) inside for i < nΣ i = n(n − 1)/2Θ(n²)
for (j = 0; j < i * i; j++) inside for i < nΣ i² = (n − 1)n(2n − 1)/6Θ(n³)
for (i = 1; i < n; i *= 2)⌈log₂ n⌉Θ(log n)
for j < n inside for (i = n; i > 0; i /= 2)n · (⌊log₂ n⌋ + 1)Θ(n log n)
for j < i inside for (i = 1; i < n; i *= 2)1 + 2 + 4 + … < 2nΘ(n)
for (i = 0; i < n; i = 1 << i) (f6)about log* n + 1Θ(log* n)*

Method: write down the inner iteration count for each outer iteration and add them up; the sum reduces to a known series. * Mathematical model; in Java n ≤ 65536.

Loop Analysis

f6: A Shifting Loop

public static int f6(int n) {
  int x = 0;
  // 1 << i equals 2^i (overflow ignored)
  for (int i = 0; i < n; i = 1 << i) {
    x++;
  }
  return x;
}
  • Initialization: int i = 0; sets i to 0; constant time, O(1).
  • Condition: i < n checks on every iteration that i is less than n.
  • Update: i = 1 << i shifts 1 left by i bits; the result is 2i, computed in constant time.
  • The update is not i = 2 * i but i = 2^i. i itself becomes the exponent, so i grows much faster.

Loop Analysis

f6(20): Tracing the Code

??????values of i (n = 20); i ← 1 << i

x is set to zero.

0?????values of i (n = 20); i ← 1 << icondition: 0 < 20 → true

Start with i = 0; the condition 0 < 20 is true.

0?????values of i (n = 20); i ← 1 << icondition: 0 < 20 → true

x++ → x = 1.

01????values of i (n = 20); i ← 1 << icondition: 1 < 20 → true

i = 1 << 0 = 1; the condition 1 < 20 is true.

01????values of i (n = 20); i ← 1 << icondition: 1 < 20 → true

x++ → x = 2.

012???values of i (n = 20); i ← 1 << icondition: 2 < 20 → true

i = 1 << 1 = 2; the condition 2 < 20 is true.

012???values of i (n = 20); i ← 1 << icondition: 2 < 20 → true

x++ → x = 3.

0124??values of i (n = 20); i ← 1 << icondition: 4 < 20 → true

i = 1 << 2 = 4; the condition 4 < 20 is true.

0124??values of i (n = 20); i ← 1 << icondition: 4 < 20 → true

x++ → x = 4.

012416?values of i (n = 20); i ← 1 << icondition: 16 < 20 → true

i = 1 << 4 = 16; the condition 16 < 20 is true.

012416?values of i (n = 20); i ← 1 << icondition: 16 < 20 → true

x++ → x = 5.

01241665536values of i (n = 20); i ← 1 << icondition: 65536 < 20 → false, loop ends

i = 1 << 16 = 65536; the condition 65536 < 20 is false → the loop ends.

01241665536values of i (n = 20); i ← 1 << icondition: 65536 < 20 → false, loop ends

return x → 5. The values of i are 0, 1, 2, 4, 16: only 5 iterations.

public static int f6(int n) {
  int x = 0;
  // 1 << i equals 2^i (overflow ignored)
  for (int i = 0; i < n; i = 1 << i) {
    x++;
  }
  return x;
}

Loop Analysis

f6: How Many Iterations Really?

Values of i: 0 → 2⁰ = 1 → 2¹ = 2 → 2² = 4 → 2⁴ = 16 → 2¹⁶ = 65536. Each iteration turns i into an exponent (a tower of powers of 2).

A common line of reasoning: "after k steps 2ᵏ ≥ n, so k ≥ log₂ n" — true for an i = 2 * i loop. In f6, after k iterations i is a tower of height k − 1; the number of iterations is about log* n (the iterated logarithm).

O(log n) is correct as an upper bound but not tight: Θ(log* n).

nf6(n)⌈log₂ n⌉
432
1644
3255
1000510
65536516
Java pitfall: the shift distance for an int is taken mod 32: 1 << 65536 = 1. So for n > 65536, i cycles through 1, 2, 4, 16, 65536 and f6 never terminates. The "ignore overflow" assumption matters here.

Analyzing Recursion

Recurrence Relations

The running time of a recursive function is written in terms of its own running time on smaller inputs:

T(n) = (time of the recursive calls) + (work outside the calls),   T(base) = Θ(1)
Expansion (substitution)Plug the recurrence into itself repeatedly to find a pattern; sum up once the base case is reached. (f3)
Recursion treeDraw each call as a node and add up the work on each level. (f3, f4, f5)
Master TheoremSolve divide-and-conquer recurrences of the form T(n) = aT(n/b) + f(n) directly. (f4, f5)
T(n)+ f(n): work outside callsT(n/b)T(n/b)T(n/b)…a calls, each of size n/b (or n − 1)
Questions to ask when setting up a recurrence: How many recursive calls are there? How much smaller is each call's input (n − 1 or n/2)? How much work is done outside the calls?

Analyzing Recursion

f3: Two Calls on n − 1 → O(2ⁿ)

public long f3(int n) {
  if (n <= 1) {
    return 1;
  }
  return f3(n - 1) + f3(n - 1);
}
  • Base case: for n ≤ 1 it returns 1; constant time, O(1).
  • Recursive case: for n > 1 it calls f3(n − 1) twice and adds the results (constant work c).
T(n) = 2T(n − 1) + c,   n > 1     T(1) = c
f3(4)1 callf3(3)2 callsf3(2)4 callsf3(1)8 callsinput shrinks by 1, calls double: 1 + 2 + 4 + 8 = 15 = 2⁴ − 1
The input shrinks by only 1 per call, but the number of calls doubles at every level. This recurrence is not of the form T(n/b), so the Master Theorem does not apply; we solve it by expansion.

Analyzing Recursion

f3(3): Tracing the Calls

f3(3)f3(2)f3(2)f3(1)f3(1)f3(1)f3(1)

f3(3) is called: 3 > 1, not the base case.

f3(3)f3(2)f3(2)f3(1)f3(1)f3(1)f3(1)

Line 5: the first f3(2) call. f3(3) waits on the stack.

f3(3)f3(2)f3(2)f3(1)= 1f3(1)f3(1)f3(1)

f3(1): n ≤ 1 → base case, returns 1.

f3(3)f3(2)f3(2)f3(1)= 1f3(1)= 1f3(1)f3(1)

f3(2) makes its second call: again f3(1) = 1.

f3(3)f3(2)= 2f3(2)f3(1)= 1f3(1)= 1f3(1)f3(1)

f3(2) returns 1 + 1 = 2. The left subtree is done.

f3(3)f3(2)= 2f3(2)f3(1)= 1f3(1)= 1f3(1)f3(1)

f3(3) calls the second f3(2): the same work is done all over again.

f3(3)f3(2)= 2f3(2)f3(1)= 1f3(1)= 1f3(1)= 1f3(1)

f3(1) = 1.

f3(3)f3(2)= 2f3(2)f3(1)= 1f3(1)= 1f3(1)= 1f3(1)= 1

f3(1) = 1.

f3(3)f3(2)= 2f3(2)= 2f3(1)= 1f3(1)= 1f3(1)= 1f3(1)= 1

The second f3(2) returns 2.

f3(3)= 4f3(2)= 2f3(2)= 2f3(1)= 1f3(1)= 1f3(1)= 1f3(1)= 1

f3(3) returns 2 + 2 = 4. 7 = 2³ − 1 calls in total; the stack depth is at most 3.

public long f3(int n) {
  if (n <= 1) {
    return 1;
  }
  return f3(n - 1) + f3(n - 1);
}

Analyzing Recursion

f3: Expanding the Recurrence

T(n) = 2T(n − 1) + c

The recurrence itself: two subcalls, constant extra work.

= 2[2T(n − 2) + c] + c = 4T(n − 2) + 3c

Substitute the definition of T(n − 1).

= 8T(n − 3) + 7c

Expanding once more reveals the pattern: coefficients 2, 4, 8 …

= 2ᵏ T(n − k) + (2ᵏ − 1)c

The general form after k steps. Ignoring the extra work c gives T(n) = 2ᵏ T(n − k); the growth class is the same.

k = n − 1: T(n) = 2ⁿ⁻¹c + (2ⁿ⁻¹ − 1)c = (2ⁿ − 1)c

The base case T(1) = c is reached when n − k = 1. Result: T(n) = Θ(2ⁿ).

4332222111111111 · c2 · c4 · c8 · c

Counted in Java: f3(16) makes 65,535 calls and f3(32) 4,294,967,295 (2ⁿ − 1).

Master Theorem

Master Theorem: General Form

Solves divide-and-conquer recurrences directly:

T(n) = a · T(n/b) + f(n),   a ≥ 1, b > 1

a: number of subproblems (calls) · b: shrink factor, subproblem size n/b · f(n): cost of dividing and combining. Depth logb n, number of leaves alogb n = nlogb a.

f(n)f(n/b)f(n/b)…f(n/b)a children· ·· ·· ·⋮⋮Θ(1)Θ(1)Θ(1)Θ(1)Θ(1)Θ(1)Θ(1)Θ(1)…f(n)a · f(n/b)a² · f(n/b²)nlog b a · Θ(1)depth: logb n

Master Theorem

Master Theorem: Three Cases

Compare f(n) with nlogb a, the total work at the leaves. Let c be the polynomial degree of f(n):

CaseConditionResultInterpretation
1f(n) = O(nc), c < logb aT(n) = Θ(nlogb a)leaves dominate
2f(n) = Θ(nc), c = logb aT(n) = Θ(nlogb a · log n)levels balanced
3f(n) = Ω(nc), c > logb a and a · f(n/b) ≤ k · f(n), k < 1T(n) = Θ(f(n))root dominates
In Cases 2 and 3, writing f(n) = O(nᶜ) is not enough. For instance f(n) = 1 is also O(n²), yet T(n) = T(n/2) + 1 cannot be Θ(n²). So Case 2 requires f = Θ(nᶜ), and Case 3 requires f = Ω(nᶜ) plus the regularity condition; the results are tight (Θ).

The regularity condition holds automatically for polynomials f(n) = nᶜ: a(n/b)ᶜ = (a/bᶜ) nᶜ and a/bᶜ < 1.

Master Theorem

Why Three Cases? Level Sums

If f(n) = nᶜ, level j of the tree has aʲ nodes, each doing (n/bʲ)ᶜ work: level sum = nᶜ · (a/bᶜ)ʲ. The total is a geometric series with ratio r = a/bᶜ.

4T(n/2) + nj=0j=1j=2j=3r = 2 > 1: grows

Case 1: last level (leaves) dominates → Θ(n²)

2T(n/2) + nj=0j=1j=2j=3r = 1: constant

Case 2: log n levels × n work → Θ(n log n)

2T(n/2) + n²j=0j=1j=2j=3r = 1/2 < 1: shrinks

Case 3: root dominates → Θ(n²)

r > 1 ⇔ bᶜ < a ⇔ c < logb a. So the three cases of the Master Theorem correspond to a geometric series that is increasing, constant or decreasing.

Master Theorem

Example: T(n) = 2T(n/2) + Θ(n)

a = 2,   b = 2,   f(n) = Θ(n)

Read off the parameters: two subproblems, each of size n/2, linear combining work.

logb a = log₂ 2 = 1

The leaf-work exponent is 1: nlog₂ 2 = n.

f(n) = Θ(n¹) → c = 1

The degree of f(n) is c = 1.

c = logb a → Case 2

Equal exponents: every level does the same (n) work.

T(n) = Θ(n log n)

Case 2: T(n) = Θ(nlogb a · log n) = Θ(n log n).

Where does this recurrence arise?
  • Merge sort: split the array in half, sort both halves, merge in Θ(n).
  • The chapter's f5: f1(n) plus two calls to f5(n/2).
  • Quicksort with balanced partitions.

Master Theorem

Master Theorem: Examples

Recurrencea, blogb acCaseResultAlgorithm
T(n/2) + 11, 2002Θ(log n)binary search
2T(n/2) + 12, 2101Θ(n)f4, tree traversal
2T(n/2) + n2, 2112Θ(n log n)f5, merge sort
3T(n/2) + n3, 2≈ 1.58511Θ(n1.585)Karatsuba multiplication
4T(n/2) + n4, 2211Θ(n²)—
7T(n/2) + n²7, 2≈ 2.80721Θ(n2.807)Strassen
2T(n/2) + n²2, 2123Θ(n²)—
T(n/2) + n1, 2013Θ(n)quickselect (average)

Regularity in the Case 3 rows: 2(n/2)² = n²/2 ≤ ½ n² and 1 · (n/2) ≤ ½ n hold with k = ½.

Master Theorem

When Does the Master Theorem Not Apply?

Subtractive recurrencesT(n) = 2T(n − 1) + c (f3) or T(n) = T(n − 1) + n: the subproblem is n − 1, not n/b. Use expansion.
Non-polynomial gapT(n) = 2T(n/2) + n / log n: f(n) is smaller than n by only a log factor; no case fits.
Unequal splitsT(n) = T(n/3) + T(2n/3) + n: subproblems of different sizes. A recursion tree (or Akra–Bazzi) gives Θ(n log n).
Non-constant a, or a < 1T(n) = 2ⁿ T(n/2) + n or T(n) = ½ T(n/2) + n: the theorem's assumption that a ≥ 1 is a constant breaks.
An extension of the basic form: if f(n) = Θ(nlogb a · logᵏ n), then T(n) = Θ(nlogb a · logᵏ⁺¹ n). For example T(n) = 2T(n/2) + n log n = Θ(n log² n).

Application: f4 and f5

f4: Two Calls on n/2 → O(n)

public int f4(int n) {
  if (n <= 1) {
    return 1;
  }
  return f4(n / 2) + f4(n / 2);
}
  • Base case: for n ≤ 1 it returns 1; O(1).
  • Recursive case: for n > 1 it calls f4(n/2) twice and adds the results (constant work).
T(n) = 2T(n/2) + O(1),   T(1) = O(1)
n = 81 calln = 42 callsn = 24 callsn = 18 callsinput halves: only log₂ 8 + 1 = 4 levels, 1 + 2 + 4 + 8 = 15 = 2n − 1 calls
The only difference from f3 is that the input is n/2 instead of n − 1, yet that brings the result down from Θ(2ⁿ) to Θ(n). Two calls do not always mean exponential time: how fast the input shrinks decides.

Application: f4 and f5

f4(8): Recursion Tree

8442222111111111 · c

Root: f4(8) does only constant work (c) and starts two calls.

8442222111111111 · c2 · c

Level 1: two f4(4) calls, 2c in total.

8442222111111111 · c2 · c4 · c

Level 2: four f4(2) calls, 4c in total. The work per level doubles at each step.

8442222111111111 · c2 · c4 · c8 · c

Leaves: eight f4(1), 8c = n · c in total. Overall c(1 + 2 + 4 + 8) = 15c = (2n − 1)c → Θ(n).

The level sums grow geometrically, so the last term (the leaves) dominates. In Java: f4(16) makes 31 calls, f4(32) 63 (2n − 1).

Application: f4 and f5

f4: Solving with the Master Theorem

T(n) = 2T(n/2) + O(1)

Match it against the general form T(n) = aT(n/b) + f(n).

a = 2, b = 2, f(n) = O(1)

Here a = 2, b = 2, f(n) = O(1).

logb a = log₂ 2 = 1

Leaf work: nlog₂ 2 = n¹ = n.

f(n) = O(n⁰) → c = 0 < 1

f(n) = O(1) corresponds to c = 0, which is less than logb a = 1.

Case 1: T(n) = Θ(nlog₂ 2) = Θ(n)

By the Master Theorem, if f(n) = O(nᶜ) with c < logb a then T(n) = Θ(nlogb a). Result: Θ(n).

Same result as the treeThe recursion tree had 2n − 1 nodes. The Master Theorem gives the same answer without computing the sum: the leaves dominate.

Note: calling f4 "O(log n)" is a common mistake; the right answer is Θ(n). log n is only the stack depth (memory).

Application: f4 and f5

f5: Linear Work + Two Calls → O(n log n)

public int f5(int n) {
  if (n <= 1) {
    return 1;
  }
  return f1(n) + f5(n / 2) + f5(n / 2);
}
  • Base case: for n ≤ 1 it returns 1; O(1).
  • Recursive case: for n > 1 it first calls f1(n) (Θ(n) work), then f5(n/2) twice.
T(n) = 2T(n/2) + f1(n) = 2T(n/2) + Θ(n)
The only difference from f4 is the linear work in each call. This is exactly the merge sort recurrence. The answer is not "O(n + log n)" but Θ(n log n).

Application: f4 and f5 · Run the code step by step

f5(4): Tracing the Calls

f5(4)f5(2)f5(2)f5(1)f5(1)f5(1)f5(1)level work

f5(4) is called: 4 > 1, not the base case.

f5(4)f1: 4f5(2)f5(2)f5(1)f5(1)f5(1)f5(1)4level work

Line 5 first computes f1(4): a 4-iteration loop, 4 units of work. Then two f5(2) calls.

f5(4)f1: 4f5(2)f5(2)f5(1)f5(1)f5(1)f5(1)4level work

f5(2) is called: 2 > 1, not the base case.

f5(4)f1: 4f5(2)f1: 2f5(2)f5(1)f5(1)f5(1)f5(1)42level work

Line 5 first computes f1(2): a 2-iteration loop, 2 units of work. Then two f5(1) calls.

f5(4)f1: 4f5(2)f1: 2f5(2)f5(1)= 1f5(1)f5(1)f5(1)421level work

f5(1): n ≤ 1 → base case, returns 1.

f5(4)f1: 4f5(2)f1: 2f5(2)f5(1)= 1f5(1)= 1f5(1)f5(1)421 + 1level work

f5(1): n ≤ 1 → base case, returns 1.

f5(4)f1: 4f5(2)f1: 2 = 4f5(2)f5(1)= 1f5(1)= 1f5(1)f5(1)421 + 1level work

f5(2) = f1(2) + f5(1) + f5(1) = 2 + 1 + 1 = 4 is returned.

f5(4)f1: 4f5(2)f1: 2 = 4f5(2)f5(1)= 1f5(1)= 1f5(1)f5(1)421 + 1level work

The second f5(2) call repeats the same work as the first.

f5(4)f1: 4f5(2)f1: 2 = 4f5(2)f1: 2f5(1)= 1f5(1)= 1f5(1)f5(1)42 + 21 + 1level work

Line 5 first computes f1(2): a 2-iteration loop, 2 units of work. Then two f5(1) calls.

f5(4)f1: 4f5(2)f1: 2 = 4f5(2)f1: 2f5(1)= 1f5(1)= 1f5(1)= 1f5(1)42 + 21 + 1 + 1level work

f5(1): n ≤ 1 → base case, returns 1.

f5(4)f1: 4f5(2)f1: 2 = 4f5(2)f1: 2f5(1)= 1f5(1)= 1f5(1)= 1f5(1)= 142 + 21 + 1 + 1 + 1level work

f5(1): n ≤ 1 → base case, returns 1.

f5(4)f1: 4f5(2)f1: 2 = 4f5(2)f1: 2 = 4f5(1)= 1f5(1)= 1f5(1)= 1f5(1)= 142 + 21 + 1 + 1 + 1level work

f5(2) = f1(2) + f5(1) + f5(1) = 2 + 1 + 1 = 4 is returned.

f5(4)f1: 4 = 12f5(2)f1: 2 = 4f5(2)f1: 2 = 4f5(1)= 1f5(1)= 1f5(1)= 1f5(1)= 142 + 21 + 1 + 1 + 1level work

f5(4) = 4 + 4 + 4 = 12. Each level does n = 4 work: log₂ 4 = 2 levels · 4 + 4 leaves → n log₂ n + n. The recurrence is 2T(n/2) + n: Θ(n log n).

public int f5(int n) {
  if (n <= 1) {
    return 1;
  }
  return f1(n) + f5(n / 2) + f5(n / 2);
}

Application: f4 and f5

f5(8): Recursion Tree

8442222111111118

Root: inside f5(8), f1(8) takes 8 steps.

84422221111111184 + 4 = 8

Level 1: two f5(4), each with f1(4) = 4 steps → 8 in total.

84422221111111184 + 4 = 84 · 2 = 8

Level 2: four f5(2), 2 steps each → again 8. Every level does n work.

84422221111111184 + 4 = 84 · 2 = 88 · 1 = 8

Leaves: 8 × 1. Total 3 · 8 + 8 = 32 = n log₂ n + n → Θ(n log n). Java: f5(8) = 32.

Each of the log₂ n levels does n work. In Java: f5(16) = 80 = 16 · 4 + 16, f5(32) = 192 = 32 · 5 + 32.

Application: f4 and f5

f5: Solving with the Master Theorem

T(n) = 2T(n/2) + Θ(n)

Of the form T(n) = aT(n/b) + f(n).

a = 2, b = 2, f(n) = Θ(n)

Same parameters as f4; only f(n) has changed.

logb a = log₂ 2 = 1,   c = 1

f(n) = Θ(n) → c = 1 = logb a.

Case 2: T(n) = Θ(nlogb a log n) = Θ(n log n)

Equal exponents: Θ(n log n). The same recurrence as the example earlier in the chapter.

f4f5
Calls2 × n/22 × n/2
Extra work f(n)O(1)Θ(n)
c vs logb a0 < 11 = 1
Case12
ResultΘ(n)Θ(n log n)

Comparison

f1–f6: Step Counts Measured in Java

FunctionRecurrence / sumn = 16n = 32RatioResult
f1 returnΣ 1 = n1632× 2Θ(n)
f2 returnΣ i²1,24010,416× 8.4Θ(n³)
f3 calls2T(n − 1) + c65,5354,294,967,295× 65,537Θ(2ⁿ)
f4 calls2T(n/2) + c3163× 2Θ(n)
f5 return2T(n/2) + n80192× 2.4Θ(n log n)
f6 returni ← 2i45+ 1Θ(log* n)
(mathematical model; in Java n ≤ 65536)

(long counter; f3(32) ≈ 4.3·10⁹ calls)

The ratio when n doubles gives away the growth class: Θ(n) → ×2, Θ(n log n) → a bit more than ×2, Θ(n³) → ×8, Θ(2ⁿ) → ×2ⁿ.

Comparison

Interactive f1–f6 Step Counter

FunctionStep countRelative size (log scale)

Steps: number of x++ for f1, f2, f6; number of calls for f3, f4; calls + f1 loop steps for f5. Values follow the logic of the code (BigInt for f3).

Comparison

A Quick Look: Amortized Analysis

  • Amortized analysis divides the total cost of a sequence of operations by the number of operations.
  • Dynamic array: when full, its capacity doubles and every element is copied (expensive); other insertions cost 1.
  • Over n insertions the copies total 1 + 2 + 4 + … < 2n → total < 3n → amortized Θ(1) per insertion.
  • Unlike the average case, there is no probability: the guarantee holds for every sequence of operations.
112233145516171899total = 24 · average = 2.67

9 insertions starting from capacity 1; orange bars include a resize (copy).

A single insertion can take Θ(n) in the worst case, yet the whole sequence costs 24 < 3 · 9 = 27 units. More detail: the aggregate, accounting and potential methods.

Summary

Summary

TopicKey points
Complexitytime = number of basic operations T(n); space = extra memory (including the stack); best / worst / average case
NotationO: upper, Ω: lower, Θ: tight bound; proved with c and n₀; drop constants and lower-order terms
Loopsadd up iteration counts: f1 Σ1 = n → Θ(n); f2 Σi² → Θ(n³); f6 i ← 2i → Θ(log* n) (mathematical model; in Java n ≤ 65536; O(log n) is not tight)
Recursionset up a recurrence: number of calls, how the input shrinks, extra work; solve by expansion or a recursion tree
f32T(n − 1) + c = (2ⁿ − 1)c → Θ(2ⁿ); the Master Theorem does not apply
Master TheoremaT(n/b) + f(n): compare f(n) with nlogb a; leaves / balanced / root dominates
f4, f52T(n/2) + 1 → Case 1, Θ(n); 2T(n/2) + n → Case 2, Θ(n log n)

Summary

Common Mistakes

Equating Big O with the worst caseO is an upper bound; the worst case is a kind of input. "The best case is O(1)" is a perfectly valid statement.
Keeping constants and small termsDon't write O(3n + 3) or O(n²/3 + n): O(n) and O(n²) are enough.
"Nested loops = n²"The inner loop's bound decides: in f2 it is i², hence Θ(n³); an i *= 2 loop runs only log n times.
"Two calls = exponential"f3 (n − 1) is Θ(2ⁿ), f4 (n/2) is Θ(n). How the input shrinks changes everything.
Applying the Master Theorem everywhereIt does not hold for recurrences with T(n − 1), unequal splits, or f(n) = n/log n.
Forgetting regularity in Case 3c > logb a is not enough; a · f(n/b) ≤ k · f(n) (k < 1) must hold too.

Summary

Sample Problems

  1. Show that 2n + 10 = O(n) by finding c and n₀.
  2. Solve T(n) = 3T(n/3) + n.
  3. Solve T(n) = 9T(n/3) + n.
  4. Solve T(n) = T(n/2) + n (check regularity).
  5. How many times does for (i = 1; i < n; i *= 2) for (j = 0; j < i; j++) x++; execute x++?
  6. What is the complexity of for (i = n; i > 0; i /= 2) for (j = 0; j < n; j++) x++;?
Show answers
  1. 2n + 10 ≤ 3n ⇔ n ≥ 10: c = 3, n₀ = 10 (or c = 12, n₀ = 1).
  2. a = 3, b = 3, log₃ 3 = 1 = c → Case 2 → Θ(n log n).
  3. log₃ 9 = 2 > c = 1 → Case 1 → Θ(n²).
  4. log₂ 1 = 0 < c = 1; 1 · (n/2) ≤ ½ · n (k = ½) → Case 3 → Θ(n).
  5. i = 1, 2, 4, …, 2ᵏ < n: total 1 + 2 + … + 2ᵏ = 2ᵏ⁺¹ − 1 < 2n → Θ(n) (n = 16: 15).
  6. The outer loop runs ⌊log₂ n⌋ + 1 times with n work each → Θ(n log n).

Summary

Exercises

  1. If the inner condition in f2 were j < i, what would the complexity be? What would f2(5) return?
  2. Write and solve the recurrence for return g(n/2) + g(n/2) + g(n/2) + g(n/2); (base case n ≤ 1).
  3. If the last line of f3 were return 2 * f3(n - 1);, how would its return value and running time change?
  4. Solve T(n) = 8T(n/2) + n² and T(n) = 7T(n/2) + n². Which algorithms do they describe?
  5. Prove by contradiction that n² ≠ O(n).
  6. If f6's loop were for (i = 1; i < n; i *= 2), how many iterations would it make for n = 1000? Compare with f6(1000).
Answers / hints
  1. Σ i = n(n − 1)/2 → Θ(n²); f2(5) = 0 + 1 + 2 + 3 + 4 = 10.
  2. T(n) = 4T(n/2) + c; log₂ 4 = 2 > 0 → Case 1 → Θ(n²).
  3. Same return value (2ⁿ⁻¹), but a single call: T(n) = T(n − 1) + c → Θ(n).
  4. log₂ 8 = 3 > 2 → Θ(n³) (divide-and-conquer matrix multiplication); log₂ 7 ≈ 2.807 → Θ(n2.807) (Strassen).
  5. Suppose n² ≤ c·n for all n ≥ n₀ ⇒ n ≤ c. Choosing n = max(n₀, ⌈c⌉ + 1) gives a contradiction.
  6. i = 1, 2, 4, …, 512 → 10 iterations (⌈log₂ 1000⌉); f6(1000) makes only 5.

Summary

References

  • Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., MIT Press, 2022 (Chapters 3 and 4).
  • Sedgewick and Wayne, Algorithms, 4th ed., Addison-Wesley, 2011 (Section 1.4: Analysis of Algorithms).
  • Kleinberg and Tardos, Algorithm Design, Pearson, 2005 (Chapters 2 and 5).
  • D. E. Knuth, "Big Omicron and Big Omega and Big Theta", SIGACT News 8(2), 1976.
  • J. L. Bentley, D. Haken and J. B. Saxe, "A General Method for Solving Divide-and-Conquer Recurrences", SIGACT News 12(3), 1980.

The End

Chapter 2: Algorithm Complexity
Previous: Chapter 1: Introduction · Next: Chapter 3: Sorting Algorithms
1 / 1 Sercan KÜLCÜ. All rights reserved.