Chapter 2: Algorithm Analysis
Introduction
Learning Outcomes
Time an algorithm experimentally and analyze it by counting its primitive operations.
Express time and space complexity with O, Ω, and Θ.
Read best-, average-, and worst-case cost off loop patterns.
Compare algorithms for the same problem by cost, correctness, and robustness.
Algorithms and Measurement
Algorithms
- An algorithm is a step-by-step procedure for solving a problem: input → steps → output.
- One problem can have many algorithms, and some are more efficient than others.
- Choosing the right one matters: a good algorithm does the job with fewer resources.
Some common algorithms:
Algorithms and Measurement
What Makes a Good Algorithm
A good algorithm solves its problem effectively. We judge it by four criteria:
Algorithms and Measurement · Measuring
Experimental Analysis: System.nanoTime()
for (int n = 1000; n <= 64000; n *= 2) { int[] a = new int[n]; for (int i = 0; i < n; i++) a[i] = i; // distinct long start = System.nanoTime(); hasDuplicates(a); // timed work long elapsed = System.nanoTime() - start; // ns System.out.println(n + " " + elapsed / 1e6 + " ms"); }
System.nanoTime() readings gives elapsed ns (not wall-clock time).new int[n] is all zeros → hasDuplicates (an O(n²) duplicate check) returns true on its first comparison. To time the worst case, use distinct data.Algorithms and Measurement · Measuring
Doubling Experiments and the Limits of Measurement
Limits of experimental analysis (Goodrich §4.1):
- Results depend on the hardware, the JVM, and the load; compare two algorithms in the same environment.
- Only a limited set of inputs can be tried; an important one may be missed.
- The algorithm must be fully implemented before it can be timed.
Algorithms and Measurement · Counting steps
Counting Primitive Operations: arrayMax()
int arrayMax(int[] a) { int currentMax = a[0]; for (int i = 1; i < a.length; i++) { if (a[i] > currentMax) currentMax = a[i]; } return currentMax; }
- Primitive operation: assignment, comparison, arithmetic, array indexing, method call / return.
- Each takes constant time → we only count how many times each one runs.
| Line | Statement | Count |
|---|---|---|
| 2 | currentMax = a[0] | 1 |
| 3 | i = 1 | 1 |
| 3 | i < a.length | n |
| 4 | a[i] > currentMax | n − 1 |
| 5 | currentMax = a[i] | 0 … n − 1 |
| 3 | i++ | n − 1 |
| 7 | return | 1 |
Algorithms and Measurement · Step through the code
Code Trace: arrayMax(a)
currentMax = a[0] = 3: the first item is the largest so far.
i = 1, 1 < 5 → continue. Invariant: currentMax = max(a[0..0]).
a[1] = 8 > 3 → true.
currentMax = 8: a new maximum.
i = 2, 2 < 5 → continue. Invariant: currentMax = max(a[0..1]).
a[2] = 2 > 8 → false; currentMax is unchanged.
i = 3, 3 < 5 → continue. Invariant: currentMax = max(a[0..2]).
a[3] = 9 > 8 → true.
currentMax = 9: a new maximum.
i = 4, 4 < 5 → continue. Invariant: currentMax = max(a[0..3]).
a[4] = 4 > 9 → false; currentMax is unchanged.
i = 5: 5 < 5 is false → loop ends. The whole array has been scanned.
return 9. n − 1 = 4 comparisons for n = 5 → Θ(n).
int arrayMax(int[] a) { int currentMax = a[0]; for (int i = 1; i < a.length; i++) { if (a[i] > currentMax) currentMax = a[i]; } return currentMax; }
Asymptotic Notation
Time Complexity and Big O
- Expresses an algorithm's running time as a function of the input size n.
- Big O gives an upper bound on running time; it is usually quoted for the worst case.
- O(1): the operation count does not grow with the input size, though the time need not be identical for every input.
- O(n): time grows linearly with the input.
3n2 + 5n + 2 → O(n2) · 4n + 100 → O(n) · 7 → O(1)
Asymptotic Notation
O, Ω, and Θ: Definition and Graphs
Big O: 2n + 10 ≤ 4·n for n ≥ n₀ = 5 → f(n) = O(n) (c = 4). For small n (n < 5) f(n) is larger, which does not matter.
Big Omega: 2n + 10 ≥ 2·n for every n ≥ 1 → f(n) = Ω(n): f grows no more slowly than n (lower bound).
Big Theta: 2n ≤ f(n) ≤ 4n for n ≥ 5 → f(n) = Θ(n): squeezed between two lines, it grows exactly like n.
Asymptotic Notation
Asymptotic Bounds and Cases
| Analysis | Question | Linear search |
|---|---|---|
| Best case | How many operations on the most favorable valid input? | Θ(1) if the first item is the target |
| Average case | What is the expected cost under an assumed input distribution? | Θ(n) if all positions are equally likely |
| Worst case | What cost is guaranteed for every valid input? | Θ(n) if the target is last or absent |
Asymptotic Notation · Case analysis
Best, Average, and Worst Case
Linear search on [4, 8, 15, 16, 23, 42]: comparisons for each possible target:
- For the same n, the cost depends on the input.
- The worst case is a guarantee and easy to compute → it is what we usually report.
- The average assumes a probability distribution (here 7 equally likely targets, including the absent 50: 27/7 ≈ 3.86).
- The best case alone is misleading.
Asymptotic Notation
Big O Examples
| Notation | Name | Example | Why? |
|---|---|---|---|
| O(1) | Constant | Accessing the first array element | Access time is the same whatever the array size |
| O(log n) | Logarithmic | Binary search in a sorted array | Each step halves the search range |
| O(n) | Linear | Scanning an array end to end | Each item is processed once |
| O(n log n) | Linearithmic | Merge sort | Divide and conquer: log n levels, n work per level |
| O(n2) | Quadratic | Comparing all pairs of items | Two nested loops |
| O(2n) | Exponential | Listing all subsets | A set of n items has 2n subsets |
Asymptotic Notation
What Happens as n Grows?
| Complexity | n = 10 | n = 100 | n = 1,000 |
|---|---|---|---|
| log2 n | 3.3 | 6.6 | 10 |
| n | 10 | 100 | 1,000 |
| n log2 n | 33 | 664 | 9,966 |
| n2 | 100 | 10,000 | 1,000,000 |
| 2n | 1,024 | ≈ 1.3·1030 | ≈ 1.1·10301 |
From Code to Complexity · Counting steps
From Loop Patterns to Complexity
| Pattern | Body runs | n = 16 | Class |
|---|---|---|---|
for (i = 0; i < n; i++) | n | 16 | O(n) |
for (i = 0; i < n; i += 2) | n / 2 | 8 | O(n) |
for (i = 1; i < n; i *= 2) | ⌈log₂ n⌉ | 4 | O(log n) |
nested: i < n, j < n | n² | 256 | O(n²) |
nested: i < n, j < i | n(n−1)/2 | 120 | O(n²) |
two consecutive i < n loops | 2n | 32 | O(n) |
From Code to Complexity · Step through the code
Code Trace: countSteps(16) — i doubles every round
countSteps(16): how many doublings does it take to reach n?
Counter k = 0.
i = 1 < 16 → body runs.
k++ → k = 1.
i = 2 < 16 → body runs.
k++ → k = 2.
i = 4 < 16 → body runs.
k++ → k = 3.
i = 8 < 16 → body runs.
k++ → k = 4.
i *= 2 → i = 16; 16 < 16 is false → loop ends.
return 4 = log₂ 16. Making n 1000 times larger adds only ~10 rounds → O(log n).
int countSteps(int n) { int k = 0; for (int i = 1; i < n; i *= 2) k++; return k; }
From Code to Complexity
From Code to Complexity
int first(int[] a) { return a[0]; }
int sum(int[] a) { int s = 0; for (int x : a) // n times s += x; return s; }
boolean hasDuplicates(int[] a) { for (int i = 0; i < a.length; i++) for (int j = i + 1; j < a.length; j++) if (a[i] == a[j]) return true; return false; } // outer loop n, inner ≈ n/2 // → n(n−1)/2 comparisons
Rule: drop constants and lower-order terms. n(n−1)/2 = n²/2 − n/2 → O(n²)
From Code to Complexity · Step through the code
Code Trace: sum()
main calls sum(a), and a new call frame is pushed.
Inside the method, a refers to the same array (no copy is made).
The running total starts at s = 0.
Round 1: x = a[0] = 7.
s += x → s = 0 + 7 = 7.
Round 2: x = a[1] = 2.
s += x → s = 7 + 2 = 9.
Round 3: x = a[2] = 9.
s += x → s = 9 + 9 = 18.
No items left; the loop exits.
return 18: the frame is popped and the value goes back to the caller.
println prints 18. The loop body ran n = 3 times → O(n).
int sum(int[] a) { int s = 0; for (int x : a) { s += x; } return s; } // in main: System.out.println(sum(a));
From Code to Complexity · Step through the code
Code Trace: linearSearch(a, 23)
Searching a sorted array for 23. Linear search ignores the order and checks items one by one from the start.
a[0] = 2 ≠ 23 → i moves on. Each step rules out only one item.
a[1] = 5 ≠ 23 → i moves on.
a[2] = 8 ≠ 23 → i moves on.
a[3] = 12 ≠ 23 → i moves on.
a[4] = 16 ≠ 23 → i moves on.
a[5] = 23 == 23 → return 5. 6 comparisons; with the target last or absent, n = 10 → O(n).
int linearSearch(int[] a, int target) { for (int i = 0; i < a.length; i++) if (a[i] == target) return i; return -1; }
From Code to Complexity · Step through the code
Code Trace: binarySearch(a, 23)
The search range is the whole array: lo = 0, hi = 9.
0 ≤ 9 → range nonempty, round 1 begins.
mid = (0 + 9) / 2 = 4, a[4] = 16.
16 < 23 → the target is in the right half: lo = mid + 1 = 5. Left half discarded.
5 ≤ 9 → range nonempty, round 2 begins.
mid = (5 + 9) / 2 = 7, a[7] = 56.
56 > 23 → the target is in the left half: hi = mid − 1 = 6. Right half discarded.
5 ≤ 6 → range nonempty, round 3 begins.
mid = (5 + 6) / 2 = 5, a[5] = 23.
23 == 23 → found, return 5. Only 3 rounds; linear search needed 6 comparisons. Each round halves the range → O(log n).
int binarySearch(int[] a, int target) { int lo = 0, hi = a.length - 1; while (lo <= hi) { int mid = (lo + hi) / 2; if (a[mid] == target) return mid; if (a[mid] < target) lo = mid + 1; else hi = mid - 1; } return -1; }
From Code to Complexity · Step through the code
Code Trace: binarySearch(a, 40) — target absent
This time 40 is absent: the worst case. The range is the whole array.
0 ≤ 9 → range nonempty, round 1.
Round 1: mid = (0 + 9) / 2 = 4, a[4] = 16.
16 < 40 → lo = 5; left side discarded.
5 ≤ 9 → range nonempty, round 2.
Round 2: mid = (5 + 9) / 2 = 7, a[7] = 56.
56 > 40 → hi = 6; right side discarded.
5 ≤ 6 → range nonempty, round 3.
Round 3: mid = (5 + 6) / 2 = 5, a[5] = 23.
23 < 40 → lo = 6; left side discarded.
6 ≤ 6 → range nonempty, round 4.
Round 4: mid = (6 + 6) / 2 = 6, a[6] = 38.
38 < 40 → lo = 7; left side discarded.
7 ≤ 6 is false → range empty, loop ends.
return -1. Even in the worst case, only 4 rounds = ⌊log₂ 10⌋ + 1; linear search would need 10 comparisons.
int binarySearch(int[] a, int target) { int lo = 0, hi = a.length - 1; while (lo <= hi) { int mid = (lo + hi) / 2; if (a[mid] == target) return mid; if (a[mid] < target) lo = mid + 1; else hi = mid - 1; } return -1; }
From Code to Complexity · Step through the code
Code Trace: hasDuplicates()
Outer loop: i = 0 (a[i] = 4). j starts at i + 1.
a[0] = 4 vs. a[1] = 7 → different, j moves on.
a[0] = 4 vs. a[2] = 1 → different, j moves on.
a[0] = 4 vs. a[3] = 7 → different, j moves on.
Outer loop: i = 1 (a[i] = 7). j starts at i + 1.
a[1] = 7 vs. a[2] = 1 → different, j moves on.
a[1] = 7 and a[3] = 7 are equal → return true. The worst case (no duplicates) takes n(n−1)/2 = 6 comparisons → O(n²).
boolean hasDuplicates(int[] a) { for (int i = 0; i < a.length; i++) for (int j = i + 1; j < a.length; j++) if (a[i] == a[j]) return true; return false; }
From Code to Complexity · One problem, two algorithms
Example Problem: Prefix Averages
Input x[0..n−1]; output a[j] = (x[0] + … + x[j]) / (j + 1). Used for moving averages and financial trend analysis (Goodrich §4.3).
a[0] = average of x[0] … x[0] = 4.0.
a[1] = average of x[0] … x[1] = 6.0.
a[2] = average of x[0] … x[2] = 6.0.
a[3] = average of x[0] … x[3] = 5.0.
Two solutions: re-sum from scratch for each j (O(n²)) · carry a running sum (O(n)).
From Code to Complexity · Step through the code
Code Trace: prefixAverage1 — re-sum from scratch for each j
n = 4; the result array a is created, filled with 0.0.
j = 0: total = 0 — each j re-sums from scratch.
total += x[0] = 4 → total = 4.0.
a[0] = 4.0 / 1 = 4.0.
j = 1: total = 0 — each j re-sums from scratch.
total += x[0] = 4 → total = 4.0.
total += x[1] = 8 → total = 12.0.
a[1] = 12.0 / 2 = 6.0.
j = 2: total = 0 — each j re-sums from scratch.
total += x[0] = 4 → total = 4.0.
total += x[1] = 8 → total = 12.0.
total += x[2] = 6 → total = 18.0.
a[2] = 18.0 / 3 = 6.0.
j = 3: total = 0 — each j re-sums from scratch.
total += x[0] = 4 → total = 4.0.
total += x[1] = 8 → total = 12.0.
total += x[2] = 6 → total = 18.0.
total += x[3] = 2 → total = 20.0.
a[3] = 20.0 / 4 = 5.0.
return a. Additions: 1 + 2 + 3 + 4 = 10 = n(n+1)/2 → O(n²).
double[] prefixAverage1(double[] x) { int n = x.length; double[] a = new double[n]; for (int j = 0; j < n; j++) { double total = 0; for (int i = 0; i <= j; i++) total += x[i]; a[j] = total / (j + 1); } return a; }
From Code to Complexity · Step through the code
Code Trace: prefixAverage2 — running sum
total is set to 0 once, before the loop, and carried across rounds.
j = 0: total += x[0] = 4 → total = 4.0 (previous sum reused).
a[0] = 4.0 / 1 = 4.0.
j = 1: total += x[1] = 8 → total = 12.0 (previous sum reused).
a[1] = 12.0 / 2 = 6.0.
j = 2: total += x[2] = 6 → total = 18.0 (previous sum reused).
a[2] = 18.0 / 3 = 6.0.
j = 3: total += x[3] = 2 → total = 20.0 (previous sum reused).
a[3] = 20.0 / 4 = 5.0.
Same result with only 4 additions (prefixAverage1: 10). One loop → O(n).
double[] prefixAverage2(double[] x) { int n = x.length; double[] a = new double[n]; double total = 0; for (int j = 0; j < n; j++) { total += x[j]; a[j] = total / (j + 1); } return a; }
From Code to Complexity · One problem, two algorithms
Comparing the Two Solutions
| n | prefixAverage1 | prefixAverage2 |
|---|---|---|
| 10 | 55 | 10 |
| 100 | 5,050 | 100 |
| 1,000 | 500,500 | 1,000 |
| 1,000,000 | 500,000,500,000 | 1,000,000 |
Space and Correctness
Space Complexity
- Measures how much memory an algorithm uses, usually in Big O notation.
- Describes how memory use grows with the input size (usually not counting the input itself: extra space).
- Memory efficiency matters on systems with limited memory.
Space and Correctness
Correctness
- A correct algorithm produces the expected output for every valid input in finite time.
- A fast algorithm that gives wrong answers is worthless.
How do we show correctness?
Space and Correctness
Robustness
- An algorithm's ability to cope with unexpected situations.
- A robust algorithm handles even out-of-range inputs without crashing or giving wrong results.
How do we test robustness?
Summary
Common Mistakes
j *= 2, it is n log n; if it runs a fixed number of times, O(n).(lo + hi) / 2 overflowFor very large arrays the int sum overflows; lo + (hi - lo) / 2 is safe.a == b== compares references; use Arrays.equals(a, b) for the contents.Lab · Java code, step by step
Lab: Binary Search and Prefix Averages
Summary
Summary
| Topic | Key points |
|---|---|
| Experimental analysis | time depends on hardware and input; doubling experiment: T(2n)/T(n) ≈ 2k → T(n) ≈ c·nk |
| Counting operations | assignment, comparison, arithmetic, array access; hardware-independent, valid for all inputs |
| O, Ω, Θ | upper, lower, and tight bounds; constant factors and lower-order terms are dropped |
| Cases | best, average, worst; the worst case for a guarantee, amortized for long sequences of operations |
| Loop patterns | single loop O(n) · nested O(n²) · halving O(log n) · outer n, inner log n → O(n log n) |
| Choosing an algorithm | prefix averages O(n²) → O(n); binary search in a sorted array O(log n) |
| Space, correctness | extra memory is measured in Big O too; a fast algorithm that is not correct and robust is worthless |
Summary
Operation Costs in Basic Data Structures
| Data structure | Access item i | Search | Insert | Delete |
|---|---|---|---|---|
| Array | O(1) | O(n) | O(n) | O(n) |
| Linked list | O(n) | O(n) | O(1)† | O(1)† |
| Stack | top only | O(n) | O(1) push | O(1) pop |
| Queue | front only | O(n) | O(1) enqueue | O(1) dequeue |
| Map / set (hash) | — | O(1)* | O(1)* | O(1)* |
† If the position (the previous node) is known. * Average case; O(n) in the worst case.
Summary
Sample Problems
- What is the time complexity of an algorithm that compares every pair of items?
- Which Θ class does f(n) = 5n³ + 2n² + 100 belong to?
- How many rounds does binary search take in the worst case on a sorted array of n = 1024 items?
- How many times does the loop
for (int i = 1; i < n; i *= 2)run? - What are the best and worst cases of linear search?
- Algorithm A takes 100n steps, algorithm B takes n². For which values of n is A faster?
Show answers
- n(n − 1)/2 comparisons → O(n2).
- Θ(n³): the fastest-growing term decides; constants are dropped.
- ⌊log₂ 1024⌋ + 1 = 11 rounds.
- i = 1, 2, 4, …: ⌈log₂ n⌉ rounds → O(log n).
- Best: the target is the first item, Θ(1). Worst: the target is last or absent, Θ(n).
- 100n < n² ⇔ n > 100: for large n, the slower-growing algorithm wins.
Summary
Exercises
- What is the Big O of this code?
for (int i = n; i > 1; i /= 2) for (int j = 0; j < n; j++) s++;
- Show that f(n) = 3n² + 5n + 2 = O(n²) by giving c and n₀.
- In a doubling experiment, doubling n multiplies the time by ≈ 8. What class is T(n) in?
- How many additions do prefixAverage1 and prefixAverage2 each make for n = 5?
- After
for (i = 0; i < n; i++) for (j = i; j < n; j++) s++;, what is s? - An O(n²) algorithm takes 2 seconds for n = 1000. About how long will it take for n = 4000?
Answers / hints
- The outer loop runs ⌊log₂ n⌋ rounds, the inner loop n → O(n log n).
- For n ≥ 1, 5n ≤ 5n² and 2 ≤ 2n² → f(n) ≤ 10n²: c = 10, n₀ = 1.
- 2³ = 8 → O(n³) (cubic).
- prefixAverage1: 1+2+3+4+5 = 15; prefixAverage2: 5.
- n + (n − 1) + … + 1 = n(n + 1)/2 → O(n²).
- Quadrupling n multiplies the time by 4² = 16: ≈ 32 seconds.
Summary
Further Reading
- Goodrich, Tamassia, and Goldwasser, Data Structures and Algorithms in Java, 6th ed., 2014.
- Cormen, Leiserson, Rivest, and Stein, Introduction to Algorithms, 4th ed., 2022.
- Sedgewick and Wayne, Algorithms, 4th ed., 2011.