Chapter 1: Introduction
Introduction
Learning Outcomes
Define an algorithm and its properties (input, output, definiteness, finiteness, effectiveness); write an everyday task as an algorithm.
Express running time as a function T(n) of the input size n; read from a growth table how time changes as n grows.
Count the operations in short code to find its complexity class (1, n, √n, n³, n!, 2ⁿ); see how memoization cuts exponential time.
Trace a gcd computation step by step; justify its correctness and its O(log min(a, b)) running time.
The Notion of an Algorithm
What Is an Algorithm?
- A finite sequence of step-by-step instructions for solving a problem or carrying out a task.
- It takes input, performs operations and produces output.
- Every step must be precise and unambiguous, and the algorithm must terminate after finitely many steps.
The Notion of an Algorithm
An Everyday Example: Making Tea
Input: water, tea leaves, sugar (optional), milk (optional)
Output: tea
- Boil the water.
- Pour the boiling water into the teapot.
- Add the tea leaves to the pot.
- Let the tea brew.
- Strain the tea into a cup.
- Add sugar and milk if desired.
The Notion of an Algorithm
Computing an Average: From Algorithm to Code
1. sum = x1 + x2 + x3
2. average = sum / 3
3. Print average.
Python
def average(x1, x2, x3): avg = (x1 + x2 + x3) / 3 return avg x1 = float(input("Enter the first number: ")) x2 = float(input("Enter the second number: ")) x3 = float(input("Enter the third number: ")) avg = average(x1, x2, x3) print("Average of the numbers:", avg)
Java
public static void main(String[] args) { Scanner in = new Scanner(System.in); System.out.println("Enter the first number: "); double x1 = in.nextDouble(); System.out.println("Enter the second number: "); double x2 = in.nextDouble(); System.out.println("Enter the third number: "); double x3 = in.nextDouble(); double avg = (x1 + x2 + x3) / 3; System.out.println("Average of the numbers: " + avg); }
The same algorithm can be written in any language: an algorithm is language-independent, and a program is one implementation of it.
The Notion of an Algorithm
Main Kinds of Algorithms
Analysis of Algorithms
Time Complexity
- Algorithm analysis measures the use of resources such as processor time and memory, so we can choose between algorithms that solve the same problem.
- Running time is expressed in terms of the input size n: T(n). We count basic operations, not seconds: seconds depend on the machine and language, operation counts do not.
Example: summing an n-element array, counted line by line
int sum = 0; // once for (int i = 0; i < n; i++) // n+1 comparisons sum += arr[i]; // n times return sum; // once
Analysis of Algorithms
Growth Table: As n Grows
| f(n) | n = 10 | n = 100 | n = 1000 | n = 10⁶ | time (n = 10⁶) |
|---|---|---|---|---|---|
| 1 | 1 | 1 | 1 | 1 | 1 ns |
| log₂ n | 3.3 | 6.6 | 10.0 | 19.9 | 19.9 ns |
| √n | 3.2 | 10 | 31.6 | 1,000 | 1 µs |
| n | 10 | 100 | 1,000 | 1,000,000 | 1 ms |
| n log₂ n | 33.2 | 664.4 | 9,966 | 2.0 · 10⁷ | 19.9 ms |
| n² | 100 | 10,000 | 1,000,000 | 1.0 · 10¹² | 16.7 min |
| n³ | 1,000 | 1,000,000 | 1.0 · 10⁹ | 1.0 · 10¹⁸ | ≈ 32 years |
| 2ⁿ | 1,024 | 1.3 · 10³⁰ | ≈ 1.1 · 10³⁰¹ | ≈ 10³⁰¹⁰³⁰ | > age of universe |
| n! | 3,628,800 | 9.3 · 10¹⁵⁷ | ≈ 10²⁵⁶⁷ | ≈ 10⁵⁵⁶⁵⁷⁰⁸ | > age of universe |
Complexity Examples
Constant and Linear Time: O(1), O(n)
Printing the first element T(n) = O(1)
void printFirst(int[] arr) { if (arr.length > 0) System.out.println("First element: " + arr[0]); else System.out.println("Array is empty"); }
Whether the array has 10 elements or 10 million, it does one comparison and one access: the time does not depend on n.
Finding the maximum T(n) = O(n)
Integer findMax(int[] arr) { if (arr.length == 0) return null; // empty array int max = arr[0]; for (int x : arr) if (x > max) max = x; return max; }
Each element is examined once: n comparisons, O(1) extra memory. Example: [5, 8, 12, 4] → 12.
Complexity Examples
Primality Test: O(√n)
boolean isPrime(int n) { if (n <= 1) return false; else if (n == 2) return true; else if (n % 2 == 0) return false; int limit = (int) Math.sqrt(n) + 1; for (int i = 3; i < limit; i += 2) if (n % i == 0) return false; return true; }
n = 97: i = 3, 5, 7, 9 → 4 divisions (95 if we tried 2 to 96).
n = 1,000,000,007: 15,810 divisions.
Note: here the input size is the value of n. If n has b bits, √n = 2b/2: the method is exponential in the number of bits.
Complexity Examples · Run the code step by step
isPrime(97): Step by Step
n = 97: greater than 1, not equal to 2, and odd. None of the first three checks returns; multiples of 2 are ruled out up front.
limit = (int) √97 + 1 = 9 + 1 = 10. Only odd numbers below 10 are tried: 3, 5, 7, 9.
i = 3: 97 % 3 = 1 ≠ 0 → 3 does not divide 97. i += 2.
i = 5: 97 % 5 = 2 ≠ 0 → 5 does not divide 97. i += 2.
i = 7: 97 % 7 = 6 ≠ 0 → 7 does not divide 97. i += 2.
i = 9: 97 % 9 = 7 ≠ 0 → 9 does not divide 97. i += 2. Next candidate: i = 11.
i = 11 < 10 fails and the loop ends. No odd number up to √97 divides 97.
return true: 97 is prime. Only 4 divisions were needed; trying every number from 2 to 96 would take 95.
boolean isPrime(int n) { if (n <= 1) return false; else if (n == 2) return true; else if (n % 2 == 0) return false; int limit = (int) Math.sqrt(n) + 1; for (int i = 3; i < limit; i += 2) if (n % i == 0) return false; return true; }
Complexity Examples
Cubic Time: Matrix Multiplication, Θ(n³)
int[][] multiply(int[][] A, int[][] B) { if (B.length == 0 || (A.length > 0 && A[0].length != B.length)) throw new IllegalArgumentException("size"); int n = A.length, m = B[0].length, p = B.length; int[][] result = new int[n][m]; for (int i = 0; i < n; i++) for (int j = 0; j < m; j++) { int sum = 0; for (int k = 0; k < p; k++) sum += A[i][k] * B[k][j]; result[i][j] = sum; } return result; }
Input: A = [aᵢⱼ], B = [bᵢⱼ] (n × n). Output: C = [cᵢⱼ] = A · B, i, j = 1, 2, …, n.
| 1 | 2 |
| 3 | 4 |
| 5 | 6 |
| 7 | 8 |
| 19 | 22 |
| 43 | 50 |
c₁₁ = 1·5 + 2·7 = 19. Each cᵢⱼ takes n multiplications and there are n² entries → n³ multiplications: T(n) = Θ(n³).
Complexity Examples · Run the code step by step
multiply: 2 × 2 Matrices Step by Step
A and B are 2 × 2 with compatible sizes: n = m = p = 2. result starts as a 2 × 2 zero matrix.
i = 0, j = 0: sum = 0. k = 0: add A[0][0] · B[0][0] = 1 · 5 = 5.
k = 1: A[0][1] · B[1][0] = 2 · 7 → sum = 5 + 14 = 19. Inner loop ends: result[0][0] = 19.
i = 0, j = 1: sum = 0. k = 0: add A[0][0] · B[0][1] = 1 · 6 = 6.
k = 1: A[0][1] · B[1][1] = 2 · 8 → sum = 6 + 16 = 22. Inner loop ends: result[0][1] = 22.
i = 1, j = 0: sum = 0. k = 0: add A[1][0] · B[0][0] = 3 · 5 = 15.
k = 1: A[1][1] · B[1][0] = 4 · 7 → sum = 15 + 28 = 43. Inner loop ends: result[1][0] = 43.
i = 1, j = 1: sum = 0. k = 0: add A[1][0] · B[0][1] = 3 · 6 = 18.
k = 1: A[1][1] · B[1][1] = 4 · 8 → sum = 18 + 32 = 50. Inner loop ends: result[1][1] = 50.
Returns result = [[19, 22], [43, 50]]. 8 = 2³ multiplications: n multiplications for each of the n² entries → n³.
int[][] multiply(int[][] A, int[][] B) { if (B.length == 0 || (A.length > 0 && A[0].length != B.length)) throw new IllegalArgumentException("size"); int n = A.length, m = B[0].length, p = B.length; int[][] result = new int[n][m]; for (int i = 0; i < n; i++) for (int j = 0; j < m; j++) { int sum = 0; for (int k = 0; k < p; k++) sum += A[i][k] * B[k][j]; result[i][j] = sum; } return result; }
Complexity Examples
Factorial Time: Generating All Permutations
void permute(int[] arr, int k) { if (k == arr.length) { // one permutation done System.out.println(Arrays.toString(arr)); return; } for (int i = k; i < arr.length; i++) { swap(arr, k, i); // put element i at k permute(arr, k + 1); // arrange the rest swap(arr, k, i); // undo the swap } }
[1, 2, 3] → 123, 132, 213, 231, 321, 312. There are n choices for the first position, n−1 for the second, …: n · (n−1) · … · 1 = n! permutations.
| n | n! |
|---|---|
| 3 | 6 |
| 5 | 120 |
| 10 | 3,628,800 |
| 15 | 1.3 · 10¹² |
| 20 | 2.4 · 10¹⁸ |
Printing each permutation takes n steps: Θ(n · n!) in total, the "O(n!) class" for short. Even n = 20 is out of reach in practice.
Complexity Examples
Exponential Time: Fibonacci Numbers
long fib(int n) { // naive recursion if (n <= 1) return n; return fib(n - 1) + fib(n - 2); } static final int N = 92; // largest n for long long[] memo = new long[N + 1]; // 0: not set long fibMemo(int n) { // memoization if (n <= 1) return n; if (memo[n] == 0) memo[n] = fibMemo(n - 1) + fibMemo(n - 2); return memo[n]; }
F(0) = 0, F(1) = 1, F(n) = F(n−1) + F(n−2).
fib: each call spawns two calls → O(2ⁿ) (exactly Θ(1.618ⁿ)); fib(30) makes 2,692,537 calls.
fibMemo: each F(k) is computed once, 2n − 1 calls → O(n); 59 calls for n = 30.
F(92) is the largest value that fits in a long; beyond that use BigInteger.
Naive recursion: 15 calls. f3 is computed twice, f2 three times, f1 five times; the tree roughly doubles at each level → exponential growth.
Memoized: 9 calls. The dashed orange nodes (f2, f3) come straight from the memo array; their subtrees are never expanded → O(n).
Euclid's Algorithm
Euclid's Algorithm (gcd)
- Finds the greatest common divisor (gcd).
- One of the oldest known algorithms (Euclid, c. 300 BC) and the simplest way to compute a gcd.
- Key idea: gcd(a, b) = gcd(b, a mod b), gcd(a, 0) = a.
- Step 1: divide the larger number by the smaller and take the remainder.
- Step 2: repeat with the pair (divisor, remainder) until the remainder is zero.
- Step 3: when the remainder is zero, the last divisor is the gcd.
int gcd(int a, int b) { if (b == 0) return a; return gcd(b, a % b); }
Euclid's Algorithm · Run the code step by step
gcd(1599, 650): Step by Step
1599 = 2 · 650 + 299. Cut 2 squares from the 1599×650 rectangle; the leftover strip is 650×299 → gcd(650, 299).
650 = 2 · 299 + 52. Cut 2 squares from the 650×299 rectangle; the leftover strip is 299×52 → gcd(299, 52).
299 = 5 · 52 + 39. Cut 5 squares from the 299×52 rectangle; the leftover strip is 52×39 → gcd(52, 39).
52 = 1 · 39 + 13. Cut 1 square from the 52×39 rectangle; the leftover strip is 39×13 → gcd(39, 13).
39 = 3 · 13 + 0. Cut 3 squares from the 39×13 rectangle; the leftover strip is 13×0 → gcd(13, 0).
b == 0 → return 13. 13 is the largest number that divides both 1599 and 650.
The value 13 is passed back unchanged through every call: gcd(1599, 650) = 13. Five divisions suffice, and 13 × 13 squares tile the 1599 × 650 rectangle exactly.
int gcd(int a, int b) { if (b == 0) return a; return gcd(b, a % b); }
Euclid's Algorithm
Euclid's Algorithm: Why Is It Fast?
- It is efficient: every two steps the larger number drops to less than half (a mod b < a/2).
- So the number of steps is at most ≈ 2 log₂ min(a, b): O(log min(a, b)).
- The worst case is consecutive Fibonacci numbers: every quotient is q = 1, so the numbers shrink as slowly as possible (Lamé's theorem).
- For comparison: trying every divisor from 1 to min(a, b) takes O(min(a, b)).
34 = 1 · 21 + 13
21 = 1 · 13 + 8
13 = 1 · 8 + 5
8 = 1 · 5 + 3
5 = 1 · 3 + 2
3 = 1 · 2 + 1
2 = 2 · 1 + 0
8 divisions, gcd = 1. gcd(1599, 650) needs only 5.
Summary
Summary
| Topic | Key points |
|---|---|
| Algorithms | input → precise, finite, effective steps → output; language-independent |
| Analysis | T(n): number of basic operations as a function of input size; time and memory |
| Growth | 1 < log n < √n < n < n log n < n² < n³ < 2ⁿ < n!; a better algorithm changes the growth rate |
| Examples | first element O(1), maximum O(n), primality O(√n), matrix multiplication Θ(n³), permutations O(n!), naive Fibonacci O(2ⁿ) |
| Memoization | storing repeated subproblems takes Fibonacci from O(2ⁿ) down to O(n) |
| Euclid | gcd(a, b) = gcd(b, a mod b); O(log min(a, b)); worst case: consecutive Fibonacci numbers |
Summary
Common Mistakes
i < (int) Math.sqrt(n), √n itself is never tried: squares such as 9, 25, 49 come out prime.swap and the array gets scrambled: for [1, 2, 3] some permutations repeat and others never appear.Summary
Sample Problems
- For
isPrime(221), which values of i does the loop try, how many divisions are made, and what is the result? - Give the Θ class of T(n) = 5n² + 100n log₂ n + 7. Which term dominates at n = 1000?
- How many multiplications does
multiplyperform on two 3 × 3 matrices? For n = 100? - How many lines does
permute([1, 2, 3, 4], 0)print? What are the first two? - Find gcd(252, 105) with Euclid's algorithm.
Show answers
- (int) √221 = 14 → limit = 15; i = 3, 5, 7, 9, 11, 13. 221 % 13 = 0 → false after 6 divisions (221 = 13 · 17).
- Θ(n²). At n = 1000: 5n² = 5,000,000 while 100n log₂ n ≈ 996,578 → 5n² dominates.
- 3 multiplications for each of the 9 entries cᵢⱼ → 27 (n³). For n = 100 → 10⁶ multiplications.
- 4! = 24 lines; the first two are [1, 2, 3, 4] and [1, 2, 4, 3].
- 252 = 2·105 + 42; 105 = 2·42 + 21; 42 = 2·21 + 0 → gcd = 21.
Summary
Exercises
- Write an algorithm that counts how many times a given number occurs in an array; give its time and space complexity.
- Nested loops: how many times does
for i in 0..n-1: for j in i..n-1: oprun? What is its class? - Modify
findMaxto return both the minimum and the maximum of the array. How many comparisons does it make? - Write Euclid's algorithm with a loop (no recursion).
- At 10⁹ operations per second, find the largest n solvable in 1 second for f(n) = n² and f(n) = 2ⁿ.
- How many times does the loop
while (n > 1) n = n / 2;run?
Answers / hints
- One pass with a counter: Θ(n) time, O(1) extra memory.
- n + (n−1) + … + 1 = n(n+1)/2 → Θ(n²).
- Comparing each element twice gives 2(n − 1); processing elements in pairs (compare the pair, then the smaller with the min and the larger with the max) gives ≈ 3n/2. Both are Θ(n).
while (b != 0) { int r = a % b; a = b; b = r; } return a;- n² ≤ 10⁹ → n ≈ 31,622; 2ⁿ ≤ 10⁹ → n = 29.
- ⌊log₂ n⌋ times → Θ(log n).
Summary
References
- Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., MIT Press, 2022 (Chapters 1–3).
- Sedgewick and Wayne, Algorithms, 4th ed., Addison-Wesley, 2011 (Section 1.4: Analysis of Algorithms).
- D. E. Knuth, The Art of Computer Programming, Vol. 1: Fundamental Algorithms, 3rd ed., 1997 (properties of algorithms; Euclid).
- J. Kleinberg and É. Tardos, Algorithm Design, Pearson, 2005 (Chapter 2: Basics of Algorithm Analysis).