Chapter 1: Introduction

Algorithms
What is an algorithm?Kinds of algorithmsTime complexityGrowth table Primality testMatrix multiplicationPermutations and FibonacciEuclid's algorithm
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Algorithms

Define an algorithm and its properties (input, output, definiteness, finiteness, effectiveness); write an everyday task as an algorithm.

Analysis

Express running time as a function T(n) of the input size n; read from a growth table how time changes as n grows.

Complexity examples

Count the operations in short code to find its complexity class (1, n, √n, n³, n!, 2ⁿ); see how memoization cuts exponential time.

Euclid's algorithm

Trace a gcd computation step by step; justify its correctness and its O(log min(a, b)) running time.

The Notion of an Algorithm

What Is an Algorithm?

  • A finite sequence of step-by-step instructions for solving a problem or carrying out a task.
  • It takes input, performs operations and produces output.
  • Every step must be precise and unambiguous, and the algorithm must terminate after finitely many steps.
InputZero or more values supplied from outside.
OutputAt least one result, related to the input.
DefinitenessEach step has exactly one meaning; nothing is left to interpretation.
FinitenessHalts after a finite number of steps on every input.
EffectivenessEach step is basic enough to actually be carried out.
CorrectnessProduces the correct output for every valid input.

The Notion of an Algorithm

An Everyday Example: Making Tea

Input: water, tea leaves, sugar (optional), milk (optional)
Output: tea

  1. Boil the water.
  2. Pour the boiling water into the teapot.
  3. Add the tea leaves to the pot.
  4. Let the tea brew.
  5. Strain the tea into a cup.
  6. Add sugar and milk if desired.
Order mattersSteps 2 and 3 can be swapped, but steps 1 and 5 cannot: an algorithm is an ordered list of steps.
Is it precise?"Let it brew" for how long? For a computer the step must read: wait 10 minutes.
DecisionsStep 6 contains a condition: "add sugar if wanted". Algorithms have decision points too.

The Notion of an Algorithm

Computing an Average: From Algorithm to Code

AlgorithmInput: x1, x2, x3 · Output: average
1. sum = x1 + x2 + x3
2. average = sum / 3
3. Print average.

Python

def average(x1, x2, x3):
    avg = (x1 + x2 + x3) / 3
    return avg

x1 = float(input("Enter the first number: "))
x2 = float(input("Enter the second number: "))
x3 = float(input("Enter the third number: "))
avg = average(x1, x2, x3)
print("Average of the numbers:", avg)

Java

public static void main(String[] args) {
  Scanner in = new Scanner(System.in);
  System.out.println("Enter the first number: ");
  double x1 = in.nextDouble();
  System.out.println("Enter the second number: ");
  double x2 = in.nextDouble();
  System.out.println("Enter the third number: ");
  double x3 = in.nextDouble();
  double avg = (x1 + x2 + x3) / 3;
  System.out.println("Average of the numbers: "
                     + avg);
}

The same algorithm can be written in any language: an algorithm is language-independent, and a program is one implementation of it.

The Notion of an Algorithm

Main Kinds of Algorithms

SortingPut items in order: insertion sort, merge sort, quicksort.
SearchingFind an item: linear and binary search, hashing.
GraphsShortest paths (Dijkstra), minimum spanning trees, maximum flow (Ford–Fulkerson).
Strings and compressionText search, tries; shrinking files: Huffman, gzip.
Design techniquesDivide and conquer (merge sort, binary search, fast exponentiation), dynamic programming, greedy, backtracking, branch and bound.
Randomized and approximateRandomized algorithms may take different paths on the same input; approximate solutions to hard problems.
A deterministic algorithm always takes the same steps and gives the same output on the same input (e.g. finding the maximum). A randomized algorithm flips coins, as when quicksort picks a random pivot.

Analysis of Algorithms

Time Complexity

  • Algorithm analysis measures the use of resources such as processor time and memory, so we can choose between algorithms that solve the same problem.
  • Running time is expressed in terms of the input size n: T(n). We count basic operations, not seconds: seconds depend on the machine and language, operation counts do not.

Example: summing an n-element array, counted line by line

int sum = 0;                      // once
for (int i = 0; i < n; i++)       // n+1 comparisons
  sum += arr[i];                  // n times
return sum;                       // once
Why does it matter?Searching a sorted array of 10⁶ items: linear search makes up to 1,000,000 comparisons, binary search at most 20.
n = 10⁶, worst-case comparisons (log scale)linear1,000,000binary20
T(n) = 1 + (n + 1) + n + 1 = 2n + 3. The constants depend on the machine; what matters is that T(n) grows linearly with n → Θ(n).

Analysis of Algorithms

Growth Table: As n Grows

f(n)n = 10n = 100n = 1000n = 10⁶time (n = 10⁶)
111111 ns
log₂ n3.36.610.019.919.9 ns
√n3.21031.61,0001 µs
n101001,0001,000,0001 ms
n log₂ n33.2664.49,9662.0 · 10⁷19.9 ms
n²10010,0001,000,0001.0 · 10¹²16.7 min
n³1,0001,000,0001.0 · 10⁹1.0 · 10¹⁸≈ 32 years
2ⁿ1,0241.3 · 10³⁰≈ 1.1 · 10³⁰¹≈ 10³⁰¹⁰³⁰> age of universe
n!3,628,8009.3 · 10¹⁵⁷≈ 10²⁵⁶⁷≈ 10⁵⁵⁶⁵⁷⁰⁸> age of universe
Last column: time for n = 10⁶ on a computer doing 10⁹ operations per second. A faster computer shrinks the constant; a better algorithm changes the growth rate. Replacing n² with n log n turns minutes into milliseconds at 10⁶ items.

Complexity Examples

Constant and Linear Time: O(1), O(n)

Printing the first element T(n) = O(1)

void printFirst(int[] arr) {
  if (arr.length > 0)
    System.out.println("First element: "
                     + arr[0]);
  else
    System.out.println("Array is empty");
}

Whether the array has 10 elements or 10 million, it does one comparison and one access: the time does not depend on n.

Finding the maximum T(n) = O(n)

Integer findMax(int[] arr) {
  if (arr.length == 0)
    return null;            // empty array
  int max = arr[0];
  for (int x : arr)
    if (x > max)
      max = x;
  return max;
}

Each element is examined once: n comparisons, O(1) extra memory. Example: [5, 8, 12, 4] → 12.

arrmax55881212412each element once: 4 elements → 4 steps

Complexity Examples

Primality Test: O(√n)

boolean isPrime(int n) {
  if (n <= 1)
    return false;
  else if (n == 2)
    return true;
  else if (n % 2 == 0)
    return false;
  int limit = (int) Math.sqrt(n) + 1;
  for (int i = 3; i < limit; i += 2)
    if (n % i == 0)
      return false;
  return true;
}
Why stop at √n?If n = a · b, one of a and b is at most √n. If nothing up to √n divides n, n is prime.
How many divisions?Even numbers are ruled out up front and only odd i are tried: at most ≈ √n / 2 divisions → T(n) = O(√n).

n = 97: i = 3, 5, 7, 9 → 4 divisions (95 if we tried 2 to 96).
n = 1,000,000,007: 15,810 divisions.

Note: here the input size is the value of n. If n has b bits, √n = 2b/2: the method is exponential in the number of bits.

Complexity Examples · Run the code step by step

isPrime(97): Step by Step

2—34—56—78—910—irem = 97 % ieven i (gray) are never tried

n = 97: greater than 1, not equal to 2, and odd. None of the first three checks returns; multiples of 2 are ruled out up front.

2—34—56—78—910—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

limit = (int) √97 + 1 = 9 + 1 = 10. Only odd numbers below 10 are tried: 3, 5, 7, 9.

2—314—56—78—910—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

i = 3: 97 % 3 = 1 ≠ 0 → 3 does not divide 97. i += 2.

2—314—526—78—910—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

i = 5: 97 % 5 = 2 ≠ 0 → 5 does not divide 97. i += 2.

2—314—526—768—910—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

i = 7: 97 % 7 = 6 ≠ 0 → 7 does not divide 97. i += 2.

2—314—526—768—9710—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

i = 9: 97 % 9 = 7 ≠ 0 → 9 does not divide 97. i += 2. Next candidate: i = 11.

2—314—526—768—9710—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

i = 11 < 10 fails and the loop ends. No odd number up to √97 divides 97.

2—314—526—768—9710—irem = 97 % ilimit = 10√97 ≈ 9.85even i (gray) are never tried

return true: 97 is prime. Only 4 divisions were needed; trying every number from 2 to 96 would take 95.

boolean isPrime(int n) {
  if (n <= 1)
    return false;
  else if (n == 2)
    return true;
  else if (n % 2 == 0)
    return false;
  int limit = (int) Math.sqrt(n) + 1;
  for (int i = 3; i < limit; i += 2)
    if (n % i == 0)
      return false;
  return true;
}

Complexity Examples

Cubic Time: Matrix Multiplication, Θ(n³)

int[][] multiply(int[][] A, int[][] B) {
  if (B.length == 0 || (A.length > 0 && A[0].length != B.length))
    throw new IllegalArgumentException("size");
  int n = A.length, m = B[0].length, p = B.length;
  int[][] result = new int[n][m];
  for (int i = 0; i < n; i++)
    for (int j = 0; j < m; j++) {
      int sum = 0;
      for (int k = 0; k < p; k++)
        sum += A[i][k] * B[k][j];
      result[i][j] = sum;
    }
  return result;
}

Input: A = [aᵢⱼ], B = [bᵢⱼ] (n × n). Output: C = [cᵢⱼ] = A · B, i, j = 1, 2, …, n.

Definitioncᵢⱼ = Σk=1..n aᵢₖ · bₖⱼ
12
34
·
56
78
=
1922
4350

c₁₁ = 1·5 + 2·7 = 19. Each cᵢⱼ takes n multiplications and there are n² entries → n³ multiplications: T(n) = Θ(n³).

Complexity Examples · Run the code step by step

multiply: 2 × 2 Matrices Step by Step

A1234·B5678=result

A and B are 2 × 2 with compatible sizes: n = m = p = 2. result starts as a 2 × 2 zero matrix.

A1234·B5678=resultsum = 1·5 = 5

i = 0, j = 0: sum = 0. k = 0: add A[0][0] · B[0][0] = 1 · 5 = 5.

A1234·B5678=result19sum = 5 + 2·7 = 19

k = 1: A[0][1] · B[1][0] = 2 · 7 → sum = 5 + 14 = 19. Inner loop ends: result[0][0] = 19.

A1234·B5678=result19sum = 1·6 = 6

i = 0, j = 1: sum = 0. k = 0: add A[0][0] · B[0][1] = 1 · 6 = 6.

A1234·B5678=result1922sum = 6 + 2·8 = 22

k = 1: A[0][1] · B[1][1] = 2 · 8 → sum = 6 + 16 = 22. Inner loop ends: result[0][1] = 22.

A1234·B5678=result1922sum = 3·5 = 15

i = 1, j = 0: sum = 0. k = 0: add A[1][0] · B[0][0] = 3 · 5 = 15.

A1234·B5678=result192243sum = 15 + 4·7 = 43

k = 1: A[1][1] · B[1][0] = 4 · 7 → sum = 15 + 28 = 43. Inner loop ends: result[1][0] = 43.

A1234·B5678=result192243sum = 3·6 = 18

i = 1, j = 1: sum = 0. k = 0: add A[1][0] · B[0][1] = 3 · 6 = 18.

A1234·B5678=result19224350sum = 18 + 4·8 = 50

k = 1: A[1][1] · B[1][1] = 4 · 8 → sum = 18 + 32 = 50. Inner loop ends: result[1][1] = 50.

A1234·B5678=result19224350

Returns result = [[19, 22], [43, 50]]. 8 = 2³ multiplications: n multiplications for each of the n² entries → n³.

int[][] multiply(int[][] A, int[][] B) {
  if (B.length == 0 || (A.length > 0 && A[0].length != B.length))
    throw new IllegalArgumentException("size");
  int n = A.length, m = B[0].length, p = B.length;
  int[][] result = new int[n][m];
  for (int i = 0; i < n; i++)
    for (int j = 0; j < m; j++) {
      int sum = 0;
      for (int k = 0; k < p; k++)
        sum += A[i][k] * B[k][j];
      result[i][j] = sum;
    }
  return result;
}

Complexity Examples

Factorial Time: Generating All Permutations

void permute(int[] arr, int k) {
  if (k == arr.length) {           // one permutation done
    System.out.println(Arrays.toString(arr));
    return;
  }
  for (int i = k; i < arr.length; i++) {
    swap(arr, k, i);                // put element i at k
    permute(arr, k + 1);            // arrange the rest
    swap(arr, k, i);                // undo the swap
  }
}

[1, 2, 3] → 123, 132, 213, 231, 321, 312. There are n choices for the first position, n−1 for the second, …: n · (n−1) · … · 1 = n! permutations.

nn!
36
5120
103,628,800
151.3 · 10¹²
202.4 · 10¹⁸

Printing each permutation takes n steps: Θ(n · n!) in total, the "O(n!) class" for short. Even n = 20 is out of reach in practice.

Complexity Examples

Exponential Time: Fibonacci Numbers

long fib(int n) {           // naive recursion
  if (n <= 1) return n;
  return fib(n - 1) + fib(n - 2);
}

static final int N = 92;            // largest n for long
long[] memo = new long[N + 1];   // 0: not set
long fibMemo(int n) {       // memoization
  if (n <= 1) return n;
  if (memo[n] == 0)
    memo[n] = fibMemo(n - 1) + fibMemo(n - 2);
  return memo[n];
}

F(0) = 0, F(1) = 1, F(n) = F(n−1) + F(n−2).
fib: each call spawns two calls → O(2ⁿ) (exactly Θ(1.618ⁿ)); fib(30) makes 2,692,537 calls.
fibMemo: each F(k) is computed once, 2n − 1 calls → O(n); 59 calls for n = 30.
F(92) is the largest value that fits in a long; beyond that use BigInteger.

f5f4f3f2f1f0f1f2f1f0f3f2f1f0f1

Naive recursion: 15 calls. f3 is computed twice, f2 three times, f1 five times; the tree roughly doubles at each level → exponential growth.

f5f4f3f2f1f0f1f2f3

Memoized: 9 calls. The dashed orange nodes (f2, f3) come straight from the memo array; their subtrees are never expanded → O(n).

Euclid's Algorithm

Euclid's Algorithm (gcd)

  • Finds the greatest common divisor (gcd).
  • One of the oldest known algorithms (Euclid, c. 300 BC) and the simplest way to compute a gcd.
  • Key idea: gcd(a, b) = gcd(b, a mod b), gcd(a, 0) = a.
  1. Step 1: divide the larger number by the smaller and take the remainder.
  2. Step 2: repeat with the pair (divisor, remainder) until the remainder is zero.
  3. Step 3: when the remainder is zero, the last divisor is the gcd.
int gcd(int a, int b) {
  if (b == 0)
    return a;
  return gcd(b, a % b);
}
Why is it correct?If a = q·b + r, every common divisor of a and b also divides r = a − q·b, and every common divisor of b and r divides a. The common divisors are the same.

Euclid's Algorithm · Run the code step by step

gcd(1599, 650): Step by Step

6506502991599 × 650 rectangle

1599 = 2 · 650 + 299. Cut 2 squares from the 1599×650 rectangle; the leftover strip is 650×299 → gcd(650, 299).

29929952650 × 299 rectangle

650 = 2 · 299 + 52. Cut 2 squares from the 650×299 rectangle; the leftover strip is 299×52 → gcd(299, 52).

525252525239299 × 52 rectangle

299 = 5 · 52 + 39. Cut 5 squares from the 299×52 rectangle; the leftover strip is 52×39 → gcd(52, 39).

391352 × 39 rectangle

52 = 1 · 39 + 13. Cut 1 square from the 52×39 rectangle; the leftover strip is 39×13 → gcd(39, 13).

13131339 × 13 rectangle

39 = 3 · 13 + 0. Cut 3 squares from the 39×13 rectangle; the leftover strip is 13×0 → gcd(13, 0).

gcd(13, 0) = 13

b == 0 → return 13. 13 is the largest number that divides both 1599 and 650.

6506502991599 × 650 rectangle

The value 13 is passed back unchanged through every call: gcd(1599, 650) = 13. Five divisions suffice, and 13 × 13 squares tile the 1599 × 650 rectangle exactly.

int gcd(int a, int b) {
  if (b == 0)
    return a;
  return gcd(b, a % b);
}

Euclid's Algorithm

Euclid's Algorithm: Why Is It Fast?

  • It is efficient: every two steps the larger number drops to less than half (a mod b < a/2).
  • So the number of steps is at most ≈ 2 log₂ min(a, b): O(log min(a, b)).
  • The worst case is consecutive Fibonacci numbers: every quotient is q = 1, so the numbers shrink as slowly as possible (Lamé's theorem).
  • For comparison: trying every divisor from 1 to min(a, b) takes O(min(a, b)).
Worst case: gcd(55, 34)55 = 1 · 34 + 21
34 = 1 · 21 + 13
21 = 1 · 13 + 8
13 = 1 · 8 + 5
8 = 1 · 5 + 3
5 = 1 · 3 + 2
3 = 1 · 2 + 1
2 = 2 · 1 + 0

8 divisions, gcd = 1. gcd(1599, 650) needs only 5.

Summary

Summary

TopicKey points
Algorithmsinput → precise, finite, effective steps → output; language-independent
AnalysisT(n): number of basic operations as a function of input size; time and memory
Growth1 < log n < √n < n < n log n < n² < n³ < 2ⁿ < n!; a better algorithm changes the growth rate
Examplesfirst element O(1), maximum O(n), primality O(√n), matrix multiplication Θ(n³), permutations O(n!), naive Fibonacci O(2ⁿ)
Memoizationstoring repeated subproblems takes Fibonacci from O(2ⁿ) down to O(n)
Euclidgcd(a, b) = gcd(b, a mod b); O(log min(a, b)); worst case: consecutive Fibonacci numbers

Summary

Common Mistakes

Measuring time in secondsSeconds depend on the machine; analysis uses operation counts as a function of n and their growth rate.
Forgetting memoizationNaive recursive Fibonacci is O(2ⁿ); storing repeated subproblems brings it down to O(n).
Getting the √n bound wrongWith i < (int) Math.sqrt(n), √n itself is never tried: squares such as 9, 25, 49 come out prime.
Confusing the input sizeIn primality testing n is the value of the number; for a b-bit number √n = 2b/2, i.e. exponential in the bit length.
Thinking matrix multiplication is n²The result has n² entries, but each needs n multiplications: n³ in total.
Not undoing the swapForget the second swap and the array gets scrambled: for [1, 2, 3] some permutations repeat and others never appear.

Summary

Sample Problems

  1. For isPrime(221), which values of i does the loop try, how many divisions are made, and what is the result?
  2. Give the Θ class of T(n) = 5n² + 100n log₂ n + 7. Which term dominates at n = 1000?
  3. How many multiplications does multiply perform on two 3 × 3 matrices? For n = 100?
  4. How many lines does permute([1, 2, 3, 4], 0) print? What are the first two?
  5. Find gcd(252, 105) with Euclid's algorithm.
Show answers
  1. (int) √221 = 14 → limit = 15; i = 3, 5, 7, 9, 11, 13. 221 % 13 = 0 → false after 6 divisions (221 = 13 · 17).
  2. Θ(n²). At n = 1000: 5n² = 5,000,000 while 100n log₂ n ≈ 996,578 → 5n² dominates.
  3. 3 multiplications for each of the 9 entries cᵢⱼ → 27 (n³). For n = 100 → 10⁶ multiplications.
  4. 4! = 24 lines; the first two are [1, 2, 3, 4] and [1, 2, 4, 3].
  5. 252 = 2·105 + 42; 105 = 2·42 + 21; 42 = 2·21 + 0 → gcd = 21.

Summary

Exercises

  1. Write an algorithm that counts how many times a given number occurs in an array; give its time and space complexity.
  2. Nested loops: how many times does for i in 0..n-1: for j in i..n-1: op run? What is its class?
  3. Modify findMax to return both the minimum and the maximum of the array. How many comparisons does it make?
  4. Write Euclid's algorithm with a loop (no recursion).
  5. At 10⁹ operations per second, find the largest n solvable in 1 second for f(n) = n² and f(n) = 2ⁿ.
  6. How many times does the loop while (n > 1) n = n / 2; run?
Answers / hints
  1. One pass with a counter: Θ(n) time, O(1) extra memory.
  2. n + (n−1) + … + 1 = n(n+1)/2 → Θ(n²).
  3. Comparing each element twice gives 2(n − 1); processing elements in pairs (compare the pair, then the smaller with the min and the larger with the max) gives ≈ 3n/2. Both are Θ(n).
  4. while (b != 0) { int r = a % b; a = b; b = r; } return a;
  5. n² ≤ 10⁹ → n ≈ 31,622; 2ⁿ ≤ 10⁹ → n = 29.
  6. ⌊log₂ n⌋ times → Θ(log n).

Summary

References

  • Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., MIT Press, 2022 (Chapters 1–3).
  • Sedgewick and Wayne, Algorithms, 4th ed., Addison-Wesley, 2011 (Section 1.4: Analysis of Algorithms).
  • D. E. Knuth, The Art of Computer Programming, Vol. 1: Fundamental Algorithms, 3rd ed., 1997 (properties of algorithms; Euclid).
  • J. Kleinberg and É. Tardos, Algorithm Design, Pearson, 2005 (Chapter 2: Basics of Algorithm Analysis).

The End

Chapter 1: Introduction
Next: Chapter 2: Algorithm Complexity
1 / 1 Sercan KÜLCÜ. All rights reserved.