Chapter 1: Introduction and Algorithm Analysis

Data Structures and Algorithms with C
System life cycleAlgorithms and ADTsSelection sortSpace and timeO, Ω, ΘMagic squareMeasuring time
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Process

Describe the five phases of the system life cycle and where data structure decisions are made.

Abstraction

Define algorithms and abstract data types (ADTs); build an interface in C with a header file.

Counting

Work out the space requirement and step count of a C function (sum, rsum, add).

Classifying

Express step counts with O, Ω and Θ; separate the best, average and worst cases.

Comparing

Explain how the common complexity classes pull apart as n grows.

Measuring

Time a program with clock() and recognize the pitfalls (clock ticks, output).

Course Introduction · not in the books!

Textbooks and Resources

Desk reference

Kernighan and Ritchie, The C Programming Language. Written by C’s designers; keep it within reach at all times.

Textbook

Horowitz, Sahni and Anderson-Freed, Fundamentals of Data Structures in C.

Fundamentals ofData StructuresSPARKSFund. of DataStructures in Cour textbookFund. of DataStructures in C++most complete

Three editions of one book: almost the same content, only the language of the code changes.

Further reading: Cormen et al. (CLRS), Introduction to Algorithms; Sedgewick, Algorithms in C.

Course Introduction · Reading

Reading: Loving Your Work

Archaeologist

Lives today but thinks 5500 years back: “The pharaoh’s wife can’t have been buried there for that reason.”

Astronomer

Photographs for years to prove a black hole exists; thrilled when a star moves one pixel.

Football fan

8 hours on the road, a 150 TL ticket, stones and sticks… Yet we call someone who travels as far for a conference a “fool”.

From the instructor“Love your profession. People listen to someone who loves their work, never to someone who just goes through the motions. With code too, understand what happens as if you lived inside it.”

“The way to a lifetime of happiness is to love your work.” — Chinese proverb

Foundations

The System Life Cycle

Software goes through a process with the following phases:

RequirementsAnalysisDesignRefinementand codingVerificationif errors are found, revisit design / coding decisions

Requirements: A project starts by specifying its purpose: which inputs are given, which outputs must be produced?

RequirementsAnalysisDesignRefinementand codingVerificationif errors are found, revisit design / coding decisions

Analysis: The problem is broken into pieces. There are two approaches: bottom-up or top-down.

RequirementsAnalysisDesignRefinementand codingVerificationif errors are found, revisit design / coding decisions

Design: Define the data objects (→ ADTs) and operations (→ algorithms); both are language-independent. More time spent on design means fewer errors.

RequirementsAnalysisDesignRefinementand codingVerificationif errors are found, revisit design / coding decisions

Refinement and coding: Algorithms are coded on the chosen data structures. The data representation determines how efficient the algorithm is.

RequirementsAnalysisDesignRefinementand codingVerificationif errors are found, revisit design / coding decisions

Verification: Correctness proofs, testing on varied data, and debugging. Each is a research area in its own right.

Horowitz–Sahni, Chapter 1

Foundations

Analysis: Bottom-Up or Top-Down?

Bottom-up

  • Split the problem into pieces and focus on each one separately.
  • Modules are visible right away; the problem is solved module by module.
  • Preferred when we are experienced in the domain.

Top-down

  • Start with a master plan: “What is the goal?”, “What is the final product?”
  • Divide the problem into subproblems; this takes more time.
  • Less backtracking and fewer errors; preferred when we lack experience.
In the design phase we decide which fields a student record has, but we can defer whether to implement it with an array, a linked list or a tree. Deferring lets us pick a more efficient implementation later.

Foundations

Coding and Verification

Correctness proofs

Mathematical techniques prove a program correct; proving a large project end to end is nearly impossible. Using proven algorithms reduces errors.

Testing

Needs working code and test data covering all scenarios. A program that runs without syntax errors is not necessarily correct.

Error removal

When proofs or tests reveal errors, debugging retraces design and coding decisions.

Running time

A correct but slow program is of little value: correctness and efficiency are separate questions.

From the instructor“The longer you delay coding, the better the method you can work out.”

Foundations

What Is an Algorithm?

An algorithm is a finite sequence of instructions that accomplishes a particular task. Every algorithm must satisfy these criteria:

CriterionMeaning
InputZero or more quantities are supplied externally.
OutputAt least one quantity is produced.
DefinitenessEach instruction is clear and unambiguous: no “add 6 or 7 to x”.
FinitenessIn every case, it terminates after a finite number of steps.
EffectivenessEach instruction is basic enough to be carried out with pencil and paper.
A program that never terminates (e.g. an operating system) is not an algorithm but a computational procedure. We can describe an algorithm in natural language, as a flowchart or in C; we will use C.

Horowitz–Sahni, Chapter 1

Foundations

Abstract Data Types

Data type = objects + operations. An ADT is a data type whose specification of objects and operations is separated from their representation and implementation.

Specification: WHAT?

Operation names, argument types, result type and what each operation does. This is all the user knows.

Implementation: HOW?

How the data is laid out in memory (struct, array…) and the code for the operations. Hidden from the user.

ADT Fraction
  objects: num/den, den ≠ 0
  operations:
    create(p, q)   → Fraction /* constructor */
    add(a, b)      → Fraction /* transformer */
    equal(a, b)    → Boolean  /* observer */
    print(a)                  /* reporter */
    destroy(a)
A constructor creates a new instance, a transformer builds a new value from existing ones, an observer/reporter returns information. The same ADT can be implemented with different data structures.

Horowitz–Sahni, Section 1.3

Foundations

Abstraction in C: Header File + struct

/* fraction.h : interface (WHAT?) */
#ifndef FRACTION_H
#define FRACTION_H
typedef struct fraction Fraction; /* opaque */

Fraction *fraction_create(int num, int den);
Fraction *fraction_add(const Fraction *a,
                       const Fraction *b);
int  fraction_equal(const Fraction *a,
                    const Fraction *b);
void fraction_print(const Fraction *f);
void fraction_free(Fraction *f);
#endif
/* fraction.c : implementation (HOW?) */
#include <stdio.h>
#include <stdlib.h>
#include "fraction.h"

struct fraction { int num, den; };

Fraction *fraction_create(int num, int den) {
  Fraction *f;
  if (den == 0) return NULL;
  f = malloc(sizeof *f);
  if (f == NULL) return NULL;
  f->num = num;
  f->den = den;
  return f;
}
typedef struct fraction Fraction; declares an incomplete (opaque) type: a file that includes fraction.h cannot see the fields and only holds a Fraction *. The fields stay hidden in fraction.c → C’s counterpart of an ADT.

Foundations

The Rest of the Implementation, and Its Use

Fraction *fraction_add(const Fraction *a,
                       const Fraction *b) {
  return fraction_create(
      a->num * b->den
        + b->num * a->den,
      a->den * b->den);
}
int fraction_equal(const Fraction *a,
                   const Fraction *b) {
  return a->num * b->den
      == b->num * a->den;
}
void fraction_print(const Fraction *f) {
  printf("%d/%d\n", f->num, f->den);
}
void fraction_free(Fraction *f) { free(f); }
/* main.c : knows only the interface */
#include "fraction.h"
#include <stddef.h> /* NULL */
int main(void) {
  Fraction *a = fraction_create(1, 2);
  Fraction *b = fraction_create(1, 3);
  Fraction *s = NULL;
  if (a && b) s = fraction_add(a, b);
  if (s) fraction_print(s);  /* 5/6 */
  fraction_free(a); fraction_free(b);
  fraction_free(s);  /* free(NULL) is safe */
  return 0;
}
Contract

fraction_create returns NULL if the denominator is 0 or memory runs out; the caller checks.

Ownership

Every result of create/add belongs to the caller and is released with fraction_free.

Selection Sort

Selection Sort

We want to sort an array into ascending order. Each pass finds the smallest of the remaining elements and moves it to the front (position i):

for (i = 0; i < n; i++) {
    examine list[i] to list[n-1];
    let list[min] be the smallest integer;
    interchange list[i] and list[min];
}
010113229337414sorted partunsorted part▲ i▲ minswap
From the instructor“Aim for the level where you can find fault with published algorithms and offer better ones.”

Selection Sort

Selection Sort in C

#include <stdio.h>
#include <stdlib.h>
#define MAX_SIZE 101
#define SWAP(x, y, t) ((t) = (x), (x) = (y), (y) = (t))
void sort(int list[], int n);

int main(void) {
  int i, n, list[MAX_SIZE];
  printf("Enter the array size: ");
  if (scanf("%d", &n) != 1 || n < 1 || n > MAX_SIZE) {
    printf("Invalid value of n!\n");
    return 1;
  }
  for (i = 0; i < n; i++) list[i] = rand() % 100;
  sort(list, n);
  for (i = 0; i < n; i++) printf("%d ", list[i]);
  return 0;
}
void sort(int list[], int n) {
  int i, j, min, temp;
  for (i = 0; i < n - 1; i++) {
    min = i;
    for (j = i + 1; j < n; j++)
      if (list[j] < list[min])
        min = j;
    SWAP(list[i], list[min], temp);
  }
}
Comparisons: (n−1) + (n−2) + … + 1 = n(n−1)/2 → O(n²). At most n−1 swaps.

Selection Sort · A common mistake

Where Does a Common Faulty sort() Go Wrong?

void sort(int list[], int n) {
  int i, j, min, temp;
  for (i = 0; i < n; i++) {
    min = i;
    for (j = 0; j < n; j++)
      if (list[j] > list[min]) {
        min = j;
        SWAP(list[i], list[min], temp);
      }
  }
}
InputVersionComp.Swaps*
{29,10,14,37,13}faulty257
selection103
100 randomfaulty100002561
selection495094
500…1 (descending)faulty250000124750
selection124750250
The faulty version on the left still sorts the array (checked on all permutations for n ≤ 7), but it is not the selection sort of the pseudocode: the inner loop starts at j = 0 and the swap sits inside the loop. The correct version (j = i + 1, swap after the inner loop) is the one used in this chapter. *Swaps: SWAPs that exchange two distinct cells (i ≠ min).

Selection Sort

SWAP: Macro or Function?

Macro

#define SWAP(x, y, t) \
        ((t) = (x), (x) = (y), (y) = (t))

int temp;
SWAP(list[i], list[min], temp);
  • No call overhead: the preprocessor substitutes the text.
  • The temporary t is supplied by the caller.

Function

void swap(int *x, int *y) {
  int temp = *x;
  *x = *y;
  *y = temp;
}

swap(&list[i], &list[min]);
  • Type-checked; temp is local.
  • We must pass the addresses of the variables.
A macro may evaluate its arguments more than once: SWAP(a[i++], a[j], t) increments i twice! A function named SWAP would clash with the macro, so the function is called swap.

Selection Sort · Run the code step by step

Code Trace: swap(&a, &b)

main() framea3b8

In main’s frame, a = 3 and b = 8.

main() framea3b8swap() framexytemp?

The call copies addresses, not values. x → a, y → b.

main() framea3b8swap() framexytemp3

temp = *x: read the value x points to (3).

main() framea8b8swap() framexytemp3

*x = *y: the 8 that y points to is written where x points, i.e. into a.

main() framea8b3swap() framexytemp3

*y = temp: 3 goes into b. The swap happened in main’s variables.

main() framea8b3

swap’s frame is gone (x, y, temp), but its effect remains.

void swap(int *x, int *y) {
  int temp = *x;
  *x = *y;
  *y = temp;
}

int main(void) {
  int a = 3, b = 8;
  swap(&a, &b);
  printf("%d %d\n", a, b);
}

Performance Analysis

How Do We Judge a Program?

  1. Does it meet the specification?
  2. Does it work correctly?
  3. Is it documented (with comments) so others can use it?
  4. Does it use functions effectively to produce logical units?
  5. Is the code readable?
  6. Does it use primary and secondary memory efficiently?
  7. Is its running time acceptable?
1 · Performance analysis

Machine-independent: estimates of time and space (complexity).

2 · Performance measurement

Machine-dependent: the actual running time is measured.

The last two questions are the subject of performance evaluation; the first five matter at least as much.

Performance Analysis

Space Complexity

The amount of memory a program needs to run to completion: S(P) = c + SP(I)

Fixed part c

Independent of input/output size: the code itself, simple variables, constants.

Variable part SP(I)

Depends on characteristics of input I (usually its size). This is what we analyze.

float abc(float a, float b, float c) {
  return a + b + b * c
         + (a + b - c) / (a + b) + 4.00;
}
/* fixed space only: S_abc(I) = 0 */
From the instructor“Write code as if every resource were scarce. Plenty of memory is no excuse for wasting it.”

Performance Analysis

sum and rsum: Variable Space

float sum(float list[], int n) {
  float tempsum = 0;
  int i;
  for (i = 0; i < n; i++)
    tempsum += list[i];
  return tempsum;
}

float rsum(float list[], int n) {
  if (n)
    return rsum(list, n - 1) + list[n - 1];
  return 0;
}
  • sum receives the address of the first element → Ssum(n) = 0. If the elements were copied by value, Ssum(n) would be n.
  • rsum(list, n) is called n+1 times in all; each call pushes a stack frame → Srsum(n) ∝ n.
system stackrsum(n=3)list · n · return addr.rsum(n=2)list · n · return addr.rsum(n=1)list · n · return addr.rsum(n=0)list · n · return addr.One frame per call:on a 16-bit machine 2 + 2 + 2 = 6 bytesn = 1000 → at least 6000 byteson 64-bit a frame is typically ≥ 24 bytes

Performance Analysis

Time Complexity

T(P) = compile time + run time. A program is compiled once; we care about the run time tP.

tP(n) = ca·ADD(n) + cs·SUB(n) + cl·LDA(n) + cst·STA(n)
Constants c

The time of each operation depends on the compiler and the machine; hard to know in advance.

Step count

Instead we count program steps: a meaningful, machine-independent unit.

From the instructor“Why does performance matter so much? Suppose you are programming a fighter jet’s bombing system. If yours releases in 3 s and the enemy’s in 2 s, the battle is lost before it starts.”

Performance Analysis

Step Count Table: sum()

Statements/eFrequencyTotal
float sum(float list[], int n) {000
float tempsum = 0;111
int i;000
for (i = 0; i < n; i++)1n + 1n + 1
tempsum += list[i];1nn
return tempsum;111
Total2n + 3
s/e (steps per execution): how many steps one execution of the statement costs. The for test runs n+1 times: n times “continue”, once “done”. Declarations count as zero steps.

Performance Analysis · Run the code step by step

Code Trace: sum(list, 3) with count

031124

Global count = 0. It goes up by one at every program step.

031124

The assignment is one step → count = 1.

031124▲ i

Loop test 0 < 3 is true → one step (count = 2).

031124▲ i

tempsum += list[0] → 3; count = 3.

031124▲ i

Loop test 1 < 3 is true → one step (count = 4).

031124▲ i

tempsum += list[1] → 4; count = 5.

031124▲ i

Loop test 2 < 3 is true → one step (count = 6).

031124▲ i

tempsum += list[2] → 8; count = 7.

031124

The final test 3 < 3 is false; it is a step too → count = 8.

031124

return: count = 9 = 2·3 + 3, matching 2n + 3 from the table.

float sum(float list[], int n) {
  float tempsum = 0; count++;
  int i;
  for (i = 0; i < n; i++) {
    count++;             /* loop test */
    tempsum += list[i]; count++;
  }
  count++;  /* last loop test */
  count++;  return tempsum;
}

Performance Analysis

Step Count Table: rsum()

Statements/eFrequencyTotal
float rsum(float list[], int n) {000
if (n)1n + 1n + 1
return rsum(list, n-1) + list[n-1];1nn
return 0;111
Total2n + 2
Step counts can mislead

rsum (2n+2) seems to take “fewer steps” than sum (2n+3), yet each recursive call (frame setup, return) costs more than one loop iteration.

Both are O(n)

What matters is how the step count grows as n grows: both are linear.

Performance Analysis · Run the code step by step

Code Trace: rsum(list, 3) and the call stack

031124listrsum(list, 3)stack (bottom up)

rsum(list, 3): if test (count = 1); n = 3 ≠ 0.

031124listrsum(list, 3)stack (bottom up)

count = 2; rsum(list, 2) must be computed first → new frame.

031124listrsum(list, 3)rsum(list, 2)stack (bottom up)

rsum(list, 2): if test (count = 3); n = 2 ≠ 0.

031124listrsum(list, 3)rsum(list, 2)stack (bottom up)

count = 4; rsum(list, 1) must be computed first → new frame.

031124listrsum(list, 3)rsum(list, 2)rsum(list, 1)stack (bottom up)

rsum(list, 1): if test (count = 5); n = 1 ≠ 0.

031124listrsum(list, 3)rsum(list, 2)rsum(list, 1)stack (bottom up)

count = 6; rsum(list, 0) must be computed first → new frame.

031124listrsum(list, 3)rsum(list, 2)rsum(list, 1)rsum(list, 0)stack (bottom up)

rsum(list, 0): if test (count = 7); n = 0 → base case.

031124listrsum(list, 3)rsum(list, 2)rsum(list, 1)rsum(list, 0)→ 0stack (bottom up)

count = 8; return 0. The stack is at its deepest: 4 frames.

031124listrsum(list, 3)rsum(list, 2)rsum(list, 1)→ 3stack (bottom up)

Add list[0] = 3 to the returned value → rsum(list, 1) = 3.

031124listrsum(list, 3)rsum(list, 2)→ 4stack (bottom up)

Add list[1] = 1 to the returned value → rsum(list, 2) = 4.

031124listrsum(list, 3)→ 8stack (bottom up)

Add list[2] = 4 to the returned value → rsum(list, 3) = 8. count = 8 = 2·3 + 2.

float rsum(float list[], int n) {
  count++;               /* if test */
  if (n) {
    count++;             /* add + return */
    return rsum(list, n - 1) + list[n - 1];
  }
  count++;
  return 0;
}

Performance Analysis

Step Count: Matrix Addition add()

StatementFrequency
void add(int a[][MAX_SIZE], int b[][MAX_SIZE],
  int c[][MAX_SIZE], int rows, int cols) {
0
int i, j;0
for (i = 0; i < rows; i++)rows + 1
for (j = 0; j < cols; j++)rows · (cols + 1)
c[i][j] = a[i][j] + b[i][j];rows · cols
Total2·rows·cols + 2·rows + 1
The result is O(rows·cols); for a square matrix (rows = cols = n) it is O(n²). Looping over the smaller dimension outside shrinks the 2·rows term: with rows ≫ cols, swapping the loops gives 2·rows·cols + 2·cols + 1. Matrix multiplication is O(n³).

Writing “O(rows, cols)” is wrong; the product is correct: O(rows·cols).

Performance Analysis

Best, Average and Worst Case

For the same n, the step count may also depend on the input itself. In linear search:

071329354158▲ best: 1 comp.worst: n comp. ▲
CaseWhen?Comparisons
BestKey is in the first cell1 → Θ(1)
AverageKey present, every position equally likely(n + 1) / 2 → Θ(n)
WorstKey in the last cell or absentn → Θ(n)
We usually report the worst case: it is a guaranteed upper bound for every input. The average case requires knowing the probability distribution of the inputs.

CLRS Section 2.2

Asymptotic Notation

Big-O Notation

Definition: f(n) = O(g(n)) if there exist constants c > 0 and n0 such that f(n) ≤ c·g(n) for all n ≥ n0: “the algorithm has complexity of order g(n)”.

  • Tsum(n) = 2n + 3 = O(n)
  • Trsum(n) = 2n + 2 = O(n)
  • Tadd = 2·rows·cols + 2·rows + 1 = O(rows·cols) → O(n²)
nstepsn₀c·g(n)f(n)f(n) ≤ c·g(n) for n ≥ n₀
From the instructor“Time matters more than memory!”

O is an upper bound; here “=” means “is a member of” and reads one way only.

Asymptotic Notation

Example: f(n) = 3n³ + 6n² + 7n = O(n³)

3n³ + 6n² + 7n ≤ 3n³ + 6n² + n² (7n ≤ n² for n ≥ 7)3n³ + 6n² + 7n ≤ 3n³ + 7n²3n³ + 6n² + 7n ≤ 3n³ + n³ (7n² ≤ n³ for n ≥ 7)3n³ + 6n² + 7n ≤ 4n³ ⇒ f(n) = O(n³) (c = 4, n₀ = 7)

Bound the lower-order term 7n by n²: 7n ≤ n² whenever n ≥ 7.

3n³ + 6n² + 7n ≤ 3n³ + 6n² + n² (7n ≤ n² for n ≥ 7)3n³ + 6n² + 7n ≤ 3n³ + 7n²3n³ + 6n² + 7n ≤ 3n³ + n³ (7n² ≤ n³ for n ≥ 7)3n³ + 6n² + 7n ≤ 4n³ ⇒ f(n) = O(n³) (c = 4, n₀ = 7)

6n² + n² = 7n².

3n³ + 6n² + 7n ≤ 3n³ + 6n² + n² (7n ≤ n² for n ≥ 7)3n³ + 6n² + 7n ≤ 3n³ + 7n²3n³ + 6n² + 7n ≤ 3n³ + n³ (7n² ≤ n³ for n ≥ 7)3n³ + 6n² + 7n ≤ 4n³ ⇒ f(n) = O(n³) (c = 4, n₀ = 7)

Same trick once more: 7n² ≤ n³ for n ≥ 7.

3n³ + 6n² + 7n ≤ 3n³ + 6n² + n² (7n ≤ n² for n ≥ 7)3n³ + 6n² + 7n ≤ 3n³ + 7n²3n³ + 6n² + 7n ≤ 3n³ + n³ (7n² ≤ n³ for n ≥ 7)3n³ + 6n² + 7n ≤ 4n³ ⇒ f(n) = O(n³) (c = 4, n₀ = 7)

We found c = 4 and n₀ = 7: the definition holds.

1 · Upper bound

The running time never exceeds 4n³ steps (n ≥ 7).

2 · Cubic growth

If sorting 100 numbers takes 2 s, sorting 300 takes at most 2·3³ = 54 s (not 6 s!).

Asymptotic Notation

Lower Bound Ω and Tight Bound Θ

NotationDefinition (for all n ≥ n₀)Meaning
f(n) = O(g(n))f(n) ≤ c·g(n)g is an upper bound on f
f(n) = Ω(g(n))f(n) ≥ c·g(n)g is a lower bound on f
f(n) = Θ(g(n))c₁·g(n) ≤ f(n) ≤ c₂·g(n)both O and Ω: a tight bound
  • 3n + 2 = Θ(n): 3n ≤ 3n + 2 ≤ 4n for n ≥ 2.
  • 3n + 2 = O(n²) is also true but says little; we write the smallest g(n) we can.
  • Selection sort makes n(n−1)/2 comparisons on every input → Θ(n²).
From the instructor“An exam answer solved the problem in O(n), then added an empty loop running m times, because the question said ‘solve it in O(n+m)’. No need. If O(n+m) is asked and you can do better, all the better!”

Asymptotic Notation

Practical Complexities

ComplexityNamen=1n=2n=4n=8n=16n=32
1Constant111111
lg nLogarithmic012345
nLinear12481632
n·lg nLinearithmic0282464160
n²Quadratic1416642561,024
n³Cubic18645124,09632,768
2ⁿExponential241625665,5364.29·109
n!Factorial122440,3202.09·10132.63·1035
lg n = log₂ n. Small values of n are misleading: at n = 4, 2ⁿ = n²; at n = 8, 2ⁿ = 256 > n² = 64. Beyond some point the exponential and factorial terms crush everything else.

Asymptotic Notation

As n Grows: Growth Chart

050100150123456789101112n →lg nnn·lg nn²n³2ⁿ

Vertical scale 0–150: lg n, n and n·lg n separate; n² climbs fast, while n³ and 2ⁿ are already off the chart.

01000200030004000123456789101112n →n·lg nn²n³2ⁿn!

Scale 0–4500: n³ = 1728 (n = 12), 2ⁿ = 4096 (n = 12); n! leaves the chart at n = 7 with 5040. The linear functions seem glued to the axis.

Asymptotic Notation

Running Times on a Machine Executing 10⁹ Instructions per Second

nnlg nn·lg nn²n³n⁴n¹⁰2ⁿ
100.01 μs0.00332 μs0.0332 μs0.1 μs1 μs10.0 μs10.0 s1.02 μs
200.02 μs0.00432 μs0.0864 μs0.4 μs8 μs160 μs2.84 h1.05 ms
300.03 μs0.00491 μs0.147 μs0.9 μs27.0 μs810 μs6.83 days1.07 s
400.04 μs0.00532 μs0.213 μs1.6 μs64.0 μs2.56 ms4.05 mo18.3 min
500.05 μs0.00564 μs0.282 μs2.5 μs125 μs6.25 ms3.1 yr13.0 days
1000.1 μs0.00664 μs0.664 μs10.0 μs1 ms100 ms31.7 cent.4.02·10¹¹ cent.
10001 μs0.00997 μs9.97 μs1 ms1 s16.7 min3.17·10¹¹ cent.3.40·10²⁸² cent.
1000010.0 μs0.0133 μs133 μs100 ms16.7 min3.86 mo3.17·10²¹ cent.6.33·10²⁹⁹¹ cent.

Each step takes 1 ns; 1 year = 365 days, 1 month = 30 days; cent. = centuries. For n = 10000, 2ⁿ is about 102992 centuries.

Asymptotic Notation

A First Look at Amortized Analysis

Some operations are occasionally expensive but usually cheap. Amortized analysis divides the total cost of a sequence of n operations by n. Example: an array whose capacity doubles when full (realloc); we trace it and count its copies in Chapter 2.

Over n insertions the copies are 1 + 2 + 4 + … < 2n → total < 3n. A single insertion may take O(n), but the amortized cost per insertion is O(1). (Growing the capacity by a constant amount would give O(n).)

CLRS 3rd ed., Chapter 17 (Amortized Analysis)

Magic Square

The Magic Square Problem

15812417161475232220136432119121092251811

Every row, column and diagonal: 65

Place the numbers 1 to n² in an n×n square so that every row, every column and both main diagonals have the same sum: n(n²+1)/2 (65 for 5×5).

Coxeter’s rule (n odd; also known as the Siamese or de la Loubère method): put 1 in the middle of the first row. Always move diagonally up and to the left; if that square is taken, move down from the current square instead. Moves off the square wrap around to the opposite edge. It finishes in n² steps → O(n²).

Brute force: (n²)! arrangements. 25! ≈ 1.55·10²⁵ possibilities; at 10⁹ tries per second that is ≈ 1.55·10¹⁶ s ≈ 4.9 million centuries.

Magic Square

“Computers Are Fast, It’ll Solve It Instantly!”?

A common objection: a new 10 GHz computer will solve this right away; no need to bother with algorithms.
A machine 10× faster

4.9 million centuries → about 490 thousand centuries. Still hopeless.

The right algorithm

Coxeter’s rule: 25 steps, under a microsecond. Algorithms come before hardware.

Calculation: 25! / (10¹⁰ · 3.15·10⁹) ≈ 4.9·10⁵ centuries.

Magic Square · Run the code step by step

Code Trace: magic(3, sq)

1

Middle of the first row: i = 0, j = 1 → write 1.

12

s = 2: the up-left square (2, 0) is free (row wrapped → bottom row) → move there.

132

s = 3: the up-left square (1, 2) is free (column wrapped → rightmost column) → move there.

1324

s = 4: the up-left square (0, 1) is taken (it holds 1) → one down: (2, 2).

15324

s = 5: the up-left square (1, 1) is free → move there.

615324

s = 6: the up-left square (0, 0) is free → move there.

6175324

s = 7: the up-left square (2, 2) is taken (it holds 4) → one down: (1, 0).

61875324

s = 8: the up-left square (0, 2) is free (column wrapped → rightmost column) → move there.

618753294

s = 9: the up-left square (2, 1) is free (row wrapped → bottom row) → move there.

618753294

s = 10 > 9: the loop ends. Every row, column and diagonal sums to 15 = 3·(9+1)/2.

void magic(int n, int sq[][MAX]) {
  int i = 0, j = (n - 1) / 2, s, r, c;
  sq[i][j] = 1;       /* sq must be all zeros */
  for (s = 2; s <= n * n; s++) {
    r = (i - 1 < 0) ? n - 1 : i - 1; /* up */
    c = (j - 1 < 0) ? n - 1 : j - 1; /* left */
    if (sq[r][c]) i = (i + 1) % n; /* down */
    else { i = r; j = c; }
    sq[i][j] = s;
  }
}

Performance Measurement

Performance Measurement: Two Methods

1 · Processor time (clock)2 · Calendar time (time)
Startstart = clock();start = time(NULL);
Stopstop = clock();stop = time(NULL);
Typeclock_ttime_t
Result (s)(double)(stop - start) / CLOCKS_PER_SECdifftime(stop, start)
clock()

Number of processor clock ticks used by the program. Resolution is limited by the system’s tick interval.

time()

Wall-clock time with one-second resolution. Fine for long jobs, too coarse for short ones.

CLK_TCK is an old, nonstandard name; standard C uses CLOCKS_PER_SEC. I/O devices (screen, disk, printer) are slow: keep printf calls outside the timed region.

Performance Measurement

Exercise 1: Timing Selection Sort

#include <stdio.h>
#include <time.h>
#define MAX_SIZE 5001
#define ITERATIONS 20
void sort(int list[], int n);   /* selection sort */

int main(void) {
  int i, j, list[MAX_SIZE];
  int sizelist[] = { 0, 100, 300, 600, 900, 1200, 1500, 2000,
    3000, 3100, 3200, 3300, 3400, 3500, 3600, 3700, 3800,
    3900, 4000, 5000 };
  clock_t start, stop;  double ms;
  for (i = 0; i < ITERATIONS; i++) {
    for (j = 0; j < sizelist[i]; j++)
      list[j] = sizelist[i] - j;   /* descending */
    start = clock();
    sort(list, sizelist[i]);
    stop = clock();
    ms = 1000.0 * (stop - start) / CLOCKS_PER_SEC;
    printf("%4d elements: %6.1f ms\n", sizelist[i], ms);
  }
  return 0;
}
  • 20 values of n; for each, an array in descending order.
  • Only the sort call lies between the two clock() calls.
  • Writing (int)(1000*(stop-start)/CLK_TCK) drops the fraction through integer division; double arithmetic with 1000.0 is more accurate.

Performance Measurement

Results and Inconsistencies

050100150200250010002000300040005000n (elements)ms

Dots: sample measurements · dashed: c·n² curve

  • Everything up to 600 elements takes 0 ms?
  • 3000 elements 110 ms, but 3100 only 93 ms?
  • 3500, 3600 and 3700 all 125 ms?
Why? If the sort finishes between two clock ticks, stop − start = 0. On a system with a ~15.6 ms tick, results are rounded to multiples of 15–16 ms.
Fix: Repeat each measurement many times and take the average.

Performance Measurement

Exercise 2: Sequential Search (with a Sentinel)

int seqsearch(int list[], int searchnum, int n) {
  int i;
  list[n] = searchnum;          /* sentinel */
  for (i = 0; list[i] != searchnum; i++)
    ;
  return (i < n) ? i : -1;
}
start = clock();
for (j = 0; j < runtimes[i]; j++)
  pos = seqsearch(list, -1, sizelist[i]);
stop = clock();
total = 1000.0 * (stop - start) / CLOCKS_PER_SEC;
avg = 1000.0 * total / runtimes[i];  /* in μs */
list[sizelist[i]] = sizelist[i];  /* undo sentinel */
  • Sentinel: the key is stored in list[n], so the loop always stops and needs no i < n test on each pass.
  • The array needs room for n + 1 elements (MAX_SIZE = 5001).
  • We search for −1, which is never found → the worst case is measured.
  • For each n the search is repeated runtimes[i] times (500000 … 5000).

Performance Measurement · Run the code step by step

Code Trace: seqsearch({4, 8, 2}, 5, 3)

0418223?

The array holds 3 elements, but room for list[3] is reserved. We search for 5.

04182235sentinel

Sentinel: list[3] = 5. The loop now stops here at the latest.

04182235▲ isentinel

list[0] = 4 ≠ 5 → i++. (One test: no bounds check.)

04182235▲ isentinel

list[1] = 8 ≠ 5 → i++. (One test: no bounds check.)

04182235▲ isentinel

list[2] = 2 ≠ 5 → i++. (One test: no bounds check.)

04182235▲ isentinel

list[3] = 5 → the loop stops, but this is the sentinel.

04182235▲ isentinel

i = 3 is not < n → return −1: not found. n + 1 = 4 comparisons.

int seqsearch(int list[], int searchnum, int n) {
  int i;
  list[n] = searchnum;          /* sentinel */
  for (i = 0; list[i] != searchnum; i++)
    ;
  return (i < n) ? i : -1;
}

Performance Measurement

Repeated Measurement: Average Search Time

05101520010002000300040005000n (elements)μs / search

Dots: measured averages · dashed: linear trend

  • The averages are no longer 0 and grow linearly with n, as O(n) predicts.
  • Fluctuation beyond 3000: with fewer repetitions (runtimes) the measurement gets coarse again.
  • The test machine ticks 1000 times per second: 1 tick = 1 ms.
Repeat short jobs many times and divide the total time by the repetition count.

Homework

Homework (Recursion)

  1. Write a recursive function that computes n!.
  2. The Fibonacci numbers are f₀ = 0, f₁ = 1, fᵢ = fᵢ₋₁ + fᵢ₋₂ (i > 1). Write a recursive function that computes fᵢ.
  3. Given C(n, m) = C(n−1, m) + C(n−1, m−1), write a recursive function that computes C(n, m) = n! / (m!·(n−m)!).
Hints
  1. Base: n ≤ 1 → 1. Step: n · fact(n−1).
  2. Base: n ≤ 1 → n. Step: fib(n−1) + fib(n−2).
  3. Base: m == 0 or m == n → 1.

Homework

Solutions and Common Mistakes

long fact(long n) {
  if (n <= 1) return 1;
  return n * fact(n - 1);
}

long fib(long n) {
  if (n <= 1) return n;   /* f0 = 0, f1 = 1 */
  return fib(n - 1) + fib(n - 2);
}

long binom(int n, int m) {
  if (m == 0 || m == n) return 1;
  return binom(n - 1, m) + binom(n - 1, m - 1);
}
fib: + not *

A common mistake is writing fib(n-1) * fib(n-2); the product is always 0 or 1. By definition it must be a sum.

binom: parameters

A common mistake is writing long binom() with no parameters; n and m must be parameters.

Cost

fact is O(n); fib and binom as written are exponential: the same subproblems are solved again and again.

Summary

Complexity Summary

FunctionTimeExtra spaceNote
sort (selection sort)Θ(n²)O(1)n(n−1)/2 comparisons, ≤ n−1 swaps
sumΘ(n)O(1)2n + 3 steps
rsumΘ(n)O(n)2n + 2 steps, n+1 frames
add (matrix addition)Θ(rows·cols)O(1)Θ(n²) for a square matrix
seqsearchO(n)O(1)best Θ(1), worst n + 1
magic (Coxeter)Θ(n²)O(1)brute force: (n²)!
Dynamic array insertionO(1) amortizedO(n)a single insertion is O(n) worst case
fib (recursive)exponentialO(n)stack depth n

Summary

Common Mistakes

1Thinking “it compiled and ran, so it’s correct”: test data must cover every scenario.
2Confusing step counts with constants: 2n + 3 and 100n are in the same class, O(n).
3Finding O(n²) and calling it “the best bound”: write the smallest g(n) you can; use Θ for a tight bound.
4Timing a short job with a single clock() pair: the result is 0 or a multiple of the tick.
5Leaving printf in the timed region: I/O time swamps the algorithm.
6Passing an argument with side effects to a macro: SWAP(a[i++], …).

Summary

Exercises

  1. How many times does x++ execute in for (i = 0; i < n; i++) for (j = 0; j < i; j++) x++;? O(?)
  2. Show that 10n² + 100n + 1000 = O(n²) by giving c and n₀.
  3. How many comparisons does selection sort make on an already sorted array? Does it have a best case?
  4. An O(n log n) algorithm takes 10 ms for n = 1000. About how long for n = 10⁶?
  5. Why is calling rsum with n = 10⁶ dangerous?
  6. Write the first row of a 7×7 magic square using Coxeter’s rule (what must the sum be?).
Show answers / hints
  1. n(n−1)/2 → Θ(n²).
  2. 10n² + 100n + 1000 ≤ 1110n² for n ≥ 1: c = 1110, n₀ = 1 (other choices work too).
  3. Still n(n−1)/2; the comparison count does not depend on the input (only the swap count does).
  4. The ratio (10⁶·log 10⁶)/(10³·log 10³) = 2000 → ≈ 20 s.
  5. 10⁶ frames may overflow the stack; the iterative sum is safe.
  6. Sum 7·50/2 = 175; first row: 28 19 10 1 48 39 30.

Summary

Chapter Summary

Process

Requirements → analysis → design → coding → verification. Deferring the implementation leads to better choices.

Abstraction

ADT = specification ⟂ implementation. In C: a header file + an opaque struct.

Counting

Space: c + SP(I). Time: program steps (sum 2n+3, rsum 2n+2).

Classifying

O upper, Ω lower, Θ tight bound. The worst case is a guarantee; amortized analysis looks at the whole sequence.

Growth

Small n misleads; 2ⁿ and n! quickly become hopeless. Algorithms come before hardware.

Measurement

clock() + repetition + averaging; keep I/O out of the timing.

Summary

References

  • Horowitz, Sahni, Anderson-Freed, Fundamentals of Data Structures in C, 2nd ed., Chapter 1.
  • Cormen, Leiserson, Rivest, Stein, Introduction to Algorithms, 3rd ed.: Chapters 2–3 (analysis, asymptotics), Chapter 17 (amortized analysis).
  • Kernighan, Ritchie, The C Programming Language, 2nd ed.

The End

Chapter 1: Introduction and Algorithm Analysis
Next chapter: Chapter 2: Arrays and Structures · Course page
1 / 1 Data Structures in C