Chapter 20: Advanced Sorting Algorithms

Data Structures and Algorithms with C
QuicksortAverage caseNon-recursiveMedian-of-3qsortMerge sortHeap sortRadix sort
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Quicksort

Explain the partition invariant; trace partition() and the recursion step by step.

Analysis

Derive best n lg n, worst n² and average ≈ 1.39 n lg n from recurrences.

Improvements

A non-recursive version (O(lg n) stack), median-of-3, insertion sort for small pieces; qsort in C.

Merge sort

Trace merge, the stack frames and the linked (index-based) version; O(n) extra memory.

Heap sort

Build a heap bottom-up in O(n) and sort in place in O(n lg n).

By digits

Implement LSD radix sort with linked-list buckets and interpret its cost.

Quicksort

Quicksort

  • Developed by C. A. R. (Tony) Hoare in 1962.
  • Divide and conquer: split the problem recursively into subproblems until they are small enough to solve directly.
  • Time proportional to n lg n on average; n² in the worst case. Uses only a small auxiliary stack.
  • Not stable, yet one of the fastest general-purpose algorithms known. The standard C library has qsort.
int partition(int a[], int L, int R);

void quicksort1(int a[], int L, int R) {
    if (R <= L) return;
    int i = partition(a, L, R);
    quicksort1(a, L, i - 1);
    quicksort1(a, i + 1, R);
}
From the lecture “This code is taken from Sedgewick's book. Very nicely written code. (Sedgewick was Knuth's PhD student and worked a lot on quicksort.)”

Quicksort

partition()

Lleft end≤ vless/equal?unscanned≥ vgreater/equalva[R]i →← j
int partition(int a[], int L, int R) {
    int i = L - 1, j = R, v = a[R]; // pivot a[R]
    for (;;) {
        while (less(a[++i], v))
            ;
        while (less(v, a[--j]))
            if (j == L) break;
        if (i >= j) break;
        SWAP(a[i], a[j]);
    }
    SWAP(a[i], a[R]);
    return i;
}
  • [2] The pivot (partitioning element) is v = a[R].
  • [4] i stops at the first element from the left that is greater than or equal to v. No bounds test is needed: at worst a[R] = v stops it at i = R.
  • [6–7] j stops at an element from the right that is less than or equal to v; j == L keeps this loop in bounds.
  • [9] Both elements are on the wrong side → swap. [8] When i and j meet, the loop ends; [11] the pivot goes to a[i]: left ≤ v ≤ right.

Quicksort · Step through the code

Code Trace: partition(a, 0, 9)

037115224333449511651777872948▲ jv48

Pivot v = a[9] = 48. i = L − 1 = −1, j = R = 9: the scans start with ++i and --j.

037115224333449511651777872948▲ i▲ jv48

a[0] = 37 < 48 → i keeps going.

037115224333449511651777872948▲ i▲ j≤ vv48

a[1] = 15 < 48 → i keeps going.

037115224333449511651777872948▲ i▲ j≤ vv48

a[2] = 24 < 48 → i keeps going.

037115224333449511651777872948▲ i▲ j≤ vv48

a[3] = 33 < 48 → i keeps going.

037115224333449511651777872948▲ i▲ j≤ vv48

a[4] = 49 ≥ 48 → stops at i = 4 (an element that belongs on the right).

037115224333449511651777872948▲ i▲ j≤ vv48

a[8] = 72 > 48 → j moves left.

037115224333449511651777872948▲ i▲ j≤ v≥ vv48

a[7] = 77 > 48 → j moves left.

037115224333449511651777872948▲ i▲ j≤ v≥ vv48

a[6] = 51 > 48 → j moves left.

037115224333449511651777872948▲ i▲ j≤ v≥ vv48

a[5] = 11 ≤ 48 → stops at j = 5.

037115224333411549651777872948▲ i▲ j≤ v≥ vv48

i < j: a[4] and a[5] are on the wrong sides → swap: [37, 15, 24, 33, 11, 49, 51, 77, 72, 48].

037115224333411549651777872948▲ i j≤ v≥ vv48

a[5] = 49 ≥ 48 → stops at i = 5 (an element that belongs on the right).

037115224333411549651777872948▲ i▲ j≤ v≥ vv48

a[4] = 11 ≤ 48 → stops at j = 4.

037115224333411549651777872948▲ i▲ j≤ v≥ vv48

i = 5 ≥ j = 4: the indices crossed → leave the loop.

037115224333411548651777872949▲ i▲ j≤ 48≥ 48v48

The pivot moves to a[5] and 5 is returned. In the sorted array, 48 would sit exactly here.

int partition(int a[], int L, int R) {
    int i = L - 1, j = R, v = a[R]; // pivot a[R]
    for (;;) {
        while (less(a[++i], v))
            ;
        while (less(v, a[--j]))
            if (j == L) break;
        if (i >= j) break;
        SWAP(a[i], a[j]);
    }
    SWAP(a[i], a[R]);
    return i;
}

Quicksort · Step through the code

Code Trace: quicksort1(a, 0, 9), the Call Stack

037115224333411548651777872949▲ L▲ R

partition(0, 9): pivot 48 → position 5. Left a[0..4] and right a[6..9] are sorted separately.

011115224333437548651777872949▲ L▲ R

partition(0, 4): pivot 11 → position 0. Left a[0..-1] and right a[1..4] are sorted separately.

011115224333437548651777872949

quicksort1(0, -1): empty range → return.

011115224333437548651777872949▲ L▲ R

partition(1, 4): pivot 37 → position 4. Left a[1..3] and right a[5..4] are sorted separately.

011115224333437548651777872949▲ L▲ R

partition(1, 3): pivot 33 → position 3. Left a[1..2] and right a[4..3] are sorted separately.

011115224333437548651777872949▲ L▲ R

partition(1, 2): pivot 24 → position 2. Left a[1..1] and right a[3..2] are sorted separately.

011115224333437548651777872949▲ L

quicksort1(1, 1): one element, already in place → return.

011115224333437548651777872949

quicksort1(3, 2): empty range → return.

011115224333437548651777872949

quicksort1(4, 3): empty range → return.

011115224333437548651777872949

quicksort1(5, 4): empty range → return.

011115224333437548649777872951▲ L▲ R

partition(6, 9): pivot 49 → position 6. Left a[6..5] and right a[7..9] are sorted separately.

011115224333437548649777872951

quicksort1(6, 5): empty range → return.

011115224333437548649751872977▲ L▲ R

partition(7, 9): pivot 51 → position 7. Left a[7..6] and right a[8..9] are sorted separately.

011115224333437548649751872977

quicksort1(7, 6): empty range → return.

011115224333437548649751872977▲ L▲ R

partition(8, 9): pivot 77 → position 9. Left a[8..8] and right a[10..9] are sorted separately.

011115224333437548649751872977▲ L

quicksort1(8, 8): one element, already in place → return.

011115224333437548649751872977

quicksort1(10, 9): empty range → return.

int partition(int a[], int L, int R);

void quicksort1(int a[], int L, int R) {
    if (R <= L) return;
    int i = partition(a, L, R);
    quicksort1(a, L, i - 1);
    quicksort1(a, i + 1, R);
}

Quicksort

Assessment: Why Is It Quick?

Right position + smaller problem

Bubble and selection sort also put one element in place at each step, but the next subproblem still has n − 1 elements. Quicksort places the pivot and also splits the problem into two smaller pieces.

Recursion tree

The quicksort1(a, 0, 9) trace makes 17 calls for 10 elements: every non-trivial call put a pivot in its final place (green cells in the trace).

From the lecture “You may say that bubble or selection sort also find the right position of one element at each step. Yes, they do, but they cannot help the next step by shrinking the problem.”
From the lecture “I will write three or four quicksort algorithms, so understanding this one matters. […] Small pieces of code should be written over and over, in different ways.”

Quicksort · Reading

Reading: Algorithms and Computers

From the lecture “— Sir! Where did this man try quicksort in the 1960s, how did he test it?
Where does the word algorithm come from? Did al-Khwarizmi have a 4 GB laptop in the 9th century? ☺”
From the lecture “An algorithm means that someone who knows nothing about it can take it, run it step by step and get the right result. Besides, there were computers in the 1960s. But our course is independent of computers. Publishing a paper on an algorithm is enough.”
Al-Khwarizmi (780–850)

Mathematician and astronomer at the House of Wisdom in Baghdad. Author of al-Kitāb al-Mukhtaṣar fī Ḥisāb al-Jabr wa'l-Muqābala, which gave systematic solutions of linear and quadratic equations.

Origin of the words

“Algebra” comes from the book's al-jabr method, and “algorithm” from Algoritmi, the Latin form of his name. His works introduced the decimal positional number system to the West.

Quicksort

Quicksort: Worst and Best Case

Worst: a sorted array

The pivot a[R] is always the largest and stays put; the next call is one element smaller:
n + (n−1) + … + 1 = n(n+1)/2 = O(n²)

Best: splitting exactly in half

At each step i and j advance n steps in total, leaving two subproblems of size n/2:
Cn = 2Cn/2 + n = 2kCn/2k + k·n = n lg n

From the lecture “Now you'll be wondering: ‘What's so quick about it?’”
The best case is n lg n; the optimal sorting bound was n lg n − 1.44n. So the leading coefficient is still 1, with an excess of 1.44n.

Quicksort · Numerical check

Counts: quicksort1 on Sorted and Random Input

nsorted: comparisonsdepthrandom: comparisonsdepthn lg n
1063936533
1005,148997219664
1000501,49899912,831259,966

Comparisons inside partition() counted by a Python simulation (random input: seed 20).

Sorted input

≈ n²/2 comparisons and recursion depth n − 1: in C, a risk of stack overflow for large n.

Random input

Comparisons of order n lg n, depth only a few lg n.

Quicksort

Average-Case Analysis: the Recurrence

Cn = (n + 1) + (1/n) Σk=1n (Ck−1 + Cn−k),   C0 = C1 = 0

  • The split point k, 1 ≤ k ≤ n, is equally likely: running partition() on all n! orderings gives equal frequencies.
  • n + 1: the comparisons with the pivot in partition(); a split at k leaves subarrays of (k−1) and (n−k) elements.
  • The two sums contain the same terms: Σ(Ck−1 + Cn−k) = 2 Σk=0n−1 Ck ⇒ n·Cn = n(n+1) + 2 Σ Ck−1 … (1)
From the lecture “Toss a coin and count the heads. 10 times, 100 times, 1000 times, even a million. Someone said: it converges to ½! That is how average-case analysis tells us where things concentrate.”
From the lecture “‘In my opinion, every k is equally likely.’ There's no ‘my opinion, your opinion’ here! There's mathematics.”

Quicksort

Average Case: the Subtraction Trick and the Result

  • Replace n by n − 1 in (1): (n−1)Cn−1 = n(n−1) + 2 Σk=1n−1 Ck−1 … (2)
  • (1) − (2): n·Cn − (n−1)Cn−1 = 2n + 2Cn−1 ⇒ n·Cn = (n+1)Cn−1 + 2n
  • (dividing by n(n+1)) Cn/(n+1) = Cn−1/n + 2/(n+1): now it unrolls by iteration.
  • Unrolling gives Cn ≈ 2(n+1)(1/4 + … + 1/(n+1)) ≤ 2(n+1) ln n.

Cn ≈ 2n ln n = 2 ln 2 · n lg n ≈ 1.39 n lg n

nCn (from the recurrence)1.39 n lg nratio
1037.146.20.80
100780.5923.50.85
100012,318.613,852.40.89

Table: the recurrence computed exactly in Python. The ratio slowly approaches 1 (a lower-order −cn term).

From the lecture “The book solves this by substitution. We will use a smarter trick.”

Improving Quicksort

Non-Recursive Quicksort (quicksort2)

void quicksort2(int a[], int L, int R) {
  stack_init(); push2(L, R);
  while (!stack_empty()) {
    L = pop(); R = pop();
    if (R <= L) continue;
    int i = partition(a, L, R);
    if (i - L > R - i) {       // larger part first
      push2(L, i - 1); push2(i + 1, R);
    } else {
      push2(i + 1, R); push2(L, i - 1);
    }
  }
}
#define STACK_MAX 64  /* > 2·lg n */
static int stack[STACK_MAX], top;
void stack_init(void) { top = 0; }
int stack_empty(void) { return top == 0; }
void push(int x) { stack[top++] = x; }
int pop(void) { return stack[--top]; }
#define push2(A, B) \
  do { push(B); push(A); } while (0)
  • Goal: use our own stack instead of the system stack. push2(A,B) pushes B first, then A, so L is popped first.
  • [7–11] The larger part is pushed first, the smaller one last: the last pushed is popped first, so the smaller part is always processed next → the stack depth stays O(lg n) (at most ≈ lg n pairs).
  • The memory recurrence Tn = Tn/2 + 2 ⇒ Tn = 2 lg n (as in binary search).

Improving Quicksort · Step through the code

Code Trace: quicksort2(a, 0, 6), an Explicit Stack

04172236415563stack (top at right)60

The first problem (0, 6) is pushed: first R = 6, then L = 0.

01122336445567▲ L▲ Rstack (top at right)6310

(0, 6): partition → pivot 3 at position 2. The right part (3,6) is not smaller → pushed first, then the left (0,1); the smaller part is on top.

01122336445567▲ L▲ Rstack (top at right)630012

(0, 1): partition → pivot 2 at position 1. The left part (0,0) is larger → pushed first, then the right (2,1); the smaller part is on top.

01122336445567stack (top at right)6300

(2, 1) popped: empty range → continue.

01122336445567▲ Lstack (top at right)63

(0, 0) popped: one element → continue.

01122336445567▲ L▲ Rstack (top at right)5367

(3, 6): partition → pivot 7 at position 6. The left part (3,5) is larger → pushed first, then the right (7,6); the smaller part is on top.

01122336445567stack (top at right)53

(7, 6) popped: empty range → continue.

01122334455667▲ L▲ Rstack (top at right)5533

(3, 5): partition → pivot 5 at position 4. The right part (5,5) is not smaller → pushed first, then the left (3,3); the smaller part is on top.

01122334455667▲ Lstack (top at right)55

(3, 3) popped: one element → continue.

01122334455667▲ Lstack (top at right)empty

(5, 5) popped: one element → continue.

01122334455667stack (top at right)empty

The stack is empty → done: [1, 2, 3, 4, 5, 6, 7]. At most 3 (L, R) pairs waited on the stack.

void quicksort2(int a[], int L, int R) {
  stack_init(); push2(L, R);
  while (!stack_empty()) {
    L = pop(); R = pop();
    if (R <= L) continue;
    int i = partition(a, L, R);
    if (i - L > R - i) {       // larger part first
      push2(L, i - 1); push2(i + 1, R);
    } else {
      push2(i + 1, R); push2(L, i - 1);
    }
  }
}

Improving Quicksort

Improved Quicksort: Median-of-3 + Insertion Sort

#define M 10    /* leave pieces smaller than this */
void quicksort3(int a[], int L, int R) {
    if (R - L <= M) return;
    SWAP(a[(L + R) / 2], a[R - 1]);
    if (a[R-1] < a[L]) SWAP(a[R-1], a[L]); // a[L] <= a[R-1]
    if (a[R] < a[L]) SWAP(a[R], a[L]);     // a[L] <= a[R]
    if (a[R] < a[R-1]) SWAP(a[R], a[R-1]); // a[R-1] <= a[R]
    int i = partition(a, L + 1, R - 1);
    quicksort3(a, L, i - 1);
    quicksort3(a, i + 1, R);
}
void sort(int a[], int L, int R) {
    quicksort3(a, L, R);
    insertion(a, L, R);   /* the small pieces left */
}
if (less(A, B)) SWAP(A, B) means “swap if A < B”; with the arguments in the order (a[L], a[R-1]) it puts the larger one in a[L] and breaks the partition (tested with random arrays: an element larger than the pivot stays on its left). So the arguments are given in reverse order: the result is a[L] ≤ a[R−1] ≤ a[R].
From the lecture “The standard C library function qsort() is probably this.”

Improving Quicksort

Why Median-of-3 and a Small-Piece Threshold?

Median-of-three

a[L], the middle element and a[R] are put in order; their median moves to a[R−1] and becomes the pivot. Quicksort did worst and insertion sort best on sorted data: the n² case on sorted input disappears.

Natural sentinels

Since a[L] ≤ pivot ≤ a[R], partitioning runs on L+1..R−1; the end elements stop the scans.

Threshold M

Pieces with R − L ≤ M are left unsorted; one final insertion pass in sort() fixes them all in nearly linear time. About 10% faster for 5 ≤ M ≤ 25.

Its limit

Median-of-3 rescues sorted and reverse-sorted input, but specially crafted bad inputs can still cause n² → the cure is introsort.

From the lecture “It never leaves you facing n². The analysis is a bit heavy…”

Improving Quicksort · Step through the code

Code Trace: Median-of-3 Setup, quicksort3(a, 0, 8)

081325314952677684▲ L▲ R

R − L = 8 > M = 3 (M = 3 in this trace) → the piece is large; partition it.

081325314652677984▲ L▲ R−1▲ R

The middle element a[4] is swapped with a[R−1] = a[7]: the candidate pivot 9 is now in a[7].

081325314652677984▲ L▲ R−1▲ R

a[7] = 9 ≥ a[0] = 8 → no swap.

041325314652677988▲ L▲ R−1▲ R

a[8] = 4 < a[0] = 8 → swap.

041325314652677889▲ L▲ R−1▲ R

a[8] = 8 < a[7] = 9 → swap.

041325314652677889▲ L▲ i▲ R

a[L] = 4 ≤ pivot 8 ≤ a[R] = 9. partition(a, 1, 7) → pivot at position 7; left ≤ 8 ≤ right.

041325314652677889

The left a[0..6] comes first: R − L = 6 > M, so median-of-3 repeats; partition(a, 1, 5) → 3 and the array becomes [1, 3, 2, 4, 6, 5, 7, 8, 9]. The remaining pieces (R − L = 2 ≤ M) return unsorted; finally the right a[8..8] (R − L = 0 ≤ M) returns. The insertion pass in sort() finishes the job.

void quicksort3(int a[], int L, int R) {
    if (R - L <= M) return;
    SWAP(a[(L + R) / 2], a[R - 1]);
    if (a[R-1] < a[L]) SWAP(a[R-1], a[L]); // a[L] <= a[R-1]
    if (a[R] < a[L]) SWAP(a[R], a[L]);     // a[L] <= a[R]
    if (a[R] < a[R-1]) SWAP(a[R], a[R-1]); // a[R-1] <= a[R]
    int i = partition(a, L + 1, R - 1);
    quicksort3(a, L, i - 1);
    quicksort3(a, i + 1, R);
}

Reading

Reading: Mandatory Internships

From the lecture “I believe internships were introduced out of necessity. In the old days you wrote a program, brought it to the lab and had it punched onto cards. You got one, at most two compile turns a day. How many times a day can you compile now? 50? 100?”
From the lecture “Now things are different… They use our students for errands: do this, do that. Like digging soil with a keyboard… We have nothing to do with the market! What matters to us is educating scientists…”

C Library and Hybrids · Step through the code

Code Trace: qsort and the Comparison Function

037115224333449

The array is ready; sizeof a / sizeof a[0] = 20 / 4 = 5 elements (assuming a 4-byte int).

037115224333449

qsort gets the array's address, the element count, the element size and the address of the comparison function. qsort does not know the element type; it works with void *.

037115224333449

Sample call ascending(&37, &15) → 1: positive = the first element goes after. (The standard does not say which pairs qsort compares, or in what order.)

037115224333449

Sample call ascending(&15, &24) → -1: negative = the first element goes before. (The standard does not say which pairs qsort compares, or in what order.)

037115224333449

Sample call ascending(&24, &24) → 0: zero = equal (relative order not guaranteed). (The standard does not say which pairs qsort compares, or in what order.)

015124233337449

qsort returned: the array is in ascending order.

015124233337449

printf: a[0] = 15.

015124233337449

printf: a[1] = 24.

015124233337449

printf: a[2] = 33.

015124233337449

printf: a[3] = 37.

015124233337449

printf: a[4] = 49.

int ascending(const void *p, const void *q) {
    int x = *(const int *)p;
    int y = *(const int *)q;
    return (x > y) - (x < y);  // -1, 0, +1
}
int main(void) {
    int a[] = {37, 15, 24, 33, 49};
    size_t n = sizeof a / sizeof a[0];
    qsort(a, n, sizeof a[0], ascending);
    for (size_t i = 0; i < n; i++)
        printf("%d ", a[i]);
    return 0;
}

C Library and Hybrids

Sorting Records (structs) with qsort

typedef struct { char name[20]; int grade; } student;

int by_name(const void *p, const void *q) {
    const student *x = p, *y = q;
    return strcmp(x->name, y->name);    // alphabetical
}
int by_grade_desc(const void *p, const void *q) {
    const student *x = p, *y = q;
    return (y->grade > x->grade) - (y->grade < x->grade);
}
/* qsort(s, n, sizeof s[0], by_name); */
The contract

Returns negative / zero / positive and must be consistent (a < b, b < c ⇒ a < c).

Trap: overflow

return x - y; is short but overflows for INT_MIN and 1; (x > y) - (x < y) is safe.

No stability

qsort need not be stable or use any set algorithm; if equal keys' order matters, add a secondary key.

C Library and Hybrids

Hybrid Sorts: Combining Algorithms

Quicksort + insertion

On small pieces the cost of recursion dominates; pieces below M are finished by insertion sort (quicksort3 + sort).

Introsort (Musser, 1997)

Starts with quicksort; if the recursion depth exceeds ≈ 2⌊lg n⌋, it switches to heap sort on that piece → worst case O(n lg n). Small pieces go to insertion sort. Common in C++ std::sort implementations.

Timsort (Peters, 2002)

Merge + insertion sort; finds already sorted runs in the data and merges them. Stable; Python's sort and Java's Arrays.sort for object arrays.

The common idea

Good asymptotic behavior at scale (n lg n) + a low constant on small inputs (insertion) + insurance against the worst case.

Merge Sort

Merge Sort

  1. A list of 0 or 1 elements is already sorted.
  2. Otherwise split the list into two halves as equal as possible.
  3. Sort each half by applying merge sort recursively.
  4. Merge the two sorted sublists into one sorted list.
  • John von Neumann, 1945; divide and conquer.
  • Idea 1: sorting a small list takes fewer steps.
  • Idea 2: building a sorted list from two sorted lists is cheaper than from two unsorted ones.
void mergesort(int a[], int b[], int L, int R) {
    if (R <= L) return;
    int m = (R + L) / 2;
    mergesort(a, b, L, m);
    mergesort(a, b, m + 1, R);
    merge(a, b, L, m, R);
}
void merge(int a[], int b[], int L, int m, int R) {
    int i, j, k;
    for (i = m + 1; i > L; i--) b[i - 1] = a[i - 1];
    for (j = m; j < R; j++) b[R + m - j] = a[j + 1];
    for (k = L; k <= R; k++)
        if (less(b[j], b[i]))
            a[k] = b[j--];
        else
            a[k] = b[i++];
}

The auxiliary array b[] (same size as a) is passed as a parameter; it cannot be used without being defined.

Merge Sort · Step through the code

Code Trace: merge(a, b, 0, 2, 6)

ab0315273445566100315273456▲ ileft half → | ← right half (reversed)

Loop 1: the left half a[0..2] is copied as is to b[0..2]; the loop exits with i = L = 0.

ab031527344556610031527310465564▲ i▲ jleft half → | ← right half (reversed)

Loop 2: the right half is copied to b[3..6] in reverse → b increases from both ends toward the middle (bitonic). The loop exits with j = R = 6.

ab031527344556610▲ k031527310465564▲ i▲ jleft half → | ← right half (reversed)

b[j] = 4 ≥ b[i] = 3 → a[0] = b[0]; i++.

ab031427344556610▲ k031527310465564▲ i▲ jleft half → | ← right half (reversed)

b[j] = 4 < b[i] = 5 → a[1] = b[6]; j--.

ab031425344556610▲ k031527310465564▲ i▲ jleft half → | ← right half (reversed)

b[j] = 5 ≥ b[i] = 5 → a[2] = b[1]; i++.

ab031425354556610▲ k031527310465564▲ i▲ jleft half → | ← right half (reversed)

b[j] = 5 < b[i] = 7 → a[3] = b[5]; j--.

ab031425354656610▲ k031527310465564▲ i▲ jleft half → | ← right half (reversed)

b[j] = 6 < b[i] = 7 → a[4] = b[4]; j--.

ab031425354657610▲ k031527310465564▲ i jleft half → | ← right half (reversed)

b[j] = 10 ≥ b[i] = 7 → a[5] = b[2]; i++.

ab031425354657610▲ k031527310465564▲ i▲ jleft half → | ← right half (reversed)

b[j] = 10 ≥ b[i] = 10 → a[6] = b[3]; i++.

ab031425354657610031527310465564▲ i▲ jleft half → | ← right half (reversed)

Done: a = [3, 4, 5, 5, 6, 7, 10]. At the end i = 4, j = 3: the two indices met in the middle; no bounds test was needed. Of the equal 5s, the left one (b[i]) was taken first.

void merge(int a[], int b[], int L, int m, int R) {
    int i, j, k;
    for (i = m + 1; i > L; i--) b[i - 1] = a[i - 1];
    for (j = m; j < R; j++) b[R + m - j] = a[j + 1];
    for (k = L; k <= R; k++)
        if (less(b[j], b[i]))
            a[k] = b[j--];
        else
            a[k] = b[i++];
}

Merge sort table · Step through the code

Code Trace: mergesort, 9 merge Calls

089167259362423593624720828960

mergesort(a, b, 0, 9): the left half is sorted completely first; the first merge happens only in the deepest call, (0, 1).

067189259362423593624720828960leftright

merge #1: a[0..0] and a[1..1] merged → a[0..1] sorted.

059167289362423593624720828960leftright

merge #2: a[0..1] and a[2..2] merged → a[0..2] sorted.

059167289323462593624720828960leftright

merge #3: a[3..3] and a[4..4] merged → a[3..4] sorted.

023159262367489593624720828960leftright

merge #4: a[0..2] and a[3..4] merged → a[0..4] sorted.

023159262367489524693720828960leftright

merge #5: a[5..5] and a[6..6] merged → a[5..6] sorted.

023159262367489520624793828960leftright

merge #6: a[5..6] and a[7..7] merged → a[5..7] sorted.

023159262367489520624793828960leftright

merge #7: a[8..8] and a[9..9] merged → a[8..9] sorted.

023159262367489520624728860993leftright

merge #8: a[5..7] and a[8..9] merged → a[5..9] sorted.

020123224328459560662767889993leftright

merge #9: a[0..4] and a[5..9] merged → a[0..9] sorted.

void mergesort(int a[], int b[], int L, int R) {
    if (R <= L) return;
    int m = (R + L) / 2;
    mergesort(a, b, L, m);
    mergesort(a, b, m + 1, R);
    merge(a, b, L, m, R);
}

Merge Sort

The Tree of Stack Frames

0,9,4#90,4,2#40,2,1#20,1,0#10,01,12,23,4,3#33,34,45,9,7#85,7,6#65,6,5#55,56,67,78,9,8#78,89,9

Box: L, R, m · orange: a frame that calls merge, #k: which merge. It goes deep on the left first and merges on the way back.

From the lecture “In Data Structures 1 I always said: ‘We call this logic merge.’”

Merge Sort

Question: What If the Two mergesort Lines Swap?

Question: Swap the two mergesort lines in the function and observe the table you get.
Answer: the right halves merge first; same result, different intermediate tables
mergerangearray
#1(8,9)89 67 59 62 23 93 24 20 28 60
#2(5,6)89 67 59 62 23 24 93 20 28 60
#3(5,7)89 67 59 62 23 20 24 93 28 60
#4(5,9)89 67 59 62 23 20 24 28 60 93
#5(3,4)89 67 59 23 62 20 24 28 60 93
#6(0,1)67 89 59 23 62 20 24 28 60 93
#7(0,2)59 67 89 23 62 20 24 28 60 93
#8(0,4)23 59 62 67 89 20 24 28 60 93
#9(0,9)20 23 24 28 59 60 62 67 89 93

Merge Sort

Merge Sort: Time and Space Complexity

an = a⌈n/2⌉ + a⌊n/2⌋ + n,   a1 = 0

n = 2k: an = 2an/2 + n ⇒ an = Θ(n lg n)

  • 16 elements split into 8 + 8, 17 into 8 + 9; merge takes n steps.
  • Number of calls Tn = 2Tn/2 + 1 ⇒ 2n − 1 (19 calls for 10 elements, 9 of which merge).
  • Independent of the input order: best = average = worst = n lg n.
  • Drawback: O(n) extra memory. Avoiding it raises the cost; if no extra memory is allowed, heap sort is preferred.
  • Because it merges sorted runs in sequence, it is the basis of external sorting on disk.
Stability

The classic merge (taking the left one on ties) is stable (quicksort and heap sort are not). The reverse-copy (bitonic) merge here is not stable: [2a, 2b, 2c, 2d] → [2a, 2b, 2d, 2c] (checked in Python).

From the lecture “Since splitting the input in half is the clever part, this one splits it in half too. But in the earlier ones the pivot landed in its right place in one step; here nothing does. It builds a new sorted array by exploiting the fact that the two arrays are already sorted.”

Merge Sort

Sorting Alphabetic Data: Linked Merge Sort

typedef struct {
    int key;       /* + large data fields */
    int link;      /* index of the next item, 0 = end */
} element;

int mergesort(element a[], int L, int R) {
    if (R <= L) return L;          /* one-item list */
    int m = (R + L) / 2;
    int right = mergesort(a, m + 1, R);
    int left = mergesort(a, L, m);
    return merge(a, left, right);  /* head index */
}

merge() links two sorted lists through their link fields and returns the head index.

  • Copying large records (e.g. alphabetic data) is slow. The fix: build a linked list inside the array with a link field; only indices change.
  • The data is in a[1..n]; a[0] is a temporary head during merge. Initially every link is 0.
  • mergesort returns the head index of the sorted sublist.
  • Common mistakes: returning a value from a void function; i1 instead of i2 in the else branch; returning a[c].link instead of a[0].
  • In C the order in which arguments are evaluated is unspecified; here the order is written out explicitly.

Merge Sort · Step through the code

Code Trace: Linked merge (#9)

keylink[0]–4[1]712[2]775[3]1220[4]381[5]1193[6]419[7]226[8]1150[9]438[10]137▲ i1▲ i2▲ last

merge #9: first list (head 4) 38→71→77→119→122; second list (head 10) 13→22→41→43→115. last = 0 (a[0] is the temporary head).

keylink[0]–10[1]712[2]775[3]1220[4]381[5]1193[6]419[7]226[8]1150[9]438[10]137▲ i1▲ i2▲ last

38 > 13 → a[0].link = 10; last = 10; i2 = 7.

keylink[0]–10[1]712[2]775[3]1220[4]381[5]1193[6]419[7]226[8]1150[9]438[10]137▲ i1▲ i2▲ last

38 > 22 → a[10].link = 7; last = 7; i2 = 6.

keylink[0]–10[1]712[2]775[3]1220[4]381[5]1193[6]419[7]224[8]1150[9]438[10]137▲ i1▲ i2▲ last

38 ≤ 41 → a[7].link = 4; last = 4; i1 = 1.

keylink[0]–10[1]712[2]775[3]1220[4]386[5]1193[6]419[7]224[8]1150[9]438[10]137▲ i1▲ i2▲ last

71 > 41 → a[4].link = 6; last = 6; i2 = 9.

keylink[0]–10[1]712[2]775[3]1220[4]386[5]1193[6]419[7]224[8]1150[9]438[10]137▲ i1▲ i2▲ last

71 > 43 → a[6].link = 9; last = 9; i2 = 8.

keylink[0]–10[1]712[2]775[3]1220[4]386[5]1193[6]419[7]224[8]1150[9]431[10]137▲ i1▲ i2▲ last

71 ≤ 115 → a[9].link = 1; last = 1; i1 = 2.

keylink[0]–10[1]712[2]775[3]1220[4]386[5]1193[6]419[7]224[8]1150[9]431[10]137▲ i1▲ i2▲ last

77 ≤ 115 → a[1].link = 2; last = 2; i1 = 5.

keylink[0]–10[1]712[2]778[3]1220[4]386[5]1193[6]419[7]224[8]1150[9]431[10]137▲ i1▲ i2▲ last

119 > 115 → a[2].link = 8; last = 8; i2 = 0.

keylink[0]–10[1]712[2]778[3]1220[4]386[5]1193[6]419[7]224[8]1155[9]431[10]137▲ last

i2 = 0 (the second list is done) → a[8].link = i1 = 5; the head a[0].link = 10 is returned: 13→22→38→41→43→71→77→115→119→122.

int merge(element a[], int first, int second) {
    int last = 0;               // a[0]: temp head
    int i1 = first, i2 = second;
    while (i1 && i2)
        if (a[i1].key <= a[i2].key) {
            a[last].link = i1; last = i1;
            i1 = a[i1].link;
        } else {
            a[last].link = i2; last = i2;
            i2 = a[i2].link;
        }
    a[last].link = i1 ? i1 : i2;
    return a[0].link;
}

Merge Sort

Linked Merge Sort: How the link Fields Change

#[1][2][3][4][5][6][7][8][9][10]returnsL, R
key71771223811941221154313
#10000000009109,10
#2000000600976,7
#3000008600976,8
#40000096087106,10
#5000509608744,5
#6200509608711,2
#7230509608711,3
#8250139608741,5
#92806394517101,10

Bold: the links changed by that merge. Sorted list: 13 → 22 → 38 → 41 → 43 → 71 → 77 → 115 → 119 → 122.

Exam Questions

Exam Question

The adaptation below sorts the element array a[1..9] by key. (i) Draw the stack frames of the recursive calls; where merge is called, write its return value. (ii) Fill in the execution table with merge numbers, return values and the link fields at each step.

int mergesort(element a[], int L, int R) {
    int m;
    if (R <= L) return L;
    m = (R + L) / 2;
    return merge(a, mergesort(a, m + 1, R),
                    mergesort(a, L, m));
}

Keys: [1] ?, [2] ?, [3] 18, [4] −5, [5] 27, [6] 11, [7] ?, [8] 83, [9] 30. In place of the “?” on the exam sheet we use 42, 7, 55. merge() is as in linked merge sort; the printed version has i1 in the else branch (a typo; it should be i2).

Exam Questions

Exam Question (question 3): Solution

Solution

If the arguments are evaluated left to right (right half first), the call order is: (1,9) → (6,9) → (8,9) → (9,9) → (8,8) → (6,7) → (7,7) → (6,6) → (1,5) …

#[1][2][3][4][5][6][7][8][9]ret.L,R,m
#100000000898,9,8
#200000700866,7,6
#300000980766,9,7
#400050980744,5,4
#501050980721,2,1
#603150980721,3,2
#703521980741,5,3
#876529380141,9,5

Resulting list (head 4): -5 → 7 → 11 → 18 → 27 → 30 → 42 → 55 → 83. The table was produced by simulation.

Heap Sort

Heap Sort

void heapify(int a[], int root, int n) {
    int t = a[root], j;
    for (j = 2 * root; j <= n; j *= 2) {
        if (j < n && a[j] < a[j + 1]) j++; // larger
        if (t >= a[j]) break;   // t fits here
        a[j / 2] = a[j];        // move child up
    }
    a[j / 2] = t;
}
void heapsort(int a[], int n) {     /* a[1..n] */
    int i;
    for (i = n / 2; i >= 1; i--)     /* build heap */
        heapify(a, i, n);
    for (i = n - 1; i >= 1; i--) {   /* sort */
        SWAP(a[1], a[i + 1]);    /* largest to the end */
        heapify(a, 1, i);            /* fix the heap */
    }
}
  • O(n lg n) like merge sort, but it uses no extra memory (in place).
  • The array a[1..n] (index 0 unused) is viewed as a complete binary tree: the children of a[i] are a[2i] and a[2i+1].
  • heapify(a, root, n): with the left and right subtrees of root already max heaps, finds the right place for the root value.
  • Leaves are skipped: heapify nodes ⌊n/2⌋, ⌊n/2⌋−1, …, 1 from the back → a max heap.
  • Then the largest at the root moves to the end and the heap is fixed for the remaining i elements.
  • A j or i declared in a for header cannot be used after the loop (out of scope in C99).

Heap Sort · Step through the code

Code Trace: the Sorting Phase, SWAP + heapify(a, 1, i)

7716125934841951162671581951017726135944851961172681591105heap region a[1..10]

The input [26, 5, 77, 1, 61, 11, 59, 15, 48, 19] (Horowitz's example) was turned bottom-up into this max heap by heapify for i = 5…1 (5 moves in all). “Fine, we got that, but when does the array get sorted?” Now: each round the largest (77) at the root moves to the end and the rest of the heap is fixed.

5161259348419511626715819771015261359448519611726815911077heap region a[1..9]

i = 9: SWAP(a[1], a[10]) → the largest, 77, goes to the end; 5 at the root breaks the heap.

6114825931541951162675819771016124835941551961172685911077heap region a[1..9] · sorted part green

heapify(a, 1, 9): 5 sank to a[8]; the new root 61 is the largest remaining.

1148259315419511626758619771011248359415519611726859611077heap region a[1..8] · sorted part green

i = 8: SWAP(a[1], a[9]) → the largest, 61, goes to the end; 1 at the root breaks the heap.

5914822631541951161758619771015924832641551961171859611077heap region a[1..8] · sorted part green

heapify(a, 1, 8): 1 sank to a[7]; the new root 59 is the largest remaining.

5148226315419511617598619771015248326415519611718599611077heap region a[1..7] · sorted part green

i = 7: SWAP(a[1], a[8]) → the largest, 59, goes to the end; 5 at the root breaks the heap.

4811922631545511617598619771014821932641555611718599611077heap region a[1..7] · sorted part green

heapify(a, 1, 7): 5 sank to a[5]; the new root 48 is the largest remaining.

1119226315455116487598619771011219326415556117488599611077heap region a[1..6] · sorted part green

i = 6: SWAP(a[1], a[7]) → the largest, 48, goes to the end; 1 at the root breaks the heap.

2611921131545516487598619771012621931141555617488599611077heap region a[1..6] · sorted part green

heapify(a, 1, 6): 1 sank to a[6]; the new root 26 is the largest remaining.

1119211315455266487598619771011219311415556267488599611077heap region a[1..5] · sorted part green

i = 5: SWAP(a[1], a[6]) → the largest, 26, goes to the end; 1 at the root breaks the heap.

1911521131455266487598619771011921531141556267488599611077heap region a[1..5] · sorted part green

heapify(a, 1, 5): 1 sank to a[4]; the new root 19 is the largest remaining.

5115211314195266487598619771015215311415196267488599611077heap region a[1..4] · sorted part green

i = 4: SWAP(a[1], a[5]) → the largest, 19, goes to the end; 5 at the root breaks the heap.

1515211314195266487598619771011525311415196267488599611077heap region a[1..4] · sorted part green

heapify(a, 1, 4): 5 sank to a[2]; the new root 15 is the largest remaining.

1152113154195266487598619771011253114155196267488599611077heap region a[1..3] · sorted part green

i = 3: SWAP(a[1], a[4]) → the largest, 15, goes to the end; 1 at the root breaks the heap.

1115213154195266487598619771011125314155196267488599611077heap region a[1..3] · sorted part green

heapify(a, 1, 3): 1 sank to a[3]; the new root 11 is the largest remaining.

1152113154195266487598619771011253114155196267488599611077heap region a[1..2] · sorted part green

i = 2: SWAP(a[1], a[3]) → the largest, 11, goes to the end; 1 at the root breaks the heap.

5112113154195266487598619771015213114155196267488599611077heap region a[1..2] · sorted part green

heapify(a, 1, 2): 1 sank to a[2]; the new root 5 is the largest remaining.

1152113154195266487598619771011253114155196267488599611077heap region a[1..1] · sorted part green

i = 1: SWAP(a[1], a[2]) → the largest, 5, goes to the end; the single element left at the root (1) is already a heap.

1152113154195266487598619771011253114155196267488599611077heap region a[1..1] · sorted part green

heapify(a, 1, 1): j = 2 > 1, the loop never runs; 1 stays put. The array is sorted: [1, 5, 11, 15, 19, 26, 48, 59, 61, 77].

void heapify(int a[], int root, int n) {
    int t = a[root], j;
    for (j = 2 * root; j <= n; j *= 2) {
        if (j < n && a[j] < a[j + 1]) j++; // larger
        if (t >= a[j]) break;   // t fits here
        a[j / 2] = a[j];        // move child up
    }
    a[j / 2] = t;
}
    for (i = n - 1; i >= 1; i--) {  /* sort */
        SWAP(a[1], a[i + 1]);
        heapify(a, 1, i);
    }

Heap Sort

Heap Sort: Time Complexity

  • Rough analysis: heapify makes at most lg n moves and is called n/2 times → building the heap costs n lg n.
  • Careful analysis: k = number of levels ≈ lg(n+1); level i has at most 2i−1 nodes, each moving at most (k − i) levels:

Σi=1k 2i−1(k − i) = 2k − k − 1 ≤ 2n = O(n)

  • The term Σ i·2i−1 comes from differentiating the geometric series S(y) = (yk+1 − 1)/(y − 1) and setting y = 2.
  • The sorting phase: n − 1 calls of heapify(a, 1, i) → O(n lg n). Total O(n lg n), also in the worst case; O(1) extra memory; not stable.
  • A bound of “≤ 2n” means O(n), not o(n) (being bounded by 2n is not little-o).
kΣ 2i−1(k−i)2k−k−1n = 2k−1
2113
3447
4111115
6575763
10101310131023
From the lecture “When we implemented Kruskal, we built the heap by inserting the elements into a min heap one by one. That does produce a heap. But here we turned the array holding the elements into a min heap at about half that cost.”

Heap Sort

Heap Sort Variants, and Knowing Your Data

When?

For fewer than 50,000 items heap sort is a good choice (≈ 2n lg n comparisons). For larger data there is bottom-up heapsort, which makes fewer comparisons, and theoretically even better variants.

The bottom-up idea

The hole at the root sinks along the larger children to a leaf without comparing with t; then t is placed by moving up. Mostly one comparison per level instead of two.

From the lecture “As with counting sort, if we know some properties of the data to sort, we can choose an approach accordingly. In that sense Shell sort and bucket sort are used together. You should know the basic few.”

Radix Sort

Radix Sort

  • Sorting by several keys: e.g. people by surname, and those with the same surname by first name.
  • Keys K0, K1, …, Kr−1: K0 is the most significant (MSD), Kr−1 the least significant (LSD). In the example, the surname is most significant.
  • LSD radix sort: sort first by the least significant digit, then by more significant ones in turn, and finally by the MSD.
  • Each pass must be stable: the order from the previous pass is kept for equal digits.
  • It never compares numbers with each other → a non-comparison sort.
#define MAX_DIGIT 3
#define RADIX 10
typedef struct node *node_ptr;
struct node {
    int digit[MAX_DIGIT];  /* 179 → {1, 7, 9} */
    node_ptr next;
};

K0: hundreds (MSD) · K1: tens · K2: units (LSD). Range [0, 999]. A typedef with no type name, like typedef struct node {…};, is meaningless.

Radix Sort · Step through the code

Code Trace: radixsort, i = 2 (units)

p179208306093859984055009271033head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

Pass i = 2 (units digit): the 10 buckets are emptied.

p208306093859984055009271033head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]179

179: digit[2] = 9 → appended to the tail of bucket 9 (arrival order is kept → stable).

p306093859984055009271033head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]208head[9]179

208: digit[2] = 8 → appended to the tail of bucket 8 (arrival order is kept → stable).

p093859984055009271033head[0]head[1]head[2]head[3]head[4]head[5]head[6]306head[7]head[8]208head[9]179

306: digit[2] = 6 → appended to the tail of bucket 6 (arrival order is kept → stable).

p859984055009271033head[0]head[1]head[2]head[3]093head[4]head[5]head[6]306head[7]head[8]208head[9]179

093: digit[2] = 3 → appended to the tail of bucket 3 (arrival order is kept → stable).

p984055009271033head[0]head[1]head[2]head[3]093head[4]head[5]head[6]306head[7]head[8]208head[9]179859

859: digit[2] = 9 → appended to the tail of bucket 9 (arrival order is kept → stable).

p055009271033head[0]head[1]head[2]head[3]093head[4]984head[5]head[6]306head[7]head[8]208head[9]179859

984: digit[2] = 4 → appended to the tail of bucket 4 (arrival order is kept → stable).

p009271033head[0]head[1]head[2]head[3]093head[4]984head[5]055head[6]306head[7]head[8]208head[9]179859

055: digit[2] = 5 → appended to the tail of bucket 5 (arrival order is kept → stable).

p271033head[0]head[1]head[2]head[3]093head[4]984head[5]055head[6]306head[7]head[8]208head[9]179859009

009: digit[2] = 9 → appended to the tail of bucket 9 (arrival order is kept → stable).

p033head[0]head[1]271head[2]head[3]093head[4]984head[5]055head[6]306head[7]head[8]208head[9]179859009

271: digit[2] = 1 → appended to the tail of bucket 1 (arrival order is kept → stable).

pNULLhead[0]head[1]271head[2]head[3]093033head[4]984head[5]055head[6]306head[7]head[8]208head[9]179859009

033: digit[2] = 3 → appended to the tail of bucket 3 (arrival order is kept → stable).

p271093033984055306208179859009head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

Gather: buckets are visited from 9 down to 0 and each is linked to the front of the list → the smallest bucket ends up first: 271 093 033 984 055 306 208 179 859 009.

for (int i = MAX_DIGIT - 1; i >= 0; i--) {
    for (int j = 0; j < RADIX; j++)
        head[j] = tail[j] = NULL;
    for (; p; p = p->next) {    // scatter
        int b = p->digit[i];
        if (!head[b]) head[b] = p;
        else tail[b]->next = p;
        tail[b] = p;
    }
    for (int j = RADIX - 1; j >= 0; j--) // gather
        if (head[j]) { tail[j]->next = p; p = head[j]; }
}
return p;

Radix Sort · Step through the code

Code Trace (continued): i = 1 and i = 0

p271093033984055306208179859009head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

Pass i = 1 (tens digit): the 10 buckets are emptied.

pNULLhead[0]306208009head[1]head[2]head[3]033head[4]head[5]055859head[6]head[7]271179head[8]984head[9]093

Scatter: each item goes to the tail of the bucket for its tens digit.

p306208009033055859271179984093head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

Gather: buckets are visited from 9 down to 0 and each is linked to the front of the list → the smallest bucket ends up first: 306 208 009 033 055 859 271 179 984 093.

p306208009033055859271179984093head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

Pass i = 0 (hundreds digit): the 10 buckets are emptied.

pNULLhead[0]009033055093head[1]179head[2]208271head[3]306head[4]head[5]head[6]head[7]head[8]859head[9]984

Scatter: each item goes to the tail of the bucket for its hundreds digit.

p009033055093179208271306859984head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

Gather: buckets are visited from 9 down to 0 and each is linked to the front of the list → the smallest bucket ends up first: 009 033 055 093 179 208 271 306 859 984.

p009033055093179208271306859984head[0]head[1]head[2]head[3]head[4]head[5]head[6]head[7]head[8]head[9]

After three passes the list is sorted; the head pointer is returned.

for (int i = MAX_DIGIT - 1; i >= 0; i--) {
    for (int j = 0; j < RADIX; j++)
        head[j] = tail[j] = NULL;
    for (; p; p = p->next) {    // scatter
        int b = p->digit[i];
        if (!head[b]) head[b] = p;
        else tail[b]->next = p;
        tail[b] = p;
    }
    for (int j = RADIX - 1; j >= 0; j--) // gather
        if (head[j]) { tail[j]->next = p; p = head[j]; }
}
return p;

Radix Sort

Radix Sort: Analysis

O(MAX_DIGIT · (RADIX + n + RADIX)) = O(MAX_DIGIT · (RADIX + n))

  • In the example O(3 · (10 + n)): it looks linear, but like counting sort it is a special case: the number of digits and the radix are taken as constants.
  • The data must be (a) nonnegative, (b) integers, (c) in a small range. With a larger range it still sorts, but the cost grows fast.
  • Extra memory: 2·RADIX pointers (the list is relinked in place).
From the lecture “Sorting only integers is a drawback. It can be adapted to floating-point types, but by the time you finish adapting it, the other algorithms will have finished sorting.”
passlist
initial179 208 306 093 859 984 055 009 271 033
i = 2271 093 033 984 055 306 208 179 859 009
i = 1306 208 009 033 055 859 271 179 984 093
i = 0009 033 055 093 179 208 271 306 859 984

Exam Questions

Exam Question: a Modified Radix Sort

A student, thinking the function should handle the most significant digit first, changed the outer loop to for (i = 0; i <= MAX_DIGIT-1; i++) and the gather loop to for (j = 0; j <= RADIX-1; j++). Run the function on the list below, write the list after each step and interpret the result.

p → 346 180 008 930 023 451 670 098 203 007

Solution
passlist
i = 0 (hundreds)930 670 451 346 203 180 008 023 098 007
i = 1 (tens)098 180 670 451 346 930 023 203 008 007
i = 2 (units)098 008 007 346 023 203 451 180 670 930

Change 1: the units digit acts as the most significant; the numbers are sorted if read backward (098, 008, 007 → 890, 800, 700 …). Change 2: the buckets are linked to the front in reverse order, so the order becomes descending. (The table was produced by simulation.)

Lab · C code step by step

Lab: Advanced Sorts

Summary

Complexity Summary and Comparison

algorithmbestaverageworstextra memorystablein place
Selectionn²n²n²O(1)noyes
Insertionnn²n²O(1)yesyes
Bubble (early exit)nn²n²O(1)yesyes
Shell (Knuth)n lg n—n3/2O(1)noyes
Quicksortn lg n1.39 n lg nn²O(lg n) stack*noyes
Merge sortn lg nn lg nn lg nO(n)yes**no
Heap sortn lg nn lg nn lg nO(1)noyes
Counting sortn + kn + kn + kO(n + k)yesno
Radix (d digits)d(n + r)d(n + r)d(n + r)O(r)***yes—

* If the smaller piece is processed first; otherwise O(n) in the worst case. ** With the classic merge. *** On a linked list. The lower bound for comparison sorting is ⌈lg n!⌉ ≈ n lg n − 1.44 n.

Summary

Which One When?

General purpose, in memory

Quicksort (median-of-3 + insertion sort on small pieces) or the library qsort.

Worst-case guarantee

Heap sort (in place) or introsort.

Stability / linked lists / disk

Merge sort: stable, merges sorted runs, the basis of external sorting.

Large records

Sort indices/links, not the records (linked merge sort).

Fixed-length integers/keys

Radix: d passes × O(n + r).

Small range

Counting sort.

Summary

Common Mistakes

1Forgetting the j == L test in the j loop of partition(): if the pivot is the smallest, the scan runs off the start of the array.
2Always taking a[R] as the pivot on sorted input: the depth becomes n and the stack overflows. Cure: median-of-3 or a random pivot.
3Pushing the pieces in arbitrary order in the non-recursive version: the stack grows to O(n).
4Calling malloc for b[] on every merge sort call (slow), or never allocating it.
5Mixing 1-based heap indices with a 0-based array in heapify: children 2i, 2i+1 are right only in a[1..n].
6return x - y; in a qsort comparator: overflow; or an inconsistent (non-transitive) comparator.

Summary

Exercises

  1. Write the array and the return value of partition(a, 0, 4) for [5, 9, 1, 7, 3].
  2. How many partition() calls does quicksort make on a sorted array with n = 8? What is the recursion depth?
  3. How many merge calls does merge sort make for 16 elements? How many comparisons at most?
  4. Write the array after building the heap for [4, 10, 3, 5, 1] (a[1..5]).
  5. Write the order of 170, 45, 75, 90, 802, 24, 2, 66 after the units pass of radix sort.
  6. Why is heap sort not stable? Give a small example.
Show answers
  1. v = 3. i stops at 0 (5), j at 2 (1) → swap: [1, 9, 5, 7, 3]; then i = 1 (9), j = 0 (1): crossed → a[1] ↔ a[4]: [1, 3, 5, 7, 9], returns 1.
  2. The pivot is always the largest: 7 partition() calls for sizes 8, 7, …, 2; recursion depth n − 1 = 7.
  3. 15 merge calls (n − 1). This merge makes exactly one comparison per element: lg 16 = 4 levels × 16 = 64.
  4. [10, 5, 3, 4, 1].
  5. 170 090 802 002 024 045 075 066 (stable distribution).
  6. [2a, 2b, 1]: the root 2a moves to the end → giving e.g. [1, 2b, 2a], so equal keys can change order; long-distance swaps do not preserve it.

Summary

Chapter Summary

Quicksort

Places the pivot and splits the problem: best n lg n, average 1.39 n lg n, worst n².

Improvements

Smaller piece first (lg n stack), median-of-3, insertion sort on small pieces; qsort and introsort.

Merge sort

n lg n in every case, O(n) extra memory; the linked version never copies large records.

Heap sort

Building the heap is O(n), sorting O(n lg n); in place but not stable.

Radix sort

Stable buckets from LSD to MSD: O(d(n + r)); nonnegative integers in a small range.

Analysis tools

Recurrences, iteration, the subtraction trick, differentiating a series: all the course's analysis tools meet here.

Summary

References

  • R. Sedgewick, Algorithms in C, Parts 1–4, Chapters 7–10 (quicksort, mergesort, priority queues/heapsort, radix sorting).
  • E. Horowitz, S. Sahni, S. Anderson-Freed, Fundamentals of Data Structures in C, Chapter 7.
  • T. H. Cormen et al., Introduction to Algorithms, Chapters 6–8; D. Musser, “Introspective Sorting and Selection Algorithms”, 1997.
  • ISO/IEC 9899 (the C standard), definition of qsort (§7.22.5.2).

The End

Chapter 20: Advanced Sorting Algorithms

“If this orphaned work stays with the master, / shame on the apprentice who does not carry it on!” — N. F. K.

End of the course · Course page
1 / 1 Sercan KÜLCÜ. All rights reserved.