Chapter 20: Advanced Sorting Algorithms
Introduction
Learning Outcomes
Explain the partition invariant; trace partition() and the recursion step by step.
Derive best n lg n, worst n² and average ≈ 1.39 n lg n from recurrences.
A non-recursive version (O(lg n) stack), median-of-3, insertion sort for small pieces; qsort in C.
Trace merge, the stack frames and the linked (index-based) version; O(n) extra memory.
Build a heap bottom-up in O(n) and sort in place in O(n lg n).
Implement LSD radix sort with linked-list buckets and interpret its cost.
Quicksort
Quicksort
- Developed by C. A. R. (Tony) Hoare in 1962.
- Divide and conquer: split the problem recursively into subproblems until they are small enough to solve directly.
- Time proportional to n lg n on average; n² in the worst case. Uses only a small auxiliary stack.
- Not stable, yet one of the fastest general-purpose algorithms known. The standard C library has
qsort.
int partition(int a[], int L, int R); void quicksort1(int a[], int L, int R) { if (R <= L) return; int i = partition(a, L, R); quicksort1(a, L, i - 1); quicksort1(a, i + 1, R); }
Quicksort
partition()
int partition(int a[], int L, int R) { int i = L - 1, j = R, v = a[R]; // pivot a[R] for (;;) { while (less(a[++i], v)) ; while (less(v, a[--j])) if (j == L) break; if (i >= j) break; SWAP(a[i], a[j]); } SWAP(a[i], a[R]); return i; }
- [2] The pivot (partitioning element) is
v = a[R]. - [4] i stops at the first element from the left that is greater than or equal to v. No bounds test is needed: at worst
a[R] = vstops it at i = R. - [6–7] j stops at an element from the right that is less than or equal to v;
j == Lkeeps this loop in bounds. - [9] Both elements are on the wrong side → swap. [8] When i and j meet, the loop ends; [11] the pivot goes to a[i]: left ≤ v ≤ right.
Quicksort · Step through the code
Code Trace: partition(a, 0, 9)
Pivot v = a[9] = 48. i = L − 1 = −1, j = R = 9: the scans start with ++i and --j.
a[0] = 37 < 48 → i keeps going.
a[1] = 15 < 48 → i keeps going.
a[2] = 24 < 48 → i keeps going.
a[3] = 33 < 48 → i keeps going.
a[4] = 49 ≥ 48 → stops at i = 4 (an element that belongs on the right).
a[8] = 72 > 48 → j moves left.
a[7] = 77 > 48 → j moves left.
a[6] = 51 > 48 → j moves left.
a[5] = 11 ≤ 48 → stops at j = 5.
i < j: a[4] and a[5] are on the wrong sides → swap: [37, 15, 24, 33, 11, 49, 51, 77, 72, 48].
a[5] = 49 ≥ 48 → stops at i = 5 (an element that belongs on the right).
a[4] = 11 ≤ 48 → stops at j = 4.
i = 5 ≥ j = 4: the indices crossed → leave the loop.
The pivot moves to a[5] and 5 is returned. In the sorted array, 48 would sit exactly here.
int partition(int a[], int L, int R) { int i = L - 1, j = R, v = a[R]; // pivot a[R] for (;;) { while (less(a[++i], v)) ; while (less(v, a[--j])) if (j == L) break; if (i >= j) break; SWAP(a[i], a[j]); } SWAP(a[i], a[R]); return i; }
Quicksort · Step through the code
Code Trace: quicksort1(a, 0, 9), the Call Stack
partition(0, 9): pivot 48 → position 5. Left a[0..4] and right a[6..9] are sorted separately.
partition(0, 4): pivot 11 → position 0. Left a[0..-1] and right a[1..4] are sorted separately.
quicksort1(0, -1): empty range → return.
partition(1, 4): pivot 37 → position 4. Left a[1..3] and right a[5..4] are sorted separately.
partition(1, 3): pivot 33 → position 3. Left a[1..2] and right a[4..3] are sorted separately.
partition(1, 2): pivot 24 → position 2. Left a[1..1] and right a[3..2] are sorted separately.
quicksort1(1, 1): one element, already in place → return.
quicksort1(3, 2): empty range → return.
quicksort1(4, 3): empty range → return.
quicksort1(5, 4): empty range → return.
partition(6, 9): pivot 49 → position 6. Left a[6..5] and right a[7..9] are sorted separately.
quicksort1(6, 5): empty range → return.
partition(7, 9): pivot 51 → position 7. Left a[7..6] and right a[8..9] are sorted separately.
quicksort1(7, 6): empty range → return.
partition(8, 9): pivot 77 → position 9. Left a[8..8] and right a[10..9] are sorted separately.
quicksort1(8, 8): one element, already in place → return.
quicksort1(10, 9): empty range → return.
int partition(int a[], int L, int R); void quicksort1(int a[], int L, int R) { if (R <= L) return; int i = partition(a, L, R); quicksort1(a, L, i - 1); quicksort1(a, i + 1, R); }
Quicksort
Assessment: Why Is It Quick?
Bubble and selection sort also put one element in place at each step, but the next subproblem still has n − 1 elements. Quicksort places the pivot and also splits the problem into two smaller pieces.
The quicksort1(a, 0, 9) trace makes 17 calls for 10 elements: every non-trivial call put a pivot in its final place (green cells in the trace).
Quicksort · Reading
Reading: Algorithms and Computers
Where does the word algorithm come from? Did al-Khwarizmi have a 4 GB laptop in the 9th century? ☺”
Mathematician and astronomer at the House of Wisdom in Baghdad. Author of al-Kitāb al-Mukhtaṣar fī Ḥisāb al-Jabr wa'l-Muqābala, which gave systematic solutions of linear and quadratic equations.
“Algebra” comes from the book's al-jabr method, and “algorithm” from Algoritmi, the Latin form of his name. His works introduced the decimal positional number system to the West.
Quicksort
Quicksort: Worst and Best Case
The pivot a[R] is always the largest and stays put; the next call is one element smaller:
n + (n−1) + … + 1 = n(n+1)/2 = O(n²)
At each step i and j advance n steps in total, leaving two subproblems of size n/2:
Cn = 2Cn/2 + n = 2kCn/2k + k·n = n lg n
Quicksort · Numerical check
Counts: quicksort1 on Sorted and Random Input
| n | sorted: comparisons | depth | random: comparisons | depth | n lg n |
|---|---|---|---|---|---|
| 10 | 63 | 9 | 36 | 5 | 33 |
| 100 | 5,148 | 99 | 721 | 9 | 664 |
| 1000 | 501,498 | 999 | 12,831 | 25 | 9,966 |
Comparisons inside partition() counted by a Python simulation (random input: seed 20).
≈ n²/2 comparisons and recursion depth n − 1: in C, a risk of stack overflow for large n.
Comparisons of order n lg n, depth only a few lg n.
Quicksort
Average-Case Analysis: the Recurrence
Cn = (n + 1) + (1/n) Σk=1n (Ck−1 + Cn−k), C0 = C1 = 0
- The split point k, 1 ≤ k ≤ n, is equally likely: running partition() on all n! orderings gives equal frequencies.
- n + 1: the comparisons with the pivot in partition(); a split at k leaves subarrays of (k−1) and (n−k) elements.
- The two sums contain the same terms: Σ(Ck−1 + Cn−k) = 2 Σk=0n−1 Ck ⇒ n·Cn = n(n+1) + 2 Σ Ck−1 … (1)
Quicksort
Average Case: the Subtraction Trick and the Result
- Replace n by n − 1 in (1): (n−1)Cn−1 = n(n−1) + 2 Σk=1n−1 Ck−1 … (2)
- (1) − (2): n·Cn − (n−1)Cn−1 = 2n + 2Cn−1 ⇒ n·Cn = (n+1)Cn−1 + 2n
- (dividing by n(n+1)) Cn/(n+1) = Cn−1/n + 2/(n+1): now it unrolls by iteration.
- Unrolling gives Cn ≈ 2(n+1)(1/4 + … + 1/(n+1)) ≤ 2(n+1) ln n.
Cn ≈ 2n ln n = 2 ln 2 · n lg n ≈ 1.39 n lg n
| n | Cn (from the recurrence) | 1.39 n lg n | ratio |
|---|---|---|---|
| 10 | 37.1 | 46.2 | 0.80 |
| 100 | 780.5 | 923.5 | 0.85 |
| 1000 | 12,318.6 | 13,852.4 | 0.89 |
Table: the recurrence computed exactly in Python. The ratio slowly approaches 1 (a lower-order −cn term).
Improving Quicksort
Non-Recursive Quicksort (quicksort2)
void quicksort2(int a[], int L, int R) { stack_init(); push2(L, R); while (!stack_empty()) { L = pop(); R = pop(); if (R <= L) continue; int i = partition(a, L, R); if (i - L > R - i) { // larger part first push2(L, i - 1); push2(i + 1, R); } else { push2(i + 1, R); push2(L, i - 1); } } }
#define STACK_MAX 64 /* > 2·lg n */ static int stack[STACK_MAX], top; void stack_init(void) { top = 0; } int stack_empty(void) { return top == 0; } void push(int x) { stack[top++] = x; } int pop(void) { return stack[--top]; } #define push2(A, B) \ do { push(B); push(A); } while (0)
- Goal: use our own stack instead of the system stack.
push2(A,B)pushes B first, then A, so L is popped first. - [7–11] The larger part is pushed first, the smaller one last: the last pushed is popped first, so the smaller part is always processed next → the stack depth stays O(lg n) (at most ≈ lg n pairs).
- The memory recurrence Tn = Tn/2 + 2 ⇒ Tn = 2 lg n (as in binary search).
Improving Quicksort · Step through the code
Code Trace: quicksort2(a, 0, 6), an Explicit Stack
The first problem (0, 6) is pushed: first R = 6, then L = 0.
(0, 6): partition → pivot 3 at position 2. The right part (3,6) is not smaller → pushed first, then the left (0,1); the smaller part is on top.
(0, 1): partition → pivot 2 at position 1. The left part (0,0) is larger → pushed first, then the right (2,1); the smaller part is on top.
(2, 1) popped: empty range → continue.
(0, 0) popped: one element → continue.
(3, 6): partition → pivot 7 at position 6. The left part (3,5) is larger → pushed first, then the right (7,6); the smaller part is on top.
(7, 6) popped: empty range → continue.
(3, 5): partition → pivot 5 at position 4. The right part (5,5) is not smaller → pushed first, then the left (3,3); the smaller part is on top.
(3, 3) popped: one element → continue.
(5, 5) popped: one element → continue.
The stack is empty → done: [1, 2, 3, 4, 5, 6, 7]. At most 3 (L, R) pairs waited on the stack.
void quicksort2(int a[], int L, int R) { stack_init(); push2(L, R); while (!stack_empty()) { L = pop(); R = pop(); if (R <= L) continue; int i = partition(a, L, R); if (i - L > R - i) { // larger part first push2(L, i - 1); push2(i + 1, R); } else { push2(i + 1, R); push2(L, i - 1); } } }
Improving Quicksort
Improved Quicksort: Median-of-3 + Insertion Sort
#define M 10 /* leave pieces smaller than this */ void quicksort3(int a[], int L, int R) { if (R - L <= M) return; SWAP(a[(L + R) / 2], a[R - 1]); if (a[R-1] < a[L]) SWAP(a[R-1], a[L]); // a[L] <= a[R-1] if (a[R] < a[L]) SWAP(a[R], a[L]); // a[L] <= a[R] if (a[R] < a[R-1]) SWAP(a[R], a[R-1]); // a[R-1] <= a[R] int i = partition(a, L + 1, R - 1); quicksort3(a, L, i - 1); quicksort3(a, i + 1, R); } void sort(int a[], int L, int R) { quicksort3(a, L, R); insertion(a, L, R); /* the small pieces left */ }
if (less(A, B)) SWAP(A, B) means “swap if A < B”; with the arguments in the order (a[L], a[R-1]) it puts the larger one in a[L] and breaks the partition (tested with random arrays: an element larger than the pivot stays on its left). So the arguments are given in reverse order: the result is a[L] ≤ a[R−1] ≤ a[R].Improving Quicksort
Why Median-of-3 and a Small-Piece Threshold?
a[L], the middle element and a[R] are put in order; their median moves to a[R−1] and becomes the pivot. Quicksort did worst and insertion sort best on sorted data: the n² case on sorted input disappears.
Since a[L] ≤ pivot ≤ a[R], partitioning runs on L+1..R−1; the end elements stop the scans.
Pieces with R − L ≤ M are left unsorted; one final insertion pass in sort() fixes them all in nearly linear time. About 10% faster for 5 ≤ M ≤ 25.
Median-of-3 rescues sorted and reverse-sorted input, but specially crafted bad inputs can still cause n² → the cure is introsort.
Improving Quicksort · Step through the code
Code Trace: Median-of-3 Setup, quicksort3(a, 0, 8)
R − L = 8 > M = 3 (M = 3 in this trace) → the piece is large; partition it.
The middle element a[4] is swapped with a[R−1] = a[7]: the candidate pivot 9 is now in a[7].
a[7] = 9 ≥ a[0] = 8 → no swap.
a[8] = 4 < a[0] = 8 → swap.
a[8] = 8 < a[7] = 9 → swap.
a[L] = 4 ≤ pivot 8 ≤ a[R] = 9. partition(a, 1, 7) → pivot at position 7; left ≤ 8 ≤ right.
The left a[0..6] comes first: R − L = 6 > M, so median-of-3 repeats; partition(a, 1, 5) → 3 and the array becomes [1, 3, 2, 4, 6, 5, 7, 8, 9]. The remaining pieces (R − L = 2 ≤ M) return unsorted; finally the right a[8..8] (R − L = 0 ≤ M) returns. The insertion pass in sort() finishes the job.
void quicksort3(int a[], int L, int R) { if (R - L <= M) return; SWAP(a[(L + R) / 2], a[R - 1]); if (a[R-1] < a[L]) SWAP(a[R-1], a[L]); // a[L] <= a[R-1] if (a[R] < a[L]) SWAP(a[R], a[L]); // a[L] <= a[R] if (a[R] < a[R-1]) SWAP(a[R], a[R-1]); // a[R-1] <= a[R] int i = partition(a, L + 1, R - 1); quicksort3(a, L, i - 1); quicksort3(a, i + 1, R); }
Reading
Reading: Mandatory Internships
C Library and Hybrids · Step through the code
Code Trace: qsort and the Comparison Function
The array is ready; sizeof a / sizeof a[0] = 20 / 4 = 5 elements (assuming a 4-byte int).
qsort gets the array's address, the element count, the element size and the address of the comparison function. qsort does not know the element type; it works with void *.
Sample call ascending(&37, &15) → 1: positive = the first element goes after. (The standard does not say which pairs qsort compares, or in what order.)
Sample call ascending(&15, &24) → -1: negative = the first element goes before. (The standard does not say which pairs qsort compares, or in what order.)
Sample call ascending(&24, &24) → 0: zero = equal (relative order not guaranteed). (The standard does not say which pairs qsort compares, or in what order.)
qsort returned: the array is in ascending order.
printf: a[0] = 15.
printf: a[1] = 24.
printf: a[2] = 33.
printf: a[3] = 37.
printf: a[4] = 49.
int ascending(const void *p, const void *q) { int x = *(const int *)p; int y = *(const int *)q; return (x > y) - (x < y); // -1, 0, +1 } int main(void) { int a[] = {37, 15, 24, 33, 49}; size_t n = sizeof a / sizeof a[0]; qsort(a, n, sizeof a[0], ascending); for (size_t i = 0; i < n; i++) printf("%d ", a[i]); return 0; }
C Library and Hybrids
Sorting Records (structs) with qsort
typedef struct { char name[20]; int grade; } student; int by_name(const void *p, const void *q) { const student *x = p, *y = q; return strcmp(x->name, y->name); // alphabetical } int by_grade_desc(const void *p, const void *q) { const student *x = p, *y = q; return (y->grade > x->grade) - (y->grade < x->grade); } /* qsort(s, n, sizeof s[0], by_name); */
Returns negative / zero / positive and must be consistent (a < b, b < c ⇒ a < c).
return x - y; is short but overflows for INT_MIN and 1; (x > y) - (x < y) is safe.
qsort need not be stable or use any set algorithm; if equal keys' order matters, add a secondary key.
C Library and Hybrids
Hybrid Sorts: Combining Algorithms
On small pieces the cost of recursion dominates; pieces below M are finished by insertion sort (quicksort3 + sort).
Starts with quicksort; if the recursion depth exceeds ≈ 2⌊lg n⌋, it switches to heap sort on that piece → worst case O(n lg n). Small pieces go to insertion sort. Common in C++ std::sort implementations.
Merge + insertion sort; finds already sorted runs in the data and merges them. Stable; Python's sort and Java's Arrays.sort for object arrays.
Good asymptotic behavior at scale (n lg n) + a low constant on small inputs (insertion) + insurance against the worst case.
Merge Sort
Merge Sort
- A list of 0 or 1 elements is already sorted.
- Otherwise split the list into two halves as equal as possible.
- Sort each half by applying merge sort recursively.
- Merge the two sorted sublists into one sorted list.
- John von Neumann, 1945; divide and conquer.
- Idea 1: sorting a small list takes fewer steps.
- Idea 2: building a sorted list from two sorted lists is cheaper than from two unsorted ones.
void mergesort(int a[], int b[], int L, int R) { if (R <= L) return; int m = (R + L) / 2; mergesort(a, b, L, m); mergesort(a, b, m + 1, R); merge(a, b, L, m, R); }
void merge(int a[], int b[], int L, int m, int R) { int i, j, k; for (i = m + 1; i > L; i--) b[i - 1] = a[i - 1]; for (j = m; j < R; j++) b[R + m - j] = a[j + 1]; for (k = L; k <= R; k++) if (less(b[j], b[i])) a[k] = b[j--]; else a[k] = b[i++]; }
The auxiliary array b[] (same size as a) is passed as a parameter; it cannot be used without being defined.
Merge Sort · Step through the code
Code Trace: merge(a, b, 0, 2, 6)
Loop 1: the left half a[0..2] is copied as is to b[0..2]; the loop exits with i = L = 0.
Loop 2: the right half is copied to b[3..6] in reverse → b increases from both ends toward the middle (bitonic). The loop exits with j = R = 6.
b[j] = 4 ≥ b[i] = 3 → a[0] = b[0]; i++.
b[j] = 4 < b[i] = 5 → a[1] = b[6]; j--.
b[j] = 5 ≥ b[i] = 5 → a[2] = b[1]; i++.
b[j] = 5 < b[i] = 7 → a[3] = b[5]; j--.
b[j] = 6 < b[i] = 7 → a[4] = b[4]; j--.
b[j] = 10 ≥ b[i] = 7 → a[5] = b[2]; i++.
b[j] = 10 ≥ b[i] = 10 → a[6] = b[3]; i++.
Done: a = [3, 4, 5, 5, 6, 7, 10]. At the end i = 4, j = 3: the two indices met in the middle; no bounds test was needed. Of the equal 5s, the left one (b[i]) was taken first.
void merge(int a[], int b[], int L, int m, int R) { int i, j, k; for (i = m + 1; i > L; i--) b[i - 1] = a[i - 1]; for (j = m; j < R; j++) b[R + m - j] = a[j + 1]; for (k = L; k <= R; k++) if (less(b[j], b[i])) a[k] = b[j--]; else a[k] = b[i++]; }
Merge sort table · Step through the code
Code Trace: mergesort, 9 merge Calls
mergesort(a, b, 0, 9): the left half is sorted completely first; the first merge happens only in the deepest call, (0, 1).
merge #1: a[0..0] and a[1..1] merged → a[0..1] sorted.
merge #2: a[0..1] and a[2..2] merged → a[0..2] sorted.
merge #3: a[3..3] and a[4..4] merged → a[3..4] sorted.
merge #4: a[0..2] and a[3..4] merged → a[0..4] sorted.
merge #5: a[5..5] and a[6..6] merged → a[5..6] sorted.
merge #6: a[5..6] and a[7..7] merged → a[5..7] sorted.
merge #7: a[8..8] and a[9..9] merged → a[8..9] sorted.
merge #8: a[5..7] and a[8..9] merged → a[5..9] sorted.
merge #9: a[0..4] and a[5..9] merged → a[0..9] sorted.
void mergesort(int a[], int b[], int L, int R) { if (R <= L) return; int m = (R + L) / 2; mergesort(a, b, L, m); mergesort(a, b, m + 1, R); merge(a, b, L, m, R); }
Merge Sort
The Tree of Stack Frames
Box: L, R, m · orange: a frame that calls merge, #k: which merge. It goes deep on the left first and merges on the way back.
Merge Sort
Question: What If the Two mergesort Lines Swap?
mergesort lines in the function and observe the table you get.Answer: the right halves merge first; same result, different intermediate tables
| merge | range | array |
|---|---|---|
| #1 | (8,9) | 89 67 59 62 23 93 24 20 28 60 |
| #2 | (5,6) | 89 67 59 62 23 24 93 20 28 60 |
| #3 | (5,7) | 89 67 59 62 23 20 24 93 28 60 |
| #4 | (5,9) | 89 67 59 62 23 20 24 28 60 93 |
| #5 | (3,4) | 89 67 59 23 62 20 24 28 60 93 |
| #6 | (0,1) | 67 89 59 23 62 20 24 28 60 93 |
| #7 | (0,2) | 59 67 89 23 62 20 24 28 60 93 |
| #8 | (0,4) | 23 59 62 67 89 20 24 28 60 93 |
| #9 | (0,9) | 20 23 24 28 59 60 62 67 89 93 |
Merge Sort
Merge Sort: Time and Space Complexity
an = a⌈n/2⌉ + a⌊n/2⌋ + n, a1 = 0
n = 2k: an = 2an/2 + n ⇒ an = Θ(n lg n)
- 16 elements split into 8 + 8, 17 into 8 + 9; merge takes n steps.
- Number of calls Tn = 2Tn/2 + 1 ⇒ 2n − 1 (19 calls for 10 elements, 9 of which merge).
- Independent of the input order: best = average = worst = n lg n.
- Drawback: O(n) extra memory. Avoiding it raises the cost; if no extra memory is allowed, heap sort is preferred.
- Because it merges sorted runs in sequence, it is the basis of external sorting on disk.
The classic merge (taking the left one on ties) is stable (quicksort and heap sort are not). The reverse-copy (bitonic) merge here is not stable: [2a, 2b, 2c, 2d] → [2a, 2b, 2d, 2c] (checked in Python).
Merge Sort
Sorting Alphabetic Data: Linked Merge Sort
typedef struct { int key; /* + large data fields */ int link; /* index of the next item, 0 = end */ } element; int mergesort(element a[], int L, int R) { if (R <= L) return L; /* one-item list */ int m = (R + L) / 2; int right = mergesort(a, m + 1, R); int left = mergesort(a, L, m); return merge(a, left, right); /* head index */ }
merge() links two sorted lists through their link fields and returns the head index.
- Copying large records (e.g. alphabetic data) is slow. The fix: build a linked list inside the array with a link field; only indices change.
- The data is in a[1..n]; a[0] is a temporary head during merge. Initially every link is 0.
- mergesort returns the head index of the sorted sublist.
- Common mistakes: returning a value from a
voidfunction; i1 instead of i2 in theelsebranch; returninga[c].linkinstead of a[0]. - In C the order in which arguments are evaluated is unspecified; here the order is written out explicitly.
Merge Sort · Step through the code
Code Trace: Linked merge (#9)
merge #9: first list (head 4) 38→71→77→119→122; second list (head 10) 13→22→41→43→115. last = 0 (a[0] is the temporary head).
38 > 13 → a[0].link = 10; last = 10; i2 = 7.
38 > 22 → a[10].link = 7; last = 7; i2 = 6.
38 ≤ 41 → a[7].link = 4; last = 4; i1 = 1.
71 > 41 → a[4].link = 6; last = 6; i2 = 9.
71 > 43 → a[6].link = 9; last = 9; i2 = 8.
71 ≤ 115 → a[9].link = 1; last = 1; i1 = 2.
77 ≤ 115 → a[1].link = 2; last = 2; i1 = 5.
119 > 115 → a[2].link = 8; last = 8; i2 = 0.
i2 = 0 (the second list is done) → a[8].link = i1 = 5; the head a[0].link = 10 is returned: 13→22→38→41→43→71→77→115→119→122.
int merge(element a[], int first, int second) { int last = 0; // a[0]: temp head int i1 = first, i2 = second; while (i1 && i2) if (a[i1].key <= a[i2].key) { a[last].link = i1; last = i1; i1 = a[i1].link; } else { a[last].link = i2; last = i2; i2 = a[i2].link; } a[last].link = i1 ? i1 : i2; return a[0].link; }
Merge Sort
Linked Merge Sort: How the link Fields Change
| # | [1] | [2] | [3] | [4] | [5] | [6] | [7] | [8] | [9] | [10] | returns | L, R |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| key | 71 | 77 | 122 | 38 | 119 | 41 | 22 | 115 | 43 | 13 | ||
| #1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 9 | 10 | 9,10 |
| #2 | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 0 | 0 | 9 | 7 | 6,7 |
| #3 | 0 | 0 | 0 | 0 | 0 | 8 | 6 | 0 | 0 | 9 | 7 | 6,8 |
| #4 | 0 | 0 | 0 | 0 | 0 | 9 | 6 | 0 | 8 | 7 | 10 | 6,10 |
| #5 | 0 | 0 | 0 | 5 | 0 | 9 | 6 | 0 | 8 | 7 | 4 | 4,5 |
| #6 | 2 | 0 | 0 | 5 | 0 | 9 | 6 | 0 | 8 | 7 | 1 | 1,2 |
| #7 | 2 | 3 | 0 | 5 | 0 | 9 | 6 | 0 | 8 | 7 | 1 | 1,3 |
| #8 | 2 | 5 | 0 | 1 | 3 | 9 | 6 | 0 | 8 | 7 | 4 | 1,5 |
| #9 | 2 | 8 | 0 | 6 | 3 | 9 | 4 | 5 | 1 | 7 | 10 | 1,10 |
Bold: the links changed by that merge. Sorted list: 13 → 22 → 38 → 41 → 43 → 71 → 77 → 115 → 119 → 122.
Exam Questions
Exam Question
The adaptation below sorts the element array a[1..9] by key. (i) Draw the stack frames of the recursive calls; where merge is called, write its return value. (ii) Fill in the execution table with merge numbers, return values and the link fields at each step.
int mergesort(element a[], int L, int R) { int m; if (R <= L) return L; m = (R + L) / 2; return merge(a, mergesort(a, m + 1, R), mergesort(a, L, m)); }
Keys: [1] ?, [2] ?, [3] 18, [4] −5, [5] 27, [6] 11, [7] ?, [8] 83, [9] 30. In place of the “?” on the exam sheet we use 42, 7, 55. merge() is as in linked merge sort; the printed version has i1 in the else branch (a typo; it should be i2).
Exam Questions
Exam Question (question 3): Solution
Solution
If the arguments are evaluated left to right (right half first), the call order is: (1,9) → (6,9) → (8,9) → (9,9) → (8,8) → (6,7) → (7,7) → (6,6) → (1,5) …
| # | [1] | [2] | [3] | [4] | [5] | [6] | [7] | [8] | [9] | ret. | L,R,m |
|---|---|---|---|---|---|---|---|---|---|---|---|
| #1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 9 | 8,9,8 |
| #2 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 8 | 6 | 6,7,6 |
| #3 | 0 | 0 | 0 | 0 | 0 | 9 | 8 | 0 | 7 | 6 | 6,9,7 |
| #4 | 0 | 0 | 0 | 5 | 0 | 9 | 8 | 0 | 7 | 4 | 4,5,4 |
| #5 | 0 | 1 | 0 | 5 | 0 | 9 | 8 | 0 | 7 | 2 | 1,2,1 |
| #6 | 0 | 3 | 1 | 5 | 0 | 9 | 8 | 0 | 7 | 2 | 1,3,2 |
| #7 | 0 | 3 | 5 | 2 | 1 | 9 | 8 | 0 | 7 | 4 | 1,5,3 |
| #8 | 7 | 6 | 5 | 2 | 9 | 3 | 8 | 0 | 1 | 4 | 1,9,5 |
Resulting list (head 4): -5 → 7 → 11 → 18 → 27 → 30 → 42 → 55 → 83. The table was produced by simulation.
Heap Sort
Heap Sort
void heapify(int a[], int root, int n) { int t = a[root], j; for (j = 2 * root; j <= n; j *= 2) { if (j < n && a[j] < a[j + 1]) j++; // larger if (t >= a[j]) break; // t fits here a[j / 2] = a[j]; // move child up } a[j / 2] = t; } void heapsort(int a[], int n) { /* a[1..n] */ int i; for (i = n / 2; i >= 1; i--) /* build heap */ heapify(a, i, n); for (i = n - 1; i >= 1; i--) { /* sort */ SWAP(a[1], a[i + 1]); /* largest to the end */ heapify(a, 1, i); /* fix the heap */ } }
- O(n lg n) like merge sort, but it uses no extra memory (in place).
- The array a[1..n] (index 0 unused) is viewed as a complete binary tree: the children of a[i] are a[2i] and a[2i+1].
heapify(a, root, n): with the left and right subtrees of root already max heaps, finds the right place for the root value.- Leaves are skipped: heapify nodes ⌊n/2⌋, ⌊n/2⌋−1, …, 1 from the back → a max heap.
- Then the largest at the root moves to the end and the heap is fixed for the remaining i elements.
- A
jorideclared in aforheader cannot be used after the loop (out of scope in C99).
Heap Sort · Step through the code
Code Trace: the Sorting Phase, SWAP + heapify(a, 1, i)
The input [26, 5, 77, 1, 61, 11, 59, 15, 48, 19] (Horowitz's example) was turned bottom-up into this max heap by heapify for i = 5…1 (5 moves in all). “Fine, we got that, but when does the array get sorted?” Now: each round the largest (77) at the root moves to the end and the rest of the heap is fixed.
i = 9: SWAP(a[1], a[10]) → the largest, 77, goes to the end; 5 at the root breaks the heap.
heapify(a, 1, 9): 5 sank to a[8]; the new root 61 is the largest remaining.
i = 8: SWAP(a[1], a[9]) → the largest, 61, goes to the end; 1 at the root breaks the heap.
heapify(a, 1, 8): 1 sank to a[7]; the new root 59 is the largest remaining.
i = 7: SWAP(a[1], a[8]) → the largest, 59, goes to the end; 5 at the root breaks the heap.
heapify(a, 1, 7): 5 sank to a[5]; the new root 48 is the largest remaining.
i = 6: SWAP(a[1], a[7]) → the largest, 48, goes to the end; 1 at the root breaks the heap.
heapify(a, 1, 6): 1 sank to a[6]; the new root 26 is the largest remaining.
i = 5: SWAP(a[1], a[6]) → the largest, 26, goes to the end; 1 at the root breaks the heap.
heapify(a, 1, 5): 1 sank to a[4]; the new root 19 is the largest remaining.
i = 4: SWAP(a[1], a[5]) → the largest, 19, goes to the end; 5 at the root breaks the heap.
heapify(a, 1, 4): 5 sank to a[2]; the new root 15 is the largest remaining.
i = 3: SWAP(a[1], a[4]) → the largest, 15, goes to the end; 1 at the root breaks the heap.
heapify(a, 1, 3): 1 sank to a[3]; the new root 11 is the largest remaining.
i = 2: SWAP(a[1], a[3]) → the largest, 11, goes to the end; 1 at the root breaks the heap.
heapify(a, 1, 2): 1 sank to a[2]; the new root 5 is the largest remaining.
i = 1: SWAP(a[1], a[2]) → the largest, 5, goes to the end; the single element left at the root (1) is already a heap.
heapify(a, 1, 1): j = 2 > 1, the loop never runs; 1 stays put. The array is sorted: [1, 5, 11, 15, 19, 26, 48, 59, 61, 77].
void heapify(int a[], int root, int n) { int t = a[root], j; for (j = 2 * root; j <= n; j *= 2) { if (j < n && a[j] < a[j + 1]) j++; // larger if (t >= a[j]) break; // t fits here a[j / 2] = a[j]; // move child up } a[j / 2] = t; } for (i = n - 1; i >= 1; i--) { /* sort */ SWAP(a[1], a[i + 1]); heapify(a, 1, i); }
Heap Sort
Heap Sort: Time Complexity
- Rough analysis: heapify makes at most lg n moves and is called n/2 times → building the heap costs n lg n.
- Careful analysis: k = number of levels ≈ lg(n+1); level i has at most 2i−1 nodes, each moving at most (k − i) levels:
Σi=1k 2i−1(k − i) = 2k − k − 1 ≤ 2n = O(n)
- The term Σ i·2i−1 comes from differentiating the geometric series S(y) = (yk+1 − 1)/(y − 1) and setting y = 2.
- The sorting phase: n − 1 calls of heapify(a, 1, i) → O(n lg n). Total O(n lg n), also in the worst case; O(1) extra memory; not stable.
- A bound of “≤ 2n” means O(n), not o(n) (being bounded by 2n is not little-o).
| k | Σ 2i−1(k−i) | 2k−k−1 | n = 2k−1 |
|---|---|---|---|
| 2 | 1 | 1 | 3 |
| 3 | 4 | 4 | 7 |
| 4 | 11 | 11 | 15 |
| 6 | 57 | 57 | 63 |
| 10 | 1013 | 1013 | 1023 |
Heap Sort
Heap Sort Variants, and Knowing Your Data
For fewer than 50,000 items heap sort is a good choice (≈ 2n lg n comparisons). For larger data there is bottom-up heapsort, which makes fewer comparisons, and theoretically even better variants.
The hole at the root sinks along the larger children to a leaf without comparing with t; then t is placed by moving up. Mostly one comparison per level instead of two.
Radix Sort
Radix Sort
- Sorting by several keys: e.g. people by surname, and those with the same surname by first name.
- Keys K0, K1, …, Kr−1: K0 is the most significant (MSD), Kr−1 the least significant (LSD). In the example, the surname is most significant.
- LSD radix sort: sort first by the least significant digit, then by more significant ones in turn, and finally by the MSD.
- Each pass must be stable: the order from the previous pass is kept for equal digits.
- It never compares numbers with each other → a non-comparison sort.
#define MAX_DIGIT 3 #define RADIX 10 typedef struct node *node_ptr; struct node { int digit[MAX_DIGIT]; /* 179 → {1, 7, 9} */ node_ptr next; };
K0: hundreds (MSD) · K1: tens · K2: units (LSD). Range [0, 999]. A typedef with no type name, like typedef struct node {…};, is meaningless.
Radix Sort · Step through the code
Code Trace: radixsort, i = 2 (units)
Pass i = 2 (units digit): the 10 buckets are emptied.
179: digit[2] = 9 → appended to the tail of bucket 9 (arrival order is kept → stable).
208: digit[2] = 8 → appended to the tail of bucket 8 (arrival order is kept → stable).
306: digit[2] = 6 → appended to the tail of bucket 6 (arrival order is kept → stable).
093: digit[2] = 3 → appended to the tail of bucket 3 (arrival order is kept → stable).
859: digit[2] = 9 → appended to the tail of bucket 9 (arrival order is kept → stable).
984: digit[2] = 4 → appended to the tail of bucket 4 (arrival order is kept → stable).
055: digit[2] = 5 → appended to the tail of bucket 5 (arrival order is kept → stable).
009: digit[2] = 9 → appended to the tail of bucket 9 (arrival order is kept → stable).
271: digit[2] = 1 → appended to the tail of bucket 1 (arrival order is kept → stable).
033: digit[2] = 3 → appended to the tail of bucket 3 (arrival order is kept → stable).
Gather: buckets are visited from 9 down to 0 and each is linked to the front of the list → the smallest bucket ends up first: 271 093 033 984 055 306 208 179 859 009.
for (int i = MAX_DIGIT - 1; i >= 0; i--) { for (int j = 0; j < RADIX; j++) head[j] = tail[j] = NULL; for (; p; p = p->next) { // scatter int b = p->digit[i]; if (!head[b]) head[b] = p; else tail[b]->next = p; tail[b] = p; } for (int j = RADIX - 1; j >= 0; j--) // gather if (head[j]) { tail[j]->next = p; p = head[j]; } } return p;
Radix Sort · Step through the code
Code Trace (continued): i = 1 and i = 0
Pass i = 1 (tens digit): the 10 buckets are emptied.
Scatter: each item goes to the tail of the bucket for its tens digit.
Gather: buckets are visited from 9 down to 0 and each is linked to the front of the list → the smallest bucket ends up first: 306 208 009 033 055 859 271 179 984 093.
Pass i = 0 (hundreds digit): the 10 buckets are emptied.
Scatter: each item goes to the tail of the bucket for its hundreds digit.
Gather: buckets are visited from 9 down to 0 and each is linked to the front of the list → the smallest bucket ends up first: 009 033 055 093 179 208 271 306 859 984.
After three passes the list is sorted; the head pointer is returned.
for (int i = MAX_DIGIT - 1; i >= 0; i--) { for (int j = 0; j < RADIX; j++) head[j] = tail[j] = NULL; for (; p; p = p->next) { // scatter int b = p->digit[i]; if (!head[b]) head[b] = p; else tail[b]->next = p; tail[b] = p; } for (int j = RADIX - 1; j >= 0; j--) // gather if (head[j]) { tail[j]->next = p; p = head[j]; } } return p;
Radix Sort
Radix Sort: Analysis
O(MAX_DIGIT · (RADIX + n + RADIX)) = O(MAX_DIGIT · (RADIX + n))
- In the example O(3 · (10 + n)): it looks linear, but like counting sort it is a special case: the number of digits and the radix are taken as constants.
- The data must be (a) nonnegative, (b) integers, (c) in a small range. With a larger range it still sorts, but the cost grows fast.
- Extra memory: 2·RADIX pointers (the list is relinked in place).
| pass | list |
|---|---|
| initial | 179 208 306 093 859 984 055 009 271 033 |
| i = 2 | 271 093 033 984 055 306 208 179 859 009 |
| i = 1 | 306 208 009 033 055 859 271 179 984 093 |
| i = 0 | 009 033 055 093 179 208 271 306 859 984 |
Exam Questions
Exam Question: a Modified Radix Sort
A student, thinking the function should handle the most significant digit first, changed the outer loop to for (i = 0; i <= MAX_DIGIT-1; i++) and the gather loop to for (j = 0; j <= RADIX-1; j++). Run the function on the list below, write the list after each step and interpret the result.
p → 346 180 008 930 023 451 670 098 203 007
Solution
| pass | list |
|---|---|
| i = 0 (hundreds) | 930 670 451 346 203 180 008 023 098 007 |
| i = 1 (tens) | 098 180 670 451 346 930 023 203 008 007 |
| i = 2 (units) | 098 008 007 346 023 203 451 180 670 930 |
Change 1: the units digit acts as the most significant; the numbers are sorted if read backward (098, 008, 007 → 890, 800, 700 …). Change 2: the buckets are linked to the front in reverse order, so the order becomes descending. (The table was produced by simulation.)
Lab · C code step by step
Lab: Advanced Sorts
Summary
Complexity Summary and Comparison
| algorithm | best | average | worst | extra memory | stable | in place |
|---|---|---|---|---|---|---|
| Selection | n² | n² | n² | O(1) | no | yes |
| Insertion | n | n² | n² | O(1) | yes | yes |
| Bubble (early exit) | n | n² | n² | O(1) | yes | yes |
| Shell (Knuth) | n lg n | — | n3/2 | O(1) | no | yes |
| Quicksort | n lg n | 1.39 n lg n | n² | O(lg n) stack* | no | yes |
| Merge sort | n lg n | n lg n | n lg n | O(n) | yes** | no |
| Heap sort | n lg n | n lg n | n lg n | O(1) | no | yes |
| Counting sort | n + k | n + k | n + k | O(n + k) | yes | no |
| Radix (d digits) | d(n + r) | d(n + r) | d(n + r) | O(r)*** | yes | — |
* If the smaller piece is processed first; otherwise O(n) in the worst case. ** With the classic merge. *** On a linked list. The lower bound for comparison sorting is ⌈lg n!⌉ ≈ n lg n − 1.44 n.
Summary
Which One When?
Quicksort (median-of-3 + insertion sort on small pieces) or the library qsort.
Heap sort (in place) or introsort.
Merge sort: stable, merges sorted runs, the basis of external sorting.
Sort indices/links, not the records (linked merge sort).
Radix: d passes × O(n + r).
Counting sort.
Summary
Common Mistakes
j == L test in the j loop of partition(): if the pivot is the smallest, the scan runs off the start of the array.malloc for b[] on every merge sort call (slow), or never allocating it.return x - y; in a qsort comparator: overflow; or an inconsistent (non-transitive) comparator.Summary
Exercises
- Write the array and the return value of
partition(a, 0, 4)for [5, 9, 1, 7, 3]. - How many partition() calls does quicksort make on a sorted array with n = 8? What is the recursion depth?
- How many merge calls does merge sort make for 16 elements? How many comparisons at most?
- Write the array after building the heap for [4, 10, 3, 5, 1] (a[1..5]).
- Write the order of 170, 45, 75, 90, 802, 24, 2, 66 after the units pass of radix sort.
- Why is heap sort not stable? Give a small example.
Show answers
- v = 3. i stops at 0 (5), j at 2 (1) → swap: [1, 9, 5, 7, 3]; then i = 1 (9), j = 0 (1): crossed → a[1] ↔ a[4]: [1, 3, 5, 7, 9], returns 1.
- The pivot is always the largest: 7 partition() calls for sizes 8, 7, …, 2; recursion depth n − 1 = 7.
- 15 merge calls (n − 1). This merge makes exactly one comparison per element: lg 16 = 4 levels × 16 = 64.
- [10, 5, 3, 4, 1].
- 170 090 802 002 024 045 075 066 (stable distribution).
- [2a, 2b, 1]: the root 2a moves to the end → giving e.g. [1, 2b, 2a], so equal keys can change order; long-distance swaps do not preserve it.
Summary
Chapter Summary
Places the pivot and splits the problem: best n lg n, average 1.39 n lg n, worst n².
Smaller piece first (lg n stack), median-of-3, insertion sort on small pieces; qsort and introsort.
n lg n in every case, O(n) extra memory; the linked version never copies large records.
Building the heap is O(n), sorting O(n lg n); in place but not stable.
Stable buckets from LSD to MSD: O(d(n + r)); nonnegative integers in a small range.
Recurrences, iteration, the subtraction trick, differentiating a series: all the course's analysis tools meet here.
Summary
References
- R. Sedgewick, Algorithms in C, Parts 1–4, Chapters 7–10 (quicksort, mergesort, priority queues/heapsort, radix sorting).
- E. Horowitz, S. Sahni, S. Anderson-Freed, Fundamentals of Data Structures in C, Chapter 7.
- T. H. Cormen et al., Introduction to Algorithms, Chapters 6–8; D. Musser, “Introspective Sorting and Selection Algorithms”, 1997.
- ISO/IEC 9899 (the C standard), definition of
qsort(§7.22.5.2).
The End
“If this orphaned work stays with the master, / shame on the apprentice who does not carry it on!” — N. F. K.
End of the course · Course page