Chapter 9: Parallel Algorithms
Multiple Choice · Models and Metrics
Question 1: PRAM Type
In a PRAM algorithm, in the same round many processors can read the same memory cell, but only one processor can write to a cell in that round. Which PRAM type is this?
C: CREW = Concurrent Read, Exclusive Write. In EREW reading is exclusive too; in CRCW concurrent writing is also allowed and conflicts are resolved by a rule (common value, priority, arbitrary). PRAM tree summation needs only EREW.
Multiple Choice · Models and Metrics
Question 2: Amdahl's Law
75% of a program can be parallelized. According to Amdahl's law, what is the speedup on p = 3 processors, and what is the upper bound as the number of processors goes to infinity?
A: S = 1 / ((1 − f) + f/p) = 1 / (0.25 + 0.25) = 2; as p → ∞, S → 1 / (1 − f) = 1 / 0.25 = 4. The value 2.25 = f · p speeds up only the parallel part and forgets the sequential part. The sequential 25% caps the speedup at 4.
Multiple Choice · Models and Metrics
Question 3: Gustafson's Law
The problem size is scaled up with the number of processors. With parallel fraction f = 0.8 and p = 10, what is the scaled speedup according to Gustafson's law?
D: S = (1 − f) + f · p = 0.2 + 8 = 8.2. 3.6 is Amdahl's result (1 / (0.2 + 0.08) ≈ 3.57): a fixed-size problem. 5.0 is Amdahl's limit (1 / 0.2). On large problems the sequential share shrinks, so parallelism pays off more.
Output Question · Models and Metrics
Question 4: Work, Depth and Brent
n = 256 numbers are summed with a balanced binary tree. What are the work W, the depth D and the parallelism W / D? On p = 16 processors, what upper bound does Brent's theorem give for Tp, and what are the lower bounds?
Show answer
- W = n − 1 = 255 additions; D = log₂ 256 = 8 rounds; parallelism 255 / 8 ≈ 31.9 (more than ≈ 32 processors brings no benefit).
- Brent: T₁₆ ≤ W/p + D = 255/16 + 8 ≈ 15.9 + 8 = 23.9 → 24 steps.
- Lower bounds: T₁₆ ≥ W/p ≈ 15.9 and T₁₆ ≥ D = 8 → T₁₆ ≥ 16.
16 ≤ T₁₆ ≤ 24: the Brent bound is at most twice the lower bound.
Multiple Choice · Models and Metrics
Question 5: Time Bounds
A parallel algorithm has work W = 1000 and depth D = 50. On p = 10 processors, which range for Tp is certainly true?
B: The lower bounds are Tp ≥ W/p = 100 and Tp ≥ D = 50 → T ≥ 100. Brent's upper bound is Tp ≤ W/p + D = 150. Exact equality with 100 holds only if there is no waiting at all; it is not guaranteed.
Step-by-Step Trace · PRAM Reduction
Question 6: Maximum with a PRAM
The PRAM tree summation is changed to take the maximum instead of the sum. How does the array change over the rounds, and what are the result and the number of rounds?
The summation tree finds the largest item with Math.max (any associative operation works). One processor holds each cell.
step = 1, i = 0: a[0] = max(3, 9) = 9. The other i in this round run at the same time.
step = 1, i = 2: a[2] = max(4, 1) = 4.
step = 1, i = 4: a[4] = max(5, 2) = 5.
step = 1, i = 6: a[6] = max(8, 6) = 8.
step = 2, i = 0: a[0] = max(9, 4) = 9. The other i in this round run at the same time.
step = 2, i = 4: a[4] = max(5, 8) = 8.
step = 4, i = 0: a[0] = max(9, 8) = 9.
step = 8 → the loop ends. The maximum is a[0] = 9: log₂ 8 = 3 rounds, 4 + 2 + 1 = 7 comparisons (W = n − 1, D = log₂ n). Faded cells are no longer read.
int[] a = {3, 9, 4, 1, 5, 2, 8, 6}; int n = a.length; for (int step = 1; step < n; step *= 2) { // all i in the same round in parallel for (int i = 0; i + step < n; i += 2 * step) a[i] = Math.max(a[i], a[i + step]); } // end of round: synchronization // the maximum is in a[0]
Output Question · PRAM Reduction
Question 7: n Not a Power of Two
The tree summation code is run on the array a = {4, 2, 7, 1, 3, 5} (n = 6). Which additions are made in each round; what are a[0] and the array at the end? How many additions and how many rounds in total?
int n = a.length; for (int step = 1; step < n; step *= 2) { // all i in the same round in parallel for (int i = 0; i + step < n; i += 2 * step) a[i] = a[i] + a[i + step]; } // end of round: synchronization
Show answer
- step = 1: i = 0, 2, 4 → a[0] = 6, a[2] = 8, a[4] = 8 → [6, 2, 8, 1, 8, 5]
- step = 2: i = 0 → a[0] = 6 + 8 = 14; for i = 4, 4 + 2 < 6 fails
- step = 4: i = 0 → a[0] = 14 + 8 = 22
The array [22, 2, 8, 1, 8, 5], a[0] = 22 = 4 + 2 + 7 + 1 + 3 + 5. 5 additions (= n − 1), 3 rounds (= ⌈log₂ 6⌉). i + step < n condition leaves the unpaired cell (a[4]) for the next round.
Classic Question · PRAM Reduction
Question 8: Cost-Optimal Summation
Why is PRAM tree summation with n processors not cost-optimal? How is it made cost-optimal with p = n / log₂ n processors? For n = 1024, compare the processor count, time and cost of the two versions.
Show answer
With n processors T = log₂ n and the cost is p · T = n log₂ n; the best sequential algorithm is Θ(n). As the rounds go on, most processors sit idle.
Fix: each processor first sums its own block of log₂ n items sequentially (O(log n)), then the n / log₂ n partial sums are combined with a tree (O(log n)). Cost (n / log n) · O(log n) = Θ(n).
| n = 1024 | p | T | Cost p · T |
|---|---|---|---|
| n processors | 1024 | 10 | 10,240 |
| n / log₂ n | ≈ 102 | ≈ 9 + 7 = 16 | ≈ 1,632 = Θ(n) |
In the second version the time grows a little (summing a block takes 9 steps, the tree over 102 partial sums takes ⌈log₂ 102⌉ = 7 rounds), but the processor-time drops by a factor of 6.
Step-by-Step Trace · Prefix Sums
Question 9: Tracing Hillis–Steele
Hillis–Steele prefix sums are applied to the array [3, 1, 4, 1, 5, 9, 2, 6]. Which values are computed in each round, and how many additions are made?
The goal is the inclusive prefix sum: for each i, x₀ + … + xᵢ. One processor holds each cell.
d = 1: the 7 processors with i ≥ 1 simultaneously compute y[i] = x[i] + x[i − 1] (the lines show the two cells read). Reads come from the old x.
End of round x = y: every cell now holds the sum of a 2-item window (the window is truncated at the start).
d = 2: the 6 processors with i ≥ 2 simultaneously compute y[i] = x[i] + x[i − 2] (the lines show the two cells read). Reads come from the old x.
End of round x = y: every cell now holds the sum of a 4-item window (the window is truncated at the start).
d = 4: the 4 processors with i ≥ 4 simultaneously compute y[i] = x[i] + x[i − 4] (the lines show the two cells read). Reads come from the old x.
Result [3, 4, 8, 9, 14, 23, 25, 31]. log₂ 8 = 3 rounds (D), but 7 + 6 + 4 = 17 additions (W = O(n log n)); the sequential algorithm makes 7 additions: not work-efficient.
int[] x = {3, 1, 4, 1, 5, 9, 2, 6}; int n = x.length; for (int d = 1; d < n; d *= 2) { int[] y = x.clone(); // copy at the start of the round for (int i = d; i < n; i++) // all in parallel y[i] = x[i] + x[i - d]; x = y; // synchronization }
Output Question · Prefix Sums
Question 10: Prefix Sums Without the Copy
In the Hillis–Steele code the per-round copy is removed and the computation is done on the same array. For {2, 4, 1, 3, 5} what is the output, and what is the correct prefix sum? What causes the error?
int[] x = {2, 4, 1, 3, 5}; int n = x.length; for (int d = 1; d < n; d *= 2) for (int i = d; i < n; i++) x[i] += x[i - d]; // no copy! System.out.println(Arrays.toString(x));
Show answer
Output: [2, 6, 9, 16, 26]; the correct answer is [2, 6, 7, 10, 15].
- d = 1: i is traversed in increasing order; when x[2] += x[1] is read, x[1] has already been updated in this round → a sequential prefix forms in a single round: [2, 6, 7, 10, 15].
- The rounds d = 2 and d = 4 add onto this correct result again: [2, 6, 9, 16, 24] → [2, 6, 9, 16, 26].
In a PRAM all reads of a round happen before the writes; in a sequential simulation the copy (or traversing i in decreasing order) provides this. In real parallel code, this corresponds to the race that appears when the synchronization (barrier) between rounds is forgotten.
Step-by-Step Trace · Pipelining
Question 11: Tracing the Pipeline
p = 4 processors, m = 3 rows; columns p0: (2, 1, 3), p1: (4, 2, 1), p2: (1, 5, 2), p3: (3, 1, 4). Which additions does the pipeline code perform at each step?
4 processors in a chain (p0 → p1 → p2 → p3), each holding a column of 3 rows. Goal: let pj, p0 … pj hold the row sums. The values of p0 are already final (green).
t = 0: p0→p1 row 0: 4 + 2 = 6.
t = 1: p0→p1 row 1: 2 + 1 = 3; p1→p2 row 0: 1 + 6 = 7.
t = 2: p0→p1 row 2: 1 + 3 = 4; p1→p2 row 1: 5 + 3 = 8; p2→p3 row 0: 3 + 7 = 10.
t = 3: p1→p2 row 2: 2 + 4 = 6; p2→p3 row 1: 1 + 8 = 9.
t = 4: p2→p3 row 2: 4 + 6 = 10. Done: m + p − 2 = 5 steps (naive method m(p − 1) = 9); p3 holds the row sums (10, 9, 10).
for (int t = 0; t < m + p - 2; t++) // all j at the same time for (int j = 0; j < p - 1; j++) { int r = t - j; if (r >= 0 && r < m) x[j + 1][r] += x[j][r]; }
Multiple Choice · Pipelining
Question 12: Number of Pipeline Steps
A pipeline on a linear chain has p = 6 processors and m = 50 rows. How many steps does the pipeline need, and how many does the naive method that carries the rows along the chain one by one take?
B: The pipeline takes m + p − 2 = 54 steps; the naive method carries each row in p − 1 = 5 steps: m(p − 1) = 250. 50 forgets the time to fill the pipeline. Efficiency m / (m + p − 2) = 50/54 ≈ 0.93: it approaches 1 when m ≫ p.
Multiple Choice · Fork/Join and MapReduce
Question 13: Depth of Parallel Merge Sort
For a Fork/Join merge sort that sorts the two halves in parallel and merges them sequentially, what are the depth D(n) and the parallelism W / D?
D: Since the two halves run in parallel, D(n) = D(n/2) + Θ(n) (the merge is sequential) = Θ(n); W = Θ(n log n) → parallelism Θ(log n): for n = 10⁶ more than ≈ 20 cores is hardly useful. B is the fully sequential version; C belongs to the version in which the merge is also parallel, with O(log n) depth.
Output Question · Fork/Join and MapReduce
Question 14: MapReduce Word Count
Words: "apple pear apple cherry pear apple banana cherry" (8 words). There are 3 map tasks; the chunk size is ⌊8/3⌋ = 2, and the last task takes all remaining words. What is the intermediate result of each map task, and what is the total after reduce?
protected Map<String, Integer> compute() { Map<String, Integer> result = new HashMap<>(); for (String word : chunk) result.merge(word, 1, Integer::sum); return result; } // Reduce for (Map<String, Integer> m : partial) m.forEach((k, v) -> total.merge(k, v, Integer::sum));
Show answer
| Task | Chunk | Intermediate result (map) |
|---|---|---|
| 1 | apple pear | apple 1, pear 1 |
| 2 | apple cherry | apple 1, cherry 1 |
| 3 | pear apple banana cherry | pear 1, apple 1, banana 1, cherry 1 |
Reduce (merge(k, v, Integer::sum)): apple 3, pear 2, cherry 2, banana 1.
Since each map task has its own HashMap, there is no shared state and no lock. The last task takes 4 words, so the load is unbalanced; because HashMap iteration order is undefined, the printed order may differ.
Step-by-Step Trace · Race Conditions
Question 15: Lost Update
Two threads increment the shared counter in the following interleaving: Th1 reads, Th2 reads, Th1 increments, Th2 increments, Th1 writes, Th2 writes. What is counter at the end?
Shared counter = 0. Two threads (Th1, Th2) each increment once; the expected result is 2. counter++ is actually these three steps.
Th1 reads: t₁ = 0.
Before Th1 writes, the scheduler switches to Th2: Th2 also reads 0.
Th1 increments its local copy: t₁ = 1 (memory is still 0).
Th2 increments too: t₂ = 1.
Th1 writes: counter = 1.
Th2 writes: counter = 1. Th1's increment was lost (lost update): the result is 1. Fix: AtomicInteger.incrementAndGet(), synchronized, or each task's local result plus a merge.
// Threads 1 and 2 run the same code: int t = counter; // read t = t + 1; // increment counter = t; // write
Multiple Choice · Race Conditions
Question 16: Outcome of the Race
Two threads each increment a shared int counter variable (initially 0), without synchronization, 1000 times each using counter++. When the program ends, which statement about counter is certainly true?
A: Every lost update erases an increment and no value is ever added: the result is ≤ 2000, but depending on timing (different on every run) it can be smaller. The bug may not show up on every run, which is why race conditions are hard to debug.
Multiple Choice · Race Conditions
Question 17: A Safe Visited Set
In a concurrent graph traversal many tasks share the same visited set. Which code guarantees that a node is traversed only once?
C: The concurrent set and the return value of add make the check and the insertion a single atomic operation. In B the set is safe, but another task can add the same node between contains and add (two separate steps). HashSet's internal structure can be corrupted by concurrent writes, so A and D are not safe.
Classic Question · Race Conditions
Question 18: A Loop That Cannot Be Parallelized
Which of the following two loops can be parallelized on a PRAM with O(log n) depth, and which cannot? Why? (a) for i: y[i] = y[i − 1] + x[i] (b) for i: x[i] = f(x[i − 1]), where f is an arbitrary function.
Show answer
Both seem to contain a data dependence (step i waits for step i − 1), but:
- (a) is a prefix sum. Because addition is associative, the parentheses can be rearranged: Hillis–Steele (W = O(n log n)) or Blelloch (W = O(n)) gives D = O(log n).
- (b) for an arbitrary f there is no associative "combine" operation: to know x[i], x[i − 1] must really have been computed. The dependence chain has length n: D = Θ(n), parallelism 1.
General rule: the depth is the longest dependence chain. An associative operation allows the chain to be turned into a tree (reduction, scan); an arbitrary recurrence does not.