Chapter 9: Parallel Algorithms
Introduction
Learning Outcomes
Explain the split–assign–combine structure of parallel algorithms, their benefits and their challenges (communication, synchronization, load balancing).
Know Flynn's taxonomy and the PRAM model; compute speedup, efficiency, cost, work and depth; apply Amdahl's law.
Run PRAM tree summation, parallel prefix sums, pipelining and the pipelined tree step by step, and count their steps.
Read examples of parallel merge sort with Fork/Join, MapReduce and data parallelism; recognize race conditions.
Introduction
Faster Together
- Some jobs take a very long time: big data analysis, scientific simulation, image processing.
- Parallel algorithms solve a problem using several processors (cores) at the same time.
- Complex tasks finish in much less time.
- Single-core clock speeds have nearly stopped rising; performance gains now come from the number of cores.
Introduction
Parallel Algorithms
- Like a group of people working together.
- Split a complex problem into smaller pieces.
- Speed up the work by assigning the pieces to different processors at the same time.
- When the processors finish, combine the results.
- The foundation of high-performance computing (HPC).
- Uses: big data analysis, scientific simulation, machine learning and more.
Introduction
Benefits and Drawbacks
Benefits
- Finish the same job sooner (speedup)
- Solve large problems that do not fit on one machine (scalability)
- Use the hardware (all cores, the GPU) efficiently
Drawbacks
- Splitting the problem and combining the results takes extra effort
- Communication and synchronization add overhead
- Load balancing: every processor should get equal work, and there must be enough processing power
- Debugging is hard (race conditions, deadlocks)
- Some problems are inherently sequential
Introduction
Applications
The core operation of scientific computing; every row (even every cell) of the result can be computed independently.
Edge detection, filtering, object recognition: the same operation on every pixel or block (data parallelism, GPU).
Flow simulation in aircraft, ship and vehicle design: the domain is split into a grid, each region on one processor.
The fitness of individuals in a population is evaluated independently and in parallel (scheduling, routing).
Models and Measures
Flynn's Taxonomy
| Class | Stands for | Meaning | Example |
|---|---|---|---|
| SISD | Single Instruction, Single Data | one instruction stream, one data stream | classic single-core processor |
| SIMD | Single Instruction, Multiple Data | the same instruction on many data items at once | vector instructions (AVX), GPU |
| MISD | Multiple Instruction, Single Data | different instructions on the same data | rare; fault-tolerant systems |
| MIMD | Multiple Instruction, Multiple Data | each processor runs its own program on its own data | multicore CPUs, clusters |
Models and Measures
Memory Architectures
- Shared memory: all processors see the same address space. Communication is cheap; concurrent access requires synchronization.
- Java threads,
ForkJoinPool, OpenMP.
- Distributed memory: each node has its own memory; data moves via messages.
- Scales well, but communication is expensive: MPI, Hadoop MapReduce.
Models and Measures
The PRAM Model
- PRAM (Parallel Random Access Machine): the theoretical model of parallel algorithms; the parallel counterpart of the sequential RAM model.
- p processors, a shared memory and a common clock: the processors take steps in lockstep.
- In each step a processor reads from memory → computes locally → writes to memory. Each takes O(1).
- Communication and synchronization costs are ignored, so the model measures an algorithm's inherent parallelism.
| Type | Read | Write |
|---|---|---|
| EREW | exclusive | exclusive |
| CREW | concurrent | exclusive |
| CRCW | concurrent | concurrent |
Models and Measures
Speedup, Efficiency, Cost
| Measure | Definition |
|---|---|
| T₁ | running time of the best sequential algorithm |
| Tₚ | parallel running time on p processors |
| Speedup | S = T₁ / Tₚ (ideal: S = p) |
| Efficiency | E = S / p (0 < E ≤ 1) |
| Cost | C = p · Tₚ |
S = 100 / 16 = 6.25
E = 6.25 / 8 ≈ 0.78
C = 8 · 16 = 128 s (> 100 s)
- Cost-optimal: C = Θ(T₁).
- Why S < p: communication, waiting, load imbalance, sequential parts.
- These bounds hold in the model; in practice cache effects or finding a target early in a search can give S > p (superlinear speedup).
Models and Measures
Amdahl's Law
- A fraction f of the program can be parallelized; the fraction 1 − f is sequential.
- S(p) = 1 / ((1 − f) + f / p)
- As p → ∞, S → 1 / (1 − f): the sequential part limits the speedup.
- f = 0.9, p = 8: S = 1 / (0.1 + 0.1125) ≈ 4.71; the limit is 10.
Models and Measures
Work and Depth
- Work W: the total number of operations = T₁ (time on one processor).
- Depth (span) D: the longest chain of dependences = T∞ (time with unlimited processors).
- Parallelism = W / D: the largest number of processors that can be used profitably.
- Lower bounds: Tₚ ≥ W / p and Tₚ ≥ D.
- Brent's theorem: Tₚ ≤ W / p + D.
- Work-efficient: W is of the same order as the best sequential algorithm.
W = n − 1 additions, D = log₂ n rounds.
n = 1024: W = 1023, D = 10, parallelism ≈ 102.
Brent with p = 32: T₃₂ ≤ 1023/32 + 10 ≈ 42 steps.
Summation on a PRAM
Problem: Parallel Summation
- Input: n numbers; output: their sum. The sequential algorithm performs n − 1 additions: O(n).
- In the sequential loop each addition waits for the previous one: a chain of length n − 1, no parallelism.
- Since addition is associative, we can regroup the parentheses:
((2 + 3) + (5 + 1)) + ((7 + 6) + (8 + 4)) → a balanced binary tree. - The sums on each level of the tree are independent → done in parallel in the same round.
- The same idea works for any associative operation: max, min, product, AND, OR (reduction).
Summation on a PRAM · Step by step
PRAM Summation Step by Step
Start: the 8 numbers are distributed one per processor (p₀ … p₇). Dashed boxes are the cells written in later rounds.
Round 1: 4 processors add adjacent pairs simultaneously: 2+3=5, 5+1=6, 7+6=13, 8+4=12.
Round 2: 2 processors combine partial sums at distance 2: 5+6=11, 13+12=25.
Round 3: p₀ computes 11+25 = 36. The 8 numbers were summed in log₂ 8 = 3 rounds (sequential: 7 steps).
Summation on a PRAM · Run the code step by step
Tree Summation: Code Trace
The array is created; n = 8. Each cell is held by one processor.
New round: step = 1. This round performs 4 additions in parallel.
a[0] = a[0] + a[1] = 2 + 3 = 5.
a[2] = a[2] + a[3] = 5 + 1 = 6.
a[4] = a[4] + a[5] = 7 + 6 = 13.
a[6] = a[6] + a[7] = 8 + 4 = 12.
New round: step = 2. This round performs 2 additions in parallel.
a[0] = a[0] + a[2] = 5 + 6 = 11.
a[4] = a[4] + a[6] = 13 + 12 = 25.
New round: step = 4. This round performs 1 addition in parallel.
a[0] = a[0] + a[4] = 11 + 25 = 36.
step = 8 → the loop ends. The sum is a[0] = 36; 3 rounds, 7 additions.
int[] a = {2, 3, 5, 1, 7, 6, 8, 4}; int n = a.length; for (int step = 1; step < n; step *= 2) { // all i in the same round run in parallel for (int i = 0; i + step < n; i += 2 * step) a[i] = a[i] + a[i + step]; } // end of round: synchronization // the sum is in a[0]
Summation on a PRAM
PRAM Summation: Analysis
| Measure | n processors (n/2 suffice) | p = n / log₂ n processors |
|---|---|---|
| Time Tₚ | log₂ n | O(log n) |
| Work W | n − 1 | n − 1 |
| Cost p · Tₚ | Θ(n log n) | Θ(n) |
| Cost-optimal? | no | yes |
- As the rounds go on, the number of active processors halves (4, 2, 1): most processors sit idle, so the cost is high.
- Fix: each processor first sums its own block of log₂ n numbers sequentially (O(log n)); then the n / log₂ n partial sums are combined with a tree (O(log n)).
- EREW suffices: in each round every cell is read/written by a single processor. Each round ends with a synchronization (barrier).
Summation on a PRAM
Parallel Prefix Sums
- Prefix sum (scan): yᵢ = x₀ + x₁ + … + xᵢ.
- Sequential: yᵢ = yᵢ₋₁ + xᵢ, O(n) but fully chained.
- Hillis–Steele: for d = 1, 2, 4, …, every i ≥ d does x[i] += x[i − d] in parallel.
- D = ⌈log₂ n⌉ rounds, W = O(n log n) → not work-efficient.
- The Blelloch algorithm (up-sweep/down-sweep) has W = O(n), D = O(log n).
- Uses: parallel sorting, compaction (filter),
Arrays.parallelPrefixin Java.
Summation on a PRAM · Step by step
Hillis–Steele Prefix Sums Step by Step
Input: 2 3 5 1 7 6 8 4. Goal: x₀ + … + xᵢ for every i (inclusive prefix sum).
d = 1: every processor with i ≥ 1 adds its left neighbor's value (7 additions at once).
d = 2: for i ≥ 2, add the value 2 positions to the left. Each cell now holds the sum of a 4-element window.
d = 4: result 2 5 10 11 18 24 32 36. Done in log₂ 8 = 3 rounds, but with 7+6+4 = 17 additions in total.
Summation on a PRAM · Run the code step by step
Blelloch Prefix Sums: Code Trace
Input 2 3 5 1 7 6 8 4 (n = 8). The array serves as an implicit binary tree: first up (reduction), then down.
d = 1: x[1] += x[0], x[3] += x[2], x[5] += x[4], x[7] += x[6] (in parallel in one round, 4 additions).
d = 2: x[3] += x[1], x[7] += x[5] (in parallel in one round, 2 additions).
d = 4: x[7] += x[3] (in parallel in one round, 1 addition). The root x[7] = 36 is the total.
The total 36 is at the root. The root is set to 0: in the down-sweep every node carries "the sum of everything to its left".
d = 4: in each pair (i − d, i), the left child takes the parent's value and the right child takes parent + the old left value.
d = 2: in each pair (i − d, i), the left child takes the parent's value and the right child takes parent + the old left value.
d = 1: the last round; 4 pairs are updated at once.
Result 0 2 5 10 11 18 24 32: the exclusive prefix sums. 2 · (n − 1) = 14 operations, O(n) work, depth 2 log₂ n = 6 rounds.
// up-sweep; x = 2 3 5 1 7 6 8 4 for (int d = 1; d < n; d *= 2) for (int i = 2*d - 1; i < n; i += 2*d) x[i] += x[i - d]; x[n - 1] = 0; // down-sweep (inner loops in parallel) for (int d = n / 2; d >= 1; d /= 2) for (int i = 2*d - 1; i < n; i += 2*d) { int t = x[i - d]; x[i - d] = x[i]; x[i] += t; } // x: exclusive prefix sums
Pipelining
The Idea of Pipelining
- A pipeline splits the work into consecutive stages; each stage is a processor.
- Like a factory assembly line: while one job moves to stage 2, stage 1 starts on the next job.
- A single job does not get faster, but the number of jobs finished per unit time (throughput) increases.
- k stages, m jobs: k · m steps sequentially; k + m − 1 steps with a pipeline.
- Some stages are idle while the pipeline fills at the start and drains at the end.
3 stages, 4 jobs: 3 + 4 − 1 = 6 steps instead of 3 · 4 = 12.
Pipelining
The Pipeline Problem
- Five processors p₀ … p₄ form a linear chain; each processor can send messages only to its right neighbor.
- Each pj holds a column of 4 values (rows 0 … 3).
- Goal: for every row r, the value at pj should become the sum of the values of p₀ … pj in that row (row-wise prefix sums). Then p₄ holds the sum of every row.
- In one step each processor sends one value; the receiver adds it to its own.
- Naive method: first move row 0 along the chain (4 steps), then row 1… → 4 · 4 = 16 steps.
- Pipeline: pj sends row r at time t = r + j; the rows flow through the chain one after another.
Pipelining · Step by step
Pipelining Step by Step: 5 Processors, 4 Rows
Each pj holds a 4-row column. Goal: in every row, pj holds the sum of the values of p₀ … pj. p₀'s values are already final.
Step 1: p₀→p₁ row 0: 2+3=5.
Step 2: p₀→p₁ row 1: 1+7=8, p₁→p₂ row 0: 5+3=8.
Step 3: p₀→p₁ row 2: 7+2=9, p₁→p₂ row 1: 8+4=12, p₂→p₃ row 0: 8+6=14.
Step 4: p₀→p₁ row 3: 4+9=13, p₁→p₂ row 2: 9+7=16, p₂→p₃ row 1: 12+1=13, p₃→p₄ row 0: 14+8=22.
Step 5: p₁→p₂ row 3: 13+3=16, p₂→p₃ row 2: 16+0=16, p₃→p₄ row 1: 13+6=19.
Step 6: p₂→p₃ row 3: 16+2=18, p₃→p₄ row 2: 16+3=19.
Step 7: p₃→p₄ row 3: 18+5=23. Done in 7 steps; p₄ holds the row sums (22 19 19 23).
Pipelining
Pipelining: Analysis
| step | p₀→p₁ | p₁→p₂ | p₂→p₃ | p₃→p₄ |
|---|---|---|---|---|
| 1 | row 0 | — | — | — |
| 2 | row 1 | row 0 | — | — |
| 3 | row 2 | row 1 | row 0 | — |
| 4 | row 3 | row 2 | row 1 | row 0 |
| 5 | — | row 3 | row 2 | row 1 |
| 6 | — | — | row 3 | row 2 |
| 7 | — | — | — | row 3 |
for (int t = 0; t < m + p - 2; t++) // all j at the same time for (int j = 0; j < p - 1; j++) { int r = t - j; if (r >= 0 && r < m) x[j + 1][r] += x[j][r]; }
- Number of steps: m + p − 2 = 4 + 5 − 2 = 7 (naive: m(p − 1) = 16).
- Efficiency: m(p − 1) additions / ((p − 1)(m + p − 2)) = m / (m + p − 2) = 4/7 ≈ 57%.
- For m ≫ p the efficiency → 1.
Pipelining · Run the code step by step
Pipelining: Code Trace
p = 5 processors, m = 4 rows. Each column holds one processor's values; row r reaches pj at time t = r + j.
t = 0: p₀→p₁ row 0: 2 + 3 = 5.
t = 1: p₀→p₁ row 1: 1 + 7 = 8; p₁→p₂ row 0: 5 + 3 = 8.
t = 2: p₀→p₁ row 2: 7 + 2 = 9; p₁→p₂ row 1: 8 + 4 = 12; p₂→p₃ row 0: 8 + 6 = 14.
t = 3: p₀→p₁ row 3: 4 + 9 = 13; p₁→p₂ row 2: 9 + 7 = 16; p₂→p₃ row 1: 12 + 1 = 13; p₃→p₄ row 0: 14 + 8 = 22. The pipeline is full: all 4 links are busy.
t = 4: p₁→p₂ row 3: 13 + 3 = 16; p₂→p₃ row 2: 16 + 0 = 16; p₃→p₄ row 1: 13 + 6 = 19.
t = 5: p₂→p₃ row 3: 16 + 2 = 18; p₃→p₄ row 2: 16 + 3 = 19.
t = 6: p₃→p₄ row 3: 18 + 5 = 23.
Done in m + p − 2 = 7 steps; p₄ holds the row sums 22 19 19 23. The naive method would take 16 steps.
for (int t = 0; t < m + p - 2; t++) // all j at the same time for (int j = 0; j < p - 1; j++) { int r = t - j; if (r >= 0 && r < m) x[j + 1][r] += x[j][r]; }
Pipelined Tree
Reduction on a Tree
- The processors are connected as a tree; each node sends its value to its parent, which adds it to its own. The result accumulates at the root.
- If the tree has depth h, reducing a single value takes h rounds (h = 3 in our example).
- Now each processor holds not one number but a vector with k components (k = 3 here: green, blue, red). Each component is summed separately.
- Naive method: reduce the components one after another → k · h = 9 rounds.
- Pipelined tree: while one component moves up, the next one sets out from the leaves.
- Rule: a node can receive at most one message per round (and may also send one in the same round).
Pipelined Tree · Step by step
Pipelined Tree Step by Step
Each of the 7 processors holds a 3-component vector (green, blue, red). Goal: the sum of each component at the root p₀; values flow bottom-up.
Round 1: green p₃→p₂: 9+8=17, p₄→p₁: 7+3=10, p₆→p₅: 2+3=5.
Round 2: green p₂→p₁: 17+10=27, p₅→p₀: 5+3=8.
Round 3: green p₁→p₀: 27+8=35; blue p₃→p₂: 5+5=10, p₄→p₁: 5+7=12, p₆→p₅: 1+4=5.
Round 4: blue p₂→p₁: 10+12=22, p₅→p₀: 5+6=11.
Round 5: blue p₁→p₀: 22+11=33; red p₃→p₂: 8+3=11, p₄→p₁: 2+5=7, p₆→p₅: 7+5=12.
Round 6: red p₂→p₁: 11+7=18, p₅→p₀: 12+1=13.
Round 7: red p₁→p₀: 18+13=31. The root holds (35, 33, 31): done in 7 rounds.
- Edge color = component carried this round; arrows point to the parent.
- Thick border = value updated this round.
- Each component reaches the root in 3 stages:
(1) p₃→p₂, p₄→p₁, p₆→p₅
(2) p₂→p₁, p₅→p₀
(3) p₁→p₀. - A new component starts every 2 rounds.
Pipelined Tree
Pipelined Tree: Analysis
| round | stage 1 p₃→p₂, p₄→p₁, p₆→p₅ | stage 2 p₂→p₁, p₅→p₀ | stage 3 p₁→p₀ |
|---|---|---|---|
| 1 | green | — | — |
| 2 | — | green | — |
| 3 | blue | — | green |
| 4 | — | blue | — |
| 5 | red | — | blue |
| 6 | — | red | — |
| 7 | — | — | red |
- Why every 2 rounds? p₁ receives from two children (p₄ and p₂); since it can receive one message per round, each component keeps p₁ busy for 2 rounds. The same holds for p₀.
- Total rounds: h + (k − 1) · 2 = 3 + 2 · 2 = 7 (naive: k · h = 9).
- As k grows: 2k + 1 ≈ 2k rounds versus 3k for the naive method. In general, rounds ≈ h + (k − 1) · (maximum number of children).
- At the root: green 35, blue 33, red 31.
Kinds of Parallel Algorithms
Kinds of Parallel Algorithms
The problem is split into independent subproblems that are solved simultaneously on different cores, and the results are combined. Parallel merge sort, quicksort, matrix multiplication.
Big data is processed on distributed nodes: the map step emits intermediate key–value pairs, and the reduce step combines them. Hadoop, Spark; counting, indexing, log analysis.
The same computation is applied to different parts of the data at the same time (SIMD instructions, GPU). Scientific simulation, image processing, numerical computing.
Kinds of Parallel Algorithms
Parallel Merge Sort
class SortTask extends RecursiveTask<int[]> { private final int[] arr; SortTask(int[] arr) { this.arr = arr; } protected int[] compute() { if (arr.length <= 1) return arr; int mid = arr.length / 2; var left = new SortTask( Arrays.copyOfRange(arr, 0, mid)); var right = new SortTask( Arrays.copyOfRange(arr, mid, arr.length)); left.fork(); // start in parallel right.fork(); return merge(left.join(), right.join()); } } // usage: int[] s = new ForkJoinPool().invoke( new SortTask(arr));
RecursiveTask: a Fork/Join task that returns a result.fork(): submits the subtask to the pool (asynchronously).join(): waits for the subtask to finish and gets its result.merge: the usual sequential merge, O(n).ForkJoinPooluses work stealing to hand work to idle cores.
Kinds of Parallel Algorithms · Step by step
Parallel Merge Sort: 2 3 5 1 7 6 8 4
Task: sort the 8-element array. SortTask is the root task.
The task splits the array in two and submits both subtasks to the pool with fork(); the halves can be processed at the same time.
Each subtask splits again: now 4 tasks can run in parallel.
Base case: one-element arrays are already sorted (8 leaf tasks).
join(): each parent waits for its two children's results and merges them: [2 3] [1 5] [6 7] [4 8].
One level up, two merges run in parallel: [1 2 3 5] and [4 6 7 8].
The root task does the final merge alone: [1 2 3 4 5 6 7 8]. This sequential merge makes the depth (span) Θ(n).
Kinds of Parallel Algorithms
Parallel Sorting: Analysis
| Version | Work W | Depth D | Parallelism W / D |
|---|---|---|---|
| Sequential merge sort | Θ(n log n) | Θ(n log n) | 1 |
| Parallel recursion + sequential merge | Θ(n log n) | Θ(n) | Θ(log n) |
| Parallel recursion + parallel merge (CLRS) | Θ(n log n) | Θ(log³ n) | Θ(n / log² n) |
- Depth recurrence: D(n) = D(n/2) + Θ(n) = Θ(n), because the two halves run in parallel but the merge is sequential.
- Parallelism is only Θ(log n): for n = 10⁶, more than ≈ 20 cores hardly help.
- In practice, switch to a sequential sort for small parts (e.g., < 1000 elements); otherwise task-creation overhead eats the gain.
- Ready-made:
Arrays.parallelSort(arr).
Kinds of Parallel Algorithms
MapReduce
- Map: each chunk is processed independently and emits intermediate key–value pairs (word → count).
- Shuffle: equal keys are gathered at the same reducer.
- Reduce: the values of each key are combined (summed).
- In the example: 13 words, 4 threads → chunk size ⌊13/4⌋ = 3; the last chunk has 4 words.
Kinds of Parallel Algorithms
MapReduce: Code
// Map: count the words in one chunk protected Map<String, Integer> compute() { Map<String, Integer> result = new HashMap<>(); for (String word : chunk) result.merge(word, 1, Integer::sum); return result; } // Main flow for (MapTask g : mapTasks) g.fork(); List<Map<String, Integer>> partial = new ArrayList<>(); for (MapTask g : mapTasks) partial.add(g.join()); // Reduce: combine the partial results Map<String, Integer> total = new HashMap<>(); for (Map<String, Integer> m : partial) m.forEach((k, v) -> total.merge(k, v, Integer::sum));
- Each map task has its own HashMap: no shared state, no locks needed.
- Result: apple 5, banana 5, orange 2, cherry 1.
- Here reduce is a single task; in real MapReduce the keys are partitioned among many reducers (hash(k) mod r).
Kinds of Parallel Algorithms · Run the code step by step
MapReduce: Code Trace
The 13 words were split into 4 chunks; for each chunk a MapTask is submitted to the pool with fork(), and the tasks run in parallel.
join(): task 0's own HashMap is ready → banana 1, apple 2. No shared state, no locks needed.
join(): task 1's own HashMap is ready → banana 1, orange 1, apple 1. No shared state, no locks needed.
join(): task 2's own HashMap is ready → orange 1, banana 2. No shared state, no locks needed.
join(): task 3's own HashMap is ready → banana 1, cherry 1, apple 2. No shared state, no locks needed.
Reduce begins: an empty total map.
partial[0] merged: equal keys are added with Integer::sum.
partial[1] merged: equal keys are added with Integer::sum.
partial[2] merged: equal keys are added with Integer::sum.
partial[3] merged: equal keys are added with Integer::sum. Result: orange 2, banana 5, cherry 1, apple 5.
// Map: count the words in one chunk protected Map<String, Integer> compute() { Map<String, Integer> result = new HashMap<>(); for (String word : chunk) result.merge(word, 1, Integer::sum); return result; } // Main flow for (MapTask g : mapTasks) g.fork(); List<Map<String, Integer>> partial = new ArrayList<>(); for (MapTask g : mapTasks) partial.add(g.join()); // Reduce: combine the partial results Map<String, Integer> total = new HashMap<>(); for (Map<String, Integer> m : partial) m.forEach((k, v) -> total.merge(k, v, Integer::sum));
Kinds of Parallel Algorithms
Data Parallelism: Matrix Multiplication
// each row is a separate task class RowTask extends RecursiveAction { protected void compute() { for (int j = 0; j < n; j++) { C[row][j] = 0; for (int k = 0; k < n; k++) C[row][j] += A[row][k] * B[k][j]; } } } try (var pool = new ForkJoinPool()) { for (int i = 0; i < n; i++) pool.submit(new RowTask(i)); } // close(): waits for all tasks
| C = A · B | ||
|---|---|---|
| 30 | 24 | 18 |
| 84 | 69 | 54 |
| 138 | 114 | 90 |
- A = [[1,2,3],[4,5,6],[7,8,9]], B = [[9,8,7],[6,5,4],[3,2,1]].
- Rows are independent: n tasks, each with Θ(n²) work. W = Θ(n³), D = Θ(n²) (per row).
- Each task writes only to its own row → no race.
Kinds of Parallel Algorithms · Run the code step by step
Row Task: Code Trace
This task computes only row 1 of C: row 1 of A (4, 5, 6) times each column of B. The other rows are in other tasks.
j = 0: C[1][0] starts at 0.
k = 0: C[1][0] += A[1][0] · B[0][0] = 4 · 9 → 36.
k = 1: C[1][0] += A[1][1] · B[1][0] = 5 · 6 → 66.
k = 2: C[1][0] += A[1][2] · B[2][0] = 6 · 3 → 84.
j = 1: 4 · 8 + 5 · 5 + 6 · 2 = 69.
j = 2: 4 · 7 + 5 · 4 + 6 · 1 = 54.
Task done: C[1] = (84, 69, 54). Each task writes only to its own row, so there is no race condition.
// RowTask, row = 1 protected void compute() { for (int j = 0; j < n; j++) { C[row][j] = 0; for (int k = 0; k < n; k++) C[row][j] += A[row][k] * B[k][j]; } }
Kinds of Parallel Algorithms
Race Conditions
Two threads increment the shared counter at the same time. counter++ is really three steps: read → increment → write.
| time | Thread 1 | Thread 2 | counter |
|---|---|---|---|
| 1 | read (0) | 0 | |
| 2 | read (0) | 0 | |
| 3 | write 0 + 1 | 1 | |
| 4 | write 0 + 1 | 1 |
Expected 2, got 1: a lost update.
- Race condition: the result depends on the timing of the threads; the bug may not show up on every run.
- Fixes: a
synchronizedblock / lock,AtomicInteger.incrementAndGet(), concurrent collections (ConcurrentHashMap). - Best of all is to avoid shared state: each task produces its own partial result, and the results are combined afterward (reduction, MapReduce).
- Locks are not free either: they create sequential regions (Amdahl) and risk deadlock.
Kinds of Parallel Algorithms
Concurrent Graph Traversal
- Idea:
fork()a new traversal task for each unvisited neighbor; subtrees are traversed in parallel. - The tasks share the
visitedset. If it is an ordinaryHashSet, concurrent access is not safe. - Moreover,
contains+addare two separate steps: two tasks may visit the same node together.
Set<Node> visited = ConcurrentHashMap.newKeySet(); if (!visited.add(neighbor)) continue; // atomic check+add
Level-synchronous BFS: all nodes of a level are expanded in parallel, with a barrier at the end of each level. D = O(diameter) on an ideal PRAM with enough processors; the number of levels does not exceed the diameter of the part reachable from the source (O(diameter · log n) in a realistic model).
Kinds of Parallel Algorithms
Parallelism Tools in Java
| Tool | When? |
|---|---|
Thread, Runnable | lowest level; managed by hand, rarely used directly |
ExecutorService | running independent tasks in a fixed thread pool |
ForkJoinPool, RecursiveTask | divide and conquer; recursive tasks with work stealing |
list.parallelStream() | data-parallel map / filter / reduce (uses the common ForkJoinPool) |
Arrays.parallelSort | sorting large arrays in parallel |
Arrays.parallelPrefix | parallel prefix sums (scan) |
AtomicInteger, ConcurrentHashMap | safe access when a shared counter/set is needed |
Map<String, Long> counts = words.parallelStream()
.collect(Collectors.groupingByConcurrent(w -> w, Collectors.counting()));Summary
Summary
| Topic | Key points |
|---|---|
| Idea | split → assign to processors → solve simultaneously → combine; the price is communication, synchronization, load balancing |
| Models | Flynn: SISD/SIMD/MISD/MIMD; PRAM: EREW, CREW, CRCW; shared / distributed memory |
| Measures | S = T₁/Tₚ, E = S/p, C = p·Tₚ; Amdahl S ≤ 1/(1 − f); Brent Tₚ ≤ W/p + D |
| PRAM summation | log₂ n rounds, W = n − 1; cost-optimal with p = n/log n |
| Prefix sums | Hillis–Steele: log₂ n rounds, O(n log n) work; Blelloch: O(n) work |
| Pipelining | p processors, m rows: m + p − 2 steps (naive m(p − 1)) |
| Pipelined tree | h + (k − 1)·2 rounds (example 7; naive k·h = 9) |
| Java | Fork/Join (fork, join), MapReduce, one task per row; races on shared state → atomic/concurrent structures |
Summary
Common Mistakes
ConcurrentHashMap, AtomicInteger, or local results + combining.fork() for every element inflates task-creation overhead. Parts below a threshold are processed sequentially.Summary
Sample Problems
- n = 16 numbers are summed with a PRAM tree. How many rounds does it take, how many processors work in the first round, and how many additions are made in total?
- How many steps does a pipeline with p = 5 processors and m = 100 rows need? Compare with the naive method.
- In the chapter's pipelined tree example (7 processors, h = 3), how many rounds would be needed if each processor held a vector with k = 10 components?
- 80% of a program is parallel. What is the Amdahl speedup for p = 4, and what is the upper limit?
- T₁ = 120 s, T₆ = 30 s. Find the speedup, efficiency and cost.
- Write the Hillis–Steele prefix sums of [1, 2, 3, 4] round by round; how many additions are made?
Show answers
- log₂ 16 = 4 rounds; 8 processors in the first round; 8 + 4 + 2 + 1 = 15 additions (= n − 1).
- m + p − 2 = 103 steps; naive m(p − 1) = 400 steps (≈ 3.9 times as many).
- 3 + (10 − 1) · 2 = 21 rounds (naive 10 · 3 = 30).
- S = 1 / (0.2 + 0.8/4) = 1 / 0.4 = 2.5; limit 1 / 0.2 = 5.
- S = 4, E = 4/6 ≈ 0.67, C = 6 · 30 = 180 s.
- d = 1: [1, 3, 5, 7]; d = 2: [1, 3, 6, 10]. 3 + 2 = 5 additions (sequential: 3).
Summary
Exercises
- Modify PRAM tree summation to find the maximum element. What are its time and work?
- How many additions does Hillis–Steele make for n = 8? Compare with the sequential algorithm.
- Make the
visitedset of the parallel traversal thread-safe. - Add a threshold to parallel merge sort; why is it better to
compute()one of the two subtasks directly instead of callingfork()? - On p = 32 processors, what minimum parallel fraction gives a speedup of S = 16 (Amdahl)?
- Express the number of steps and the efficiency of a pipeline with m rows and p processors in general.
Answers / hints
a[i] = Math.max(a[i], a[i + step]); D = log₂ n, W = n − 1.- d = 1: 7, d = 2: 6, d = 4: 4 → 17 additions; sequential 7. In general: n log₂ n − (n − 1).
- Use
ConcurrentHashMap.newKeySet();visited.add(neighbor)checks and inserts in one atomic step. - If size < threshold, use
Arrays.sort.left.fork(); x = right.compute(); y = left.join();: the thread does not sit idle, and the number of tasks halves. - 1 / ((1 − f) + f/32) = 16 → 1 − (31/32) f = 1/16 → f = 30/31 ≈ 96.8%.
- Steps: m + p − 2; efficiency: m / (m + p − 2).
Summary
References
- Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., MIT Press, 2022 (Chapter 26: Parallel Algorithms).
- J. JáJá, An Introduction to Parallel Algorithms, Addison-Wesley, 1992.
- A. Grama, A. Gupta, G. Karypis and V. Kumar, Introduction to Parallel Computing, 2nd ed., Addison-Wesley, 2003.
- G. M. Amdahl, "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities", AFIPS, 1967.
- J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", OSDI, 2004.
- D. Lea, "A Java Fork/Join Framework", ACM Java Grande Conference, 2000.