Chapter 9: Parallel Algorithms

Advanced Algorithms
Why parallelism?Flynn and PRAM modelsSpeedup, AmdahlWork and depth Summation on a PRAMPrefix sumsPipeliningPipelined treeFork/Join, MapReduce
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Concepts

Explain the split–assign–combine structure of parallel algorithms, their benefits and their challenges (communication, synchronization, load balancing).

Models and measures

Know Flynn's taxonomy and the PRAM model; compute speedup, efficiency, cost, work and depth; apply Amdahl's law.

Core techniques

Run PRAM tree summation, parallel prefix sums, pipelining and the pipelined tree step by step, and count their steps.

Parallelism in Java

Read examples of parallel merge sort with Fork/Join, MapReduce and data parallelism; recognize race conditions.

Introduction

Faster Together

  • Some jobs take a very long time: big data analysis, scientific simulation, image processing.
  • Parallel algorithms solve a problem using several processors (cores) at the same time.
  • Complex tasks finish in much less time.
  • Single-core clock speeds have nearly stopped rising; performance gains now come from the number of cores.
Analogy: one teacher grades 1000 exams in 10 hours. If 10 teachers share the exams it takes ≈ 1 hour; but handing out the exams and collecting the grades also takes time.

Introduction

Parallel Algorithms

  • Like a group of people working together.
  • Split a complex problem into smaller pieces.
  • Speed up the work by assigning the pieces to different processors at the same time.
  • When the processors finish, combine the results.
  • The foundation of high-performance computing (HPC).
  • Uses: big data analysis, scientific simulation, machine learning and more.
Problempiece 1p₁piece 2p₂piece 3p₃piece 4p₄Result

Introduction

Benefits and Drawbacks

Benefits

  • Finish the same job sooner (speedup)
  • Solve large problems that do not fit on one machine (scalability)
  • Use the hardware (all cores, the GPU) efficiently

Drawbacks

  • Splitting the problem and combining the results takes extra effort
  • Communication and synchronization add overhead
  • Load balancing: every processor should get equal work, and there must be enough processing power
  • Debugging is hard (race conditions, deadlocks)
  • Some problems are inherently sequential
With data dependences (step k waits for the result of step k − 1), that part cannot be parallelized: e.g., the recurrence xₖ = f(xₖ₋₁).

Introduction

Applications

Matrix multiplication

The core operation of scientific computing; every row (even every cell) of the result can be computed independently.

Image processing

Edge detection, filtering, object recognition: the same operation on every pixel or block (data parallelism, GPU).

Computational fluid dynamics

Flow simulation in aircraft, ship and vehicle design: the domain is split into a grid, each region on one processor.

Genetic algorithms

The fitness of individuals in a population is evaluated independently and in parallel (scheduling, routing).

Models and Measures

Flynn's Taxonomy

ClassStands forMeaningExample
SISDSingle Instruction, Single Dataone instruction stream, one data streamclassic single-core processor
SIMDSingle Instruction, Multiple Datathe same instruction on many data items at oncevector instructions (AVX), GPU
MISDMultiple Instruction, Single Datadifferent instructions on the same datarare; fault-tolerant systems
MIMDMultiple Instruction, Multiple Dataeach processor runs its own program on its own datamulticore CPUs, clusters
Data parallelism: the same operation on different parts of the data (SIMD style).
Task parallelism: different tasks at the same time (MIMD style).

Models and Measures

Memory Architectures

Shared memorycommon memoryP₀P₁P₂P₃threads, locksDistributed memoryP₀MP₁MP₂MP₃Mnetwork (messages)MPI, Hadoop, Spark
  • Shared memory: all processors see the same address space. Communication is cheap; concurrent access requires synchronization.
  • Java threads, ForkJoinPool, OpenMP.
  • Distributed memory: each node has its own memory; data moves via messages.
  • Scales well, but communication is expensive: MPI, Hadoop MapReduce.

Models and Measures

The PRAM Model

  • PRAM (Parallel Random Access Machine): the theoretical model of parallel algorithms; the parallel counterpart of the sequential RAM model.
  • p processors, a shared memory and a common clock: the processors take steps in lockstep.
  • In each step a processor reads from memory → computes locally → writes to memory. Each takes O(1).
  • Communication and synchronization costs are ignored, so the model measures an algorithm's inherent parallelism.
TypeReadWrite
EREWexclusiveexclusive
CREWconcurrentexclusive
CRCWconcurrentconcurrent
common memoryp₀p₁p₂p₃common clock
E = Exclusive (one processor per cell per step), C = Concurrent (several processors). In CRCW, conflicting writes are resolved by a rule (common value, arbitrary one, priority…).

Models and Measures

Speedup, Efficiency, Cost

MeasureDefinition
T₁running time of the best sequential algorithm
Tₚparallel running time on p processors
SpeedupS = T₁ / Tₚ (ideal: S = p)
EfficiencyE = S / p (0 < E ≤ 1)
CostC = p · Tₚ
Example: T₁ = 100 s, p = 8, T₈ = 16 s
S = 100 / 16 = 6.25
E = 6.25 / 8 ≈ 0.78
C = 8 · 16 = 128 s (> 100 s)
  • Cost-optimal: C = Θ(T₁).
  • Why S < p: communication, waiting, load imbalance, sequential parts.
  • These bounds hold in the model; in practice cache effects or finding a target early in a search can give S > p (superlinear speedup).
p = 1T₁ = 100 sp = 8T₈ = 16 sgreen: useful work (100 s / 8 = 12.5 s)red: communication, waiting (3.5 s) → C = 8 · 16 = 128 s

Models and Measures

Amdahl's Law

  • A fraction f of the program can be parallelized; the fraction 1 − f is sequential.
  • S(p) = 1 / ((1 − f) + f / p)
  • As p → ∞, S → 1 / (1 − f): the sequential part limits the speedup.
  • f = 0.9, p = 8: S = 1 / (0.1 + 0.1125) ≈ 4.71; the limit is 10.
Gustafson's law: if the problem size grows with p, S = (1 − f) + f · p. f = 0.9, p = 8 → S = 7.3. Parallelism pays off more on large problems.
051015201416642561024number of processors p (log scale)speedup S(p)f = 0.95f = 0.9f = 0.75f = 0.5

Models and Measures

Work and Depth

  • Work W: the total number of operations = T₁ (time on one processor).
  • Depth (span) D: the longest chain of dependences = T∞ (time with unlimited processors).
  • Parallelism = W / D: the largest number of processors that can be used profitably.
  • Lower bounds: Tₚ ≥ W / p and Tₚ ≥ D.
  • Brent's theorem: Tₚ ≤ W / p + D.
  • Work-efficient: W is of the same order as the best sequential algorithm.
Example: summing n numbers with a tree
W = n − 1 additions, D = log₂ n rounds.
n = 1024: W = 1023, D = 10, parallelism ≈ 102.
Brent with p = 32: T₃₂ ≤ 1023/32 + 10 ≈ 42 steps.
W = 7 operations (nodes), D = 4 (red path)parallelism W / D = 1.75

Summation on a PRAM

Problem: Parallel Summation

  • Input: n numbers; output: their sum. The sequential algorithm performs n − 1 additions: O(n).
  • In the sequential loop each addition waits for the previous one: a chain of length n − 1, no parallelism.
  • Since addition is associative, we can regroup the parentheses:
    ((2 + 3) + (5 + 1)) + ((7 + 6) + (8 + 4)) → a balanced binary tree.
  • The sums on each level of the tree are independent → done in parallel in the same round.
  • The same idea works for any associative operation: max, min, product, AND, OR (reduction).
23517684561312112536round 1round 2round 3
Example array: 2, 3, 5, 1, 7, 6, 8, 4 → sum 36. Eight processors p₀ … p₇ hold one number each.

Summation on a PRAM · Step by step

PRAM Summation Step by Step

p0p1p2p3p4p5p6p723517684round 1round 2round 3

Start: the 8 numbers are distributed one per processor (p₀ … p₇). Dashed boxes are the cells written in later rounds.

p0p1p2p3p4p5p6p723517684561312round 1round 2round 3

Round 1: 4 processors add adjacent pairs simultaneously: 2+3=5, 5+1=6, 7+6=13, 8+4=12.

p0p1p2p3p4p5p6p7235176845613121125round 1round 2round 3

Round 2: 2 processors combine partial sums at distance 2: 5+6=11, 13+12=25.

p0p1p2p3p4p5p6p723517684561312112536round 1round 2round 3

Round 3: p₀ computes 11+25 = 36. The 8 numbers were summed in log₂ 8 = 3 rounds (sequential: 7 steps).

Summation on a PRAM · Run the code step by step

Tree Summation: Code Trace

2031521374658647

The array is created; n = 8. Each cell is held by one processor.

2031521374658647

New round: step = 1. This round performs 4 additions in parallel.

5031521374658647

a[0] = a[0] + a[1] = 2 + 3 = 5.

5031621374658647

a[2] = a[2] + a[3] = 5 + 1 = 6.

50316213134658647

a[4] = a[4] + a[5] = 7 + 6 = 13.

503162131346512647

a[6] = a[6] + a[7] = 8 + 4 = 12.

503162131346512647

New round: step = 2. This round performs 2 additions in parallel.

1103162131346512647

a[0] = a[0] + a[2] = 5 + 6 = 11.

1103162132546512647

a[4] = a[4] + a[6] = 13 + 12 = 25.

1103162132546512647

New round: step = 4. This round performs 1 addition in parallel.

3603162132546512647

a[0] = a[0] + a[4] = 11 + 25 = 36.

3603162132546512647

step = 8 → the loop ends. The sum is a[0] = 36; 3 rounds, 7 additions.

int[] a = {2, 3, 5, 1, 7, 6, 8, 4};
int n = a.length;
for (int step = 1; step < n; step *= 2) {
  // all i in the same round run in parallel
  for (int i = 0; i + step < n; i += 2 * step)
    a[i] = a[i] + a[i + step];
}  // end of round: synchronization
// the sum is in a[0]

Summation on a PRAM

PRAM Summation: Analysis

Measuren processors (n/2 suffice)p = n / log₂ n processors
Time Tₚlog₂ nO(log n)
Work Wn − 1n − 1
Cost p · TₚΘ(n log n)Θ(n)
Cost-optimal?noyes
  • As the rounds go on, the number of active processors halves (4, 2, 1): most processors sit idle, so the cost is high.
  • Fix: each processor first sums its own block of log₂ n numbers sequentially (O(log n)); then the n / log₂ n partial sums are combined with a tree (O(log n)).
  • EREW suffices: in each round every cell is read/written by a single processor. Each round ends with a synchronization (barrier).

Summation on a PRAM

Parallel Prefix Sums

  • Prefix sum (scan): yᵢ = x₀ + x₁ + … + xᵢ.
  • Sequential: yᵢ = yᵢ₋₁ + xᵢ, O(n) but fully chained.
  • Hillis–Steele: for d = 1, 2, 4, …, every i ≥ d does x[i] += x[i − d] in parallel.
  • D = ⌈log₂ n⌉ rounds, W = O(n log n) → not work-efficient.
  • The Blelloch algorithm (up-sweep/down-sweep) has W = O(n), D = O(log n).
  • Uses: parallel sorting, compaction (filter), Arrays.parallelPrefix in Java.
The summation tree yields only the last cell (the sum); in a prefix sum every cell must learn its own prefix.

Summation on a PRAM · Step by step

Hillis–Steele Prefix Sums Step by Step

initial23517684

Input: 2 3 5 1 7 6 8 4. Goal: x₀ + … + xᵢ for every i (inclusive prefix sum).

initial23517684d = 125868131412

d = 1: every processor with i ≥ 1 adds its left neighbor's value (7 additions at once).

initial23517684d = 125868131412d = 225101116192225

d = 2: for i ≥ 2, add the value 2 positions to the left. Each cell now holds the sum of a 4-element window.

initial23517684d = 125868131412d = 225101116192225d = 425101118243236

d = 4: result 2 5 10 11 18 24 32 36. Done in log₂ 8 = 3 rounds, but with 7+6+4 = 17 additions in total.

Summation on a PRAM · Run the code step by step

Blelloch Prefix Sums: Code Trace

2031521374658647input

Input 2 3 5 1 7 6 8 4 (n = 8). The array serves as an implicit binary tree: first up (reduction), then down.

205152637413586127up-sweep, d = 1

d = 1: x[1] += x[0], x[3] += x[2], x[5] += x[4], x[7] += x[6] (in parallel in one round, 4 additions).

2051521137413586257up-sweep, d = 2

d = 2: x[3] += x[1], x[7] += x[5] (in parallel in one round, 2 additions).

2051521137413586367up-sweep, d = 4

d = 4: x[7] += x[3] (in parallel in one round, 1 addition). The root x[7] = 36 is the total.

205152113741358607root set to 0

The total 36 is at the root. The root is set to 0: in the down-sweep every node carries "the sum of everything to its left".

205152037413586117down-sweep, d = 4

d = 4: in each pair (i − d, i), the left child takes the parent's value and the right child takes parent + the old left value.

200152537411586247down-sweep, d = 2

d = 2: in each pair (i − d, i), the left child takes the parent's value and the right child takes parent + the old left value.

002152103114185246327down-sweep, d = 1

d = 1: the last round; 4 pairs are updated at once.

002152103114185246327result (exclusive prefix)

Result 0 2 5 10 11 18 24 32: the exclusive prefix sums. 2 · (n − 1) = 14 operations, O(n) work, depth 2 log₂ n = 6 rounds.

// up-sweep; x = 2 3 5 1 7 6 8 4
for (int d = 1; d < n; d *= 2)
  for (int i = 2*d - 1; i < n; i += 2*d)
    x[i] += x[i - d];
x[n - 1] = 0;
// down-sweep (inner loops in parallel)
for (int d = n / 2; d >= 1; d /= 2)
  for (int i = 2*d - 1; i < n; i += 2*d) {
    int t = x[i - d];
    x[i - d] = x[i];
    x[i] += t;
  }
// x: exclusive prefix sums

Pipelining

The Idea of Pipelining

  • A pipeline splits the work into consecutive stages; each stage is a processor.
  • Like a factory assembly line: while one job moves to stage 2, stage 1 starts on the next job.
  • A single job does not get faster, but the number of jobs finished per unit time (throughput) increases.
  • k stages, m jobs: k · m steps sequentially; k + m − 1 steps with a pipeline.
  • Some stages are idle while the pipeline fills at the start and drains at the end.
t₁t₂t₃t₄t₅t₆stage 1job 1job 2job 3job 4stage 2job 1job 2job 3job 4stage 3job 1job 2job 3job 4

3 stages, 4 jobs: 3 + 4 − 1 = 6 steps instead of 3 · 4 = 12.

Pipelining

The Pipeline Problem

  • Five processors p₀ … p₄ form a linear chain; each processor can send messages only to its right neighbor.
  • Each pj holds a column of 4 values (rows 0 … 3).
  • Goal: for every row r, the value at pj should become the sum of the values of p₀ … pj in that row (row-wise prefix sums). Then p₄ holds the sum of every row.
  • In one step each processor sends one value; the receiver adds it to its own.
  • Naive method: first move row 0 along the chain (4 steps), then row 1… → 4 · 4 = 16 steps.
  • Pipeline: pj sends row r at time t = r + j; the rows flow through the chain one after another.

Pipelining · Step by step

Pipelining Step by Step: 5 Processors, 4 Rows

p02174p13729p23473p36102p48635

Each pj holds a 4-row column. Goal: in every row, pj holds the sum of the values of p₀ … pj. p₀'s values are already final.

p02174p15729p23473p36102p48635+

Step 1: p₀→p₁ row 0: 2+3=5.

p02174p15829p28473p36102p48635++

Step 2: p₀→p₁ row 1: 1+7=8, p₁→p₂ row 0: 5+3=8.

p02174p15899p281273p314102p48635+++

Step 3: p₀→p₁ row 2: 7+2=9, p₁→p₂ row 1: 8+4=12, p₂→p₃ row 0: 8+6=14.

p02174p158913p2812163p3141302p422635++++

Step 4: p₀→p₁ row 3: 4+9=13, p₁→p₂ row 2: 9+7=16, p₂→p₃ row 1: 12+1=13, p₃→p₄ row 0: 14+8=22.

p02174p158913p28121616p31413162p4221935+++

Step 5: p₁→p₂ row 3: 13+3=16, p₂→p₃ row 2: 16+0=16, p₃→p₄ row 1: 13+6=19.

p02174p158913p28121616p314131618p42219195++

Step 6: p₂→p₃ row 3: 16+2=18, p₃→p₄ row 2: 16+3=19.

p02174p158913p28121616p314131618p422191923+

Step 7: p₃→p₄ row 3: 18+5=23. Done in 7 steps; p₄ holds the row sums (22 19 19 23).

Pipelining

Pipelining: Analysis

stepp₀→p₁p₁→p₂p₂→p₃p₃→p₄
1row 0———
2row 1row 0——
3row 2row 1row 0—
4row 3row 2row 1row 0
5—row 3row 2row 1
6——row 3row 2
7———row 3
for (int t = 0; t < m + p - 2; t++)
  // all j at the same time
  for (int j = 0; j < p - 1; j++) {
    int r = t - j;
    if (r >= 0 && r < m)
      x[j + 1][r] += x[j][r];
  }
  • Number of steps: m + p − 2 = 4 + 5 − 2 = 7 (naive: m(p − 1) = 16).
  • Efficiency: m(p − 1) additions / ((p − 1)(m + p − 2)) = m / (m + p − 2) = 4/7 ≈ 57%.
  • For m ≫ p the efficiency → 1.

Pipelining · Run the code step by step

Pipelining: Code Trace

p₀2174p₁3729p₂3473p₃6102p₄86350123

p = 5 processors, m = 4 rows. Each column holds one processor's values; row r reaches pj at time t = r + j.

p₀2→174p₁5729p₂3473p₃6102p₄86350123

t = 0: p₀→p₁ row 0: 2 + 3 = 5.

p₀21→74p₁5→829p₂8473p₃6102p₄86350123

t = 1: p₀→p₁ row 1: 1 + 7 = 8; p₁→p₂ row 0: 5 + 3 = 8.

p₀217→4p₁58→99p₂8→1273p₃14102p₄86350123

t = 2: p₀→p₁ row 2: 7 + 2 = 9; p₁→p₂ row 1: 8 + 4 = 12; p₂→p₃ row 0: 8 + 6 = 14.

p₀2174→p₁589→13p₂812→163p₃14→1302p₄226350123

t = 3: p₀→p₁ row 3: 4 + 9 = 13; p₁→p₂ row 2: 9 + 7 = 16; p₂→p₃ row 1: 12 + 1 = 13; p₃→p₄ row 0: 14 + 8 = 22. The pipeline is full: all 4 links are busy.

p₀2174p₁58913→p₂81216→16p₃1413→162p₄2219350123

t = 4: p₁→p₂ row 3: 13 + 3 = 16; p₂→p₃ row 2: 16 + 0 = 16; p₃→p₄ row 1: 13 + 6 = 19.

p₀2174p₁58913p₂8121616→p₃141316→18p₄22191950123

t = 5: p₂→p₃ row 3: 16 + 2 = 18; p₃→p₄ row 2: 16 + 3 = 19.

p₀2174p₁58913p₂8121616p₃14131618→p₄221919230123

t = 6: p₃→p₄ row 3: 18 + 5 = 23.

p₀2174p₁58913p₂8121616p₃14131618p₄221919230123

Done in m + p − 2 = 7 steps; p₄ holds the row sums 22 19 19 23. The naive method would take 16 steps.

for (int t = 0; t < m + p - 2; t++)
  // all j at the same time
  for (int j = 0; j < p - 1; j++) {
    int r = t - j;
    if (r >= 0 && r < m)
      x[j + 1][r] += x[j][r];
  }

Pipelined Tree

Reduction on a Tree

  • The processors are connected as a tree; each node sends its value to its parent, which adds it to its own. The result accumulates at the root.
  • If the tree has depth h, reducing a single value takes h rounds (h = 3 in our example).
  • Now each processor holds not one number but a vector with k components (k = 3 here: green, blue, red). Each component is summed separately.
  • Naive method: reduce the components one after another → k · h = 9 rounds.
  • Pipelined tree: while one component moves up, the next one sets out from the leaves.
  • Rule: a node can receive at most one message per round (and may also send one in the same round).

Pipelined Tree · Step by step

Pipelined Tree Step by Step

361p0375p1853p2958p3752p4345p5217p6

Each of the 7 processors holds a 3-component vector (green, blue, red). Goal: the sum of each component at the root p₀; values flow bottom-up.

361p01075p11753p2958p3752p4545p5217p6

Round 1: green p₃→p₂: 9+8=17, p₄→p₁: 7+3=10, p₆→p₅: 2+3=5.

861p02775p11753p2958p3752p4545p5217p6

Round 2: green p₂→p₁: 17+10=27, p₅→p₀: 5+3=8.

3561p027125p117103p2958p3752p4555p5217p6

Round 3: green p₁→p₀: 27+8=35; blue p₃→p₂: 5+5=10, p₄→p₁: 5+7=12, p₆→p₅: 1+4=5.

35111p027225p117103p2958p3752p4555p5217p6

Round 4: blue p₂→p₁: 10+12=22, p₅→p₀: 5+6=11.

35331p027227p1171011p2958p3752p45512p5217p6

Round 5: blue p₁→p₀: 22+11=33; red p₃→p₂: 8+3=11, p₄→p₁: 2+5=7, p₆→p₅: 7+5=12.

353313p0272218p1171011p2958p3752p45512p5217p6

Round 6: red p₂→p₁: 11+7=18, p₅→p₀: 12+1=13.

353331p0272218p1171011p2958p3752p45512p5217p6

Round 7: red p₁→p₀: 18+13=31. The root holds (35, 33, 31): done in 7 rounds.

  • Edge color = component carried this round; arrows point to the parent.
  • Thick border = value updated this round.
  • Each component reaches the root in 3 stages:
    (1) p₃→p₂, p₄→p₁, p₆→p₅
    (2) p₂→p₁, p₅→p₀
    (3) p₁→p₀.
  • A new component starts every 2 rounds.

Pipelined Tree

Pipelined Tree: Analysis

roundstage 1
p₃→p₂, p₄→p₁, p₆→p₅
stage 2
p₂→p₁, p₅→p₀
stage 3
p₁→p₀
1green——
2—green—
3blue—green
4—blue—
5red—blue
6—red—
7——red
  • Why every 2 rounds? p₁ receives from two children (p₄ and p₂); since it can receive one message per round, each component keeps p₁ busy for 2 rounds. The same holds for p₀.
  • Total rounds: h + (k − 1) · 2 = 3 + 2 · 2 = 7 (naive: k · h = 9).
  • As k grows: 2k + 1 ≈ 2k rounds versus 3k for the naive method. In general, rounds ≈ h + (k − 1) · (maximum number of children).
  • At the root: green 35, blue 33, red 31.

Kinds of Parallel Algorithms

Kinds of Parallel Algorithms

Divide and conquer

The problem is split into independent subproblems that are solved simultaneously on different cores, and the results are combined. Parallel merge sort, quicksort, matrix multiplication.

MapReduce

Big data is processed on distributed nodes: the map step emits intermediate key–value pairs, and the reduce step combines them. Hadoop, Spark; counting, indexing, log analysis.

Data parallel

The same computation is applied to different parts of the data at the same time (SIMD instructions, GPU). Scientific simulation, image processing, numerical computing.

tasksubtask 1subtask 2subtask 3combinefork()join()

Kinds of Parallel Algorithms

Parallel Merge Sort

class SortTask extends RecursiveTask<int[]> {
  private final int[] arr;
  SortTask(int[] arr) { this.arr = arr; }

  protected int[] compute() {
    if (arr.length <= 1) return arr;
    int mid = arr.length / 2;
    var left = new SortTask(
        Arrays.copyOfRange(arr, 0, mid));
    var right = new SortTask(
        Arrays.copyOfRange(arr, mid, arr.length));
    left.fork();              // start in parallel
    right.fork();
    return merge(left.join(), right.join());
  }
}
// usage:
int[] s = new ForkJoinPool().invoke(
    new SortTask(arr));
  • RecursiveTask: a Fork/Join task that returns a result.
  • fork(): submits the subtask to the pool (asynchronously).
  • join(): waits for the subtask to finish and gets its result.
  • merge: the usual sequential merge, O(n).
  • ForkJoinPool uses work stealing to hand work to idle cores.

Kinds of Parallel Algorithms · Step by step

Parallel Merge Sort: 2 3 5 1 7 6 8 4

array23517684

Task: sort the 8-element array. SortTask is the root task.

array23517684fork (split)23517684

The task splits the array in two and submits both subtasks to the pool with fork(); the halves can be processed at the same time.

array23517684fork (split)23517684fork (split)23517684

Each subtask splits again: now 4 tasks can run in parallel.

array23517684fork (split)23517684fork (split)23517684fork (split)23517684

Base case: one-element arrays are already sorted (8 leaf tasks).

array23517684fork (split)23517684fork (split)23517684fork (split)23517684join (merge)23156748

join(): each parent waits for its two children's results and merges them: [2 3] [1 5] [6 7] [4 8].

array23517684fork (split)23517684fork (split)23517684fork (split)23517684join (merge)23156748join (merge)12354678

One level up, two merges run in parallel: [1 2 3 5] and [4 6 7 8].

array23517684fork (split)23517684fork (split)23517684fork (split)23517684join (merge)23156748join (merge)12354678join (merge)12345678

The root task does the final merge alone: [1 2 3 4 5 6 7 8]. This sequential merge makes the depth (span) Θ(n).

Kinds of Parallel Algorithms

Parallel Sorting: Analysis

VersionWork WDepth DParallelism W / D
Sequential merge sortΘ(n log n)Θ(n log n)1
Parallel recursion + sequential mergeΘ(n log n)Θ(n)Θ(log n)
Parallel recursion + parallel merge (CLRS)Θ(n log n)Θ(log³ n)Θ(n / log² n)
  • Depth recurrence: D(n) = D(n/2) + Θ(n) = Θ(n), because the two halves run in parallel but the merge is sequential.
  • Parallelism is only Θ(log n): for n = 10⁶, more than ≈ 20 cores hardly help.
  • In practice, switch to a sequential sort for small parts (e.g., < 1000 elements); otherwise task-creation overhead eats the gain.
  • Ready-made: Arrays.parallelSort(arr).

Kinds of Parallel Algorithms

MapReduce

  • Map: each chunk is processed independently and emits intermediate key–value pairs (word → count).
  • Shuffle: equal keys are gathered at the same reducer.
  • Reduce: the values of each key are combined (summed).
  • In the example: 13 words, 4 threads → chunk size ⌊13/4⌋ = 3; the last chunk has 4 words.
SplitMap tasksReduceapple, bananaappleapple:2, banana:1orange, bananaappleorange:1, banana:1apple:1banana, bananaorangebanana:2, orange:1apple, cherrybanana, appleapple:2, cherry:1banana:1apple: 5banana: 5orange: 2cherry: 1

Kinds of Parallel Algorithms

MapReduce: Code

// Map: count the words in one chunk
protected Map<String, Integer> compute() {
  Map<String, Integer> result = new HashMap<>();
  for (String word : chunk)
    result.merge(word, 1, Integer::sum);
  return result;
}
// Main flow
for (MapTask g : mapTasks) g.fork();
List<Map<String, Integer>> partial = new ArrayList<>();
for (MapTask g : mapTasks) partial.add(g.join());
// Reduce: combine the partial results
Map<String, Integer> total = new HashMap<>();
for (Map<String, Integer> m : partial)
  m.forEach((k, v) -> total.merge(k, v, Integer::sum));
  • Each map task has its own HashMap: no shared state, no locks needed.
  • Result: apple 5, banana 5, orange 2, cherry 1.
  • Here reduce is a single task; in real MapReduce the keys are partitioned among many reducers (hash(k) mod r).

Kinds of Parallel Algorithms · Run the code step by step

MapReduce: Code Trace

apple, banana, appleorange, banana, applebanana, banana, orangeapple, cherry, banana, appletotal

The 13 words were split into 4 chunks; for each chunk a MapTask is submitted to the pool with fork(), and the tasks run in parallel.

apple, banana, apple→ banana:1, apple:2orange, banana, applebanana, banana, orangeapple, cherry, banana, appletotal

join(): task 0's own HashMap is ready → banana 1, apple 2. No shared state, no locks needed.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orangeapple, cherry, banana, appletotal

join(): task 1's own HashMap is ready → banana 1, orange 1, apple 1. No shared state, no locks needed.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, appletotal

join(): task 2's own HashMap is ready → orange 1, banana 2. No shared state, no locks needed.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, apple→ banana:1, cherry:1, apple:2total

join(): task 3's own HashMap is ready → banana 1, cherry 1, apple 2. No shared state, no locks needed.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, apple→ banana:1, cherry:1, apple:2total

Reduce begins: an empty total map.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, apple→ banana:1, cherry:1, apple:2totalbanana 1apple 2

partial[0] merged: equal keys are added with Integer::sum.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, apple→ banana:1, cherry:1, apple:2totalorange 1banana 2apple 3

partial[1] merged: equal keys are added with Integer::sum.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, apple→ banana:1, cherry:1, apple:2totalorange 2banana 4apple 3

partial[2] merged: equal keys are added with Integer::sum.

apple, banana, apple→ banana:1, apple:2orange, banana, apple→ banana:1, orange:1, apple:1banana, banana, orange→ orange:1, banana:2apple, cherry, banana, apple→ banana:1, cherry:1, apple:2totalorange 2banana 5cherry 1apple 5

partial[3] merged: equal keys are added with Integer::sum. Result: orange 2, banana 5, cherry 1, apple 5.

// Map: count the words in one chunk
protected Map<String, Integer> compute() {
  Map<String, Integer> result = new HashMap<>();
  for (String word : chunk)
    result.merge(word, 1, Integer::sum);
  return result;
}
// Main flow
for (MapTask g : mapTasks) g.fork();
List<Map<String, Integer>> partial = new ArrayList<>();
for (MapTask g : mapTasks) partial.add(g.join());
// Reduce: combine the partial results
Map<String, Integer> total = new HashMap<>();
for (Map<String, Integer> m : partial)
  m.forEach((k, v) -> total.merge(k, v, Integer::sum));

Kinds of Parallel Algorithms

Data Parallelism: Matrix Multiplication

// each row is a separate task
class RowTask extends RecursiveAction {
  protected void compute() {
    for (int j = 0; j < n; j++) {
      C[row][j] = 0;
      for (int k = 0; k < n; k++)
        C[row][j] += A[row][k] * B[k][j];
    }
  }
}
try (var pool = new ForkJoinPool()) {
  for (int i = 0; i < n; i++)
    pool.submit(new RowTask(i));
}   // close(): waits for all tasks
C = A · B
302418
846954
13811490
  • A = [[1,2,3],[4,5,6],[7,8,9]], B = [[9,8,7],[6,5,4],[3,2,1]].
  • Rows are independent: n tasks, each with Θ(n²) work. W = Θ(n³), D = Θ(n²) (per row).
  • Each task writes only to its own row → no race.

Kinds of Parallel Algorithms · Run the code step by step

Row Task: Code Trace

A123456789·M987654321=C

This task computes only row 1 of C: row 1 of A (4, 5, 6) times each column of B. The other rows are in other tasks.

A123456789·M987654321=C0

j = 0: C[1][0] starts at 0.

A123456789·M987654321=C36

k = 0: C[1][0] += A[1][0] · B[0][0] = 4 · 9 → 36.

A123456789·M987654321=C66

k = 1: C[1][0] += A[1][1] · B[1][0] = 5 · 6 → 66.

A123456789·M987654321=C84

k = 2: C[1][0] += A[1][2] · B[2][0] = 6 · 3 → 84.

A123456789·M987654321=C8469

j = 1: 4 · 8 + 5 · 5 + 6 · 2 = 69.

A123456789·M987654321=C846954

j = 2: 4 · 7 + 5 · 4 + 6 · 1 = 54.

A123456789·M987654321=C846954Rows 0 and 2 are computed at the same time in other tasks.

Task done: C[1] = (84, 69, 54). Each task writes only to its own row, so there is no race condition.

// RowTask, row = 1
protected void compute() {
  for (int j = 0; j < n; j++) {
    C[row][j] = 0;
    for (int k = 0; k < n; k++)
      C[row][j] += A[row][k] * B[k][j];
  }
}

Kinds of Parallel Algorithms

Race Conditions

Two threads increment the shared counter at the same time. counter++ is really three steps: read → increment → write.

timeThread 1Thread 2counter
1read (0)0
2read (0)0
3write 0 + 11
4write 0 + 11

Expected 2, got 1: a lost update.

  • Race condition: the result depends on the timing of the threads; the bug may not show up on every run.
  • Fixes: a synchronized block / lock, AtomicInteger.incrementAndGet(), concurrent collections (ConcurrentHashMap).
  • Best of all is to avoid shared state: each task produces its own partial result, and the results are combined afterward (reduction, MapReduce).
  • Locks are not free either: they create sequential regions (Amdahl) and risk deadlock.

Kinds of Parallel Algorithms

Concurrent Graph Traversal

12345678
  • Idea: fork() a new traversal task for each unvisited neighbor; subtrees are traversed in parallel.
  • The tasks share the visited set. If it is an ordinary HashSet, concurrent access is not safe.
  • Moreover, contains + add are two separate steps: two tasks may visit the same node together.
Set<Node> visited = ConcurrentHashMap.newKeySet();
if (!visited.add(neighbor)) continue; // atomic check+add

Level-synchronous BFS: all nodes of a level are expanded in parallel, with a barrier at the end of each level. D = O(diameter) on an ideal PRAM with enough processors; the number of levels does not exceed the diameter of the part reachable from the source (O(diameter · log n) in a realistic model).

Kinds of Parallel Algorithms

Parallelism Tools in Java

ToolWhen?
Thread, Runnablelowest level; managed by hand, rarely used directly
ExecutorServicerunning independent tasks in a fixed thread pool
ForkJoinPool, RecursiveTaskdivide and conquer; recursive tasks with work stealing
list.parallelStream()data-parallel map / filter / reduce (uses the common ForkJoinPool)
Arrays.parallelSortsorting large arrays in parallel
Arrays.parallelPrefixparallel prefix sums (scan)
AtomicInteger, ConcurrentHashMapsafe access when a shared counter/set is needed
Map<String, Long> counts = words.parallelStream()
    .collect(Collectors.groupingByConcurrent(w -> w, Collectors.counting()));

Summary

Summary

TopicKey points
Ideasplit → assign to processors → solve simultaneously → combine; the price is communication, synchronization, load balancing
ModelsFlynn: SISD/SIMD/MISD/MIMD; PRAM: EREW, CREW, CRCW; shared / distributed memory
MeasuresS = T₁/Tₚ, E = S/p, C = p·Tₚ; Amdahl S ≤ 1/(1 − f); Brent Tₚ ≤ W/p + D
PRAM summationlog₂ n rounds, W = n − 1; cost-optimal with p = n/log n
Prefix sumsHillis–Steele: log₂ n rounds, O(n log n) work; Blelloch: O(n) work
Pipeliningp processors, m rows: m + p − 2 steps (naive m(p − 1))
Pipelined treeh + (k − 1)·2 rounds (example 7; naive k·h = 9)
JavaFork/Join (fork, join), MapReduce, one task per row; races on shared state → atomic/concurrent structures

Summary

Common Mistakes

Thinking p processors = p times fasterThe sequential part (Amdahl), communication and waiting make S < p. 10% sequential code caps the speedup at 10.
A shared HashSet / counterConcurrent writes lead to race conditions. Use ConcurrentHashMap, AtomicInteger, or local results + combining.
Overly fine-grained tasksCalling fork() for every element inflates task-creation overhead. Parts below a threshold are processed sequentially.
Forgetting synchronization between roundsIn PRAM summation, if round t + 1 starts before all writes of round t finish, stale values are read.
Looking only at depthHillis–Steele finishes in O(log n) rounds but does O(n log n) work; with few processors it can be slower than a sequential scan.
Miscounting pipeline stepsThe fill time is forgotten: for m rows and p processors it takes m + p − 2 steps, not m.

Summary

Sample Problems

  1. n = 16 numbers are summed with a PRAM tree. How many rounds does it take, how many processors work in the first round, and how many additions are made in total?
  2. How many steps does a pipeline with p = 5 processors and m = 100 rows need? Compare with the naive method.
  3. In the chapter's pipelined tree example (7 processors, h = 3), how many rounds would be needed if each processor held a vector with k = 10 components?
  4. 80% of a program is parallel. What is the Amdahl speedup for p = 4, and what is the upper limit?
  5. T₁ = 120 s, T₆ = 30 s. Find the speedup, efficiency and cost.
  6. Write the Hillis–Steele prefix sums of [1, 2, 3, 4] round by round; how many additions are made?
Show answers
  1. log₂ 16 = 4 rounds; 8 processors in the first round; 8 + 4 + 2 + 1 = 15 additions (= n − 1).
  2. m + p − 2 = 103 steps; naive m(p − 1) = 400 steps (≈ 3.9 times as many).
  3. 3 + (10 − 1) · 2 = 21 rounds (naive 10 · 3 = 30).
  4. S = 1 / (0.2 + 0.8/4) = 1 / 0.4 = 2.5; limit 1 / 0.2 = 5.
  5. S = 4, E = 4/6 ≈ 0.67, C = 6 · 30 = 180 s.
  6. d = 1: [1, 3, 5, 7]; d = 2: [1, 3, 6, 10]. 3 + 2 = 5 additions (sequential: 3).

Summary

Exercises

  1. Modify PRAM tree summation to find the maximum element. What are its time and work?
  2. How many additions does Hillis–Steele make for n = 8? Compare with the sequential algorithm.
  3. Make the visited set of the parallel traversal thread-safe.
  4. Add a threshold to parallel merge sort; why is it better to compute() one of the two subtasks directly instead of calling fork()?
  5. On p = 32 processors, what minimum parallel fraction gives a speedup of S = 16 (Amdahl)?
  6. Express the number of steps and the efficiency of a pipeline with m rows and p processors in general.
Answers / hints
  1. a[i] = Math.max(a[i], a[i + step]); D = log₂ n, W = n − 1.
  2. d = 1: 7, d = 2: 6, d = 4: 4 → 17 additions; sequential 7. In general: n log₂ n − (n − 1).
  3. Use ConcurrentHashMap.newKeySet(); visited.add(neighbor) checks and inserts in one atomic step.
  4. If size < threshold, use Arrays.sort. left.fork(); x = right.compute(); y = left.join();: the thread does not sit idle, and the number of tasks halves.
  5. 1 / ((1 − f) + f/32) = 16 → 1 − (31/32) f = 1/16 → f = 30/31 ≈ 96.8%.
  6. Steps: m + p − 2; efficiency: m / (m + p − 2).

Summary

References

  • Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, 4th ed., MIT Press, 2022 (Chapter 26: Parallel Algorithms).
  • J. JáJá, An Introduction to Parallel Algorithms, Addison-Wesley, 1992.
  • A. Grama, A. Gupta, G. Karypis and V. Kumar, Introduction to Parallel Computing, 2nd ed., Addison-Wesley, 2003.
  • G. M. Amdahl, "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities", AFIPS, 1967.
  • J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", OSDI, 2004.
  • D. Lea, "A Java Fork/Join Framework", ACM Java Grande Conference, 2000.

The End

Chapter 9: Parallel Algorithms
End of the course · Previous chapter: Chapter 8: Online Algorithms
1 / 1 Sercan KÜLCÜ. All rights reserved.