Chapter 12: Multiprocessor Systems

Operating Systems
UMA: bus, crossbar, omega NUMA and directory-based coherence MESI and false sharing Multicore, SMT, GPUs Multiprocessor OS types Synchronization: TSL, ticket, MCS Scheduling: time, space, gang Multicomputer topologies Message passing and RPC DSM and load balancing Virtualization and containers Distributed systems and middleware Cloud and clusters
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Hardware

Compare UMA (bus, crossbar, omega) and NUMA architectures; trace directory-based coherence.

Cache coherence

Apply MESI state transitions to a sequence of reads and writes; find and fix false sharing.

OS and locks

Evaluate separate-OS, master-slave and SMP models; TSL, ticket and MCS locks; spinning versus switching.

Scheduling

Analyze a shared ready queue, space sharing and gang scheduling with an example.

Multicomputers

Compute topology diameters; explain message passing, RPC, DSM and load-balancing algorithms.

Virtualization and distributed systems

Recognize hypervisor types, VT-x/EPT and containers; middleware types and the cloud.

Introduction

Why Multiple Processors?

Symmetric (SMP)

All processors have equal access to memory and I/O; kernel code can run on any of them.

Asymmetric (AMP)

Processors are not equal: one runs the OS, or each runs separate software.

Cluster

Independent computers on a network act as one system; each node has its own OS.

  • Clock rates have been stuck at 3–5 GHz since ≈ 2005 (the end of Dennard scaling); the transistor budget now goes to more cores.
  • The OS handles processor allocation, synchronization, interprocess communication and the consistency of shared data.
MemoryCPUCPUCPUCPUCPUCPU(a) shared memoryinterconnectC+MC+MC+MC+MC+MC+M(b) message-passing multicomputerInternetcomputercomputercomputercomputercomputercomputer(c) wide-area distributed system

Amdahl: with parallel fraction p, the speedup on n processors is 1 / ((1 − p) + p/n). For p = 0.95: 5.93 on 8 processors, 15.42 on 64, with a limit of 20. “More processors, more performance” holds only when the serial part is small.

Multiprocessor Hardware

UMA: Bus-Based Architecture

busCPUCPUMemoryshared(a) no cachebusCPUcacheCPUcacheMemoryshared(b) a cache per CPUbusCPUcacheprivateCPUcacheprivateMemoryshared(c) cache + private memory
  • UMA (Uniform Memory Access): every CPU reaches every memory word in the same time, which makes parallel programming easier.
  • (a) Single bus: as CPUs are added the bus saturates; beyond 2–4 processors the CPUs spend most of their time waiting.
  • (b) Caches serve most reads locally and cut traffic, but copies of the same line must be kept coherent (MESI).
  • (c) Private memory holds program code, constants, stacks and locals; shared memory is used only for shared variables.

UMA: Uniform Memory Access. Today an on-chip ring or mesh replaces the single shared bus (Intel mesh, AMD Infinity Fabric).

Multiprocessor Hardware

Crossbar Switch

M0M1M2M3M4M5M6M7CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5CPU 6CPU 7crosspointopenclosedn × n switch → n² crosspoints8 → 64 · 64 → 4.0961,024 → 1,048,576

8 CPUs (rows) and 8 memory modules (columns); each crosspoint is a small electronic switch. All are open.

M0M1M2M3M4M5M6M7CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5CPU 6CPU 7crosspointopenclosedn × n switch → n² crosspoints8 → 64 · 64 → 4.0961,024 → 1,048,576

CPU 2 wants M5: crosspoint (2, 5) closes, creating a direct path.

M0M1M2M3M4M5M6M7CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5CPU 6CPU 7crosspointopenclosedn × n switch → n² crosspoints8 → 64 · 64 → 4.0961,024 → 1,048,5763 transfers at once

CPU 0 → M1 and CPU 6 → M3 are set up at the same time. A crossbar is nonblocking: requests to different modules never wait for each other.

M0M1M2M3M4M5M6M7CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5CPU 6CPU 7crosspointopenclosedn × n switch → n² crosspoints8 → 64 · 64 → 4.0961,024 → 1,048,576M5 busy → CPU 4 waits

CPU 4 also wants M5: the network has a path, but the module is busy; the only contention is at the memory module. The cost: n² crosspoints.

Example: the old Sun Fire E25K used an 18 × 18 crossbar; today small crossbars between cores and L3 slices are common on chip.

Multiprocessor Hardware

Routing in an Omega Network

2 × 2ABXYany input can connect to any outputbit = 0 → upper output (X)bit = 1 → lower output (Y)message formatModuleAddressOpcodeValueModule: which memory module (routing tag)Address: word within the moduleOpcode: READ / WRITEValue: word to write (optional)n CPUs · n memories → log₂n stages × n/2 switchesn = 8: 3 × 4 = 12 switches (a crossbar needs 64)n = 1,024: 10 × 512 = 5,120 switches (1,048,576)

The building block is a 2 × 2 switch: two inputs, two outputs. Switches look only at the message's Module field, the target memory module.

000001010011100101110111CPUMemory0000010100111001011101111A1B1C1D2A2B2C2D3A3B3C3D

8 CPUs, 8 memories, 3 stages × 4 switches. The wiring between stages is a perfect shuffle (address bits rotate left by one).

000001010011100101110111CPUMemory0000010100111001011101111A1B1C1D2A2B2C2D3A3B3C3D

CPU 011 reads memory 110. Stage 1 (1D) checks the first target bit: 1 → lower output; 2D, second bit 1 → lower; 3D, third bit 0 → upper. Arrives at 110.

000001010011100101110111CPUMemory0000010100111001011101111A1B1C1D2A2B2C2D3A3B3C3Dreply: the source (011) is written into Module and the path is retraced

The reply returns through the same network: each switch writes the source bit into the Module field, and memory uses that tag to reply to 011.

000001010011100101110111CPUMemory0000010100111001011101111A1B1C1D2A2B2C2D3A3B3C3Dconflict: two requests want 2D's lower output

Meanwhile CPU 001 wants memory 111; both requests need 2D's lower output → one waits. Omega is a blocking network.

An omega network carries many transfers concurrently (not “synchronously”), making conflicting ones wait; with (n/2)·log₂n switches it needs far less hardware and power than a crossbar.

Multiprocessor Hardware

NUMA Multiprocessors

Node 0 (socket)corecorecorecoreL3 (shared)mem. ctrl.local DRAMNode 1 (socket)corecorecorecoreL3 (shared)mem. ctrl.local DRAMlocal ≈ 80 nslocal ≈ 80 nsUPI / IFremote ≈ 140 ns: node 0 → node 1 memory (over the interconnect)
  • NUMA (Non-Uniform Memory Access): every CPU sees a single address space; remote memory is also reached with ordinary LOAD/STORE.
  • Remote access is slower than local. NC-NUMA: no cache coherence; CC-NUMA: directory-based coherence (every server today).
  • The OS keeps a process and its pages on the same node (first-touch policy, Linux automatic NUMA balancing, numactl).
Average access = y · tlocal + (1 − y) · tremote
y = 0.9 → 0.9·80 + 0.1·140 = 86 ns
y = 0.5 → 110 ns (28% slower)
latencies are sample values

The term is Non-Uniform Memory Access. Even single-socket chips can be NUMA inside (NPS setting on AMD EPYC, Intel SNC).

Multiprocessor Hardware

Directory-Based CC-NUMA

address 0x24000108Node (8 bits)Block (18 bits)Offset (6 bits)interconnection network (256 nodes)Node 20CPUcacheNode 82CPUblock 2Node 36RAM 16 MiB2¹⁸ lines × 64 Bdirectory0–1–2823–4–32-bit address = 8-bit node + 18-bit block + 6-bit offset256 nodes × 16 MiB = 4 GiB · each node keeps the directory for its own blocks

Tanenbaum's example: 256 nodes, 16 MiB per node, 64-byte lines. Each node keeps the directory for the lines in its own memory.

address 0x24000108Node (8 bits) = 36Block (18 bits) = 4Offset (6 bits) = 8interconnection network (256 nodes)Node 20CPUcacheNode 82CPUblock 2Node 36RAM 16 MiB2¹⁸ lines × 64 Bdirectory0–1–2823–4–request: node 36, block 4 (READ)

A CPU on node 20 executes LOAD 0x24000108: node = 0x24 = 36, block = 4, offset = 8. Not local → the request goes to 36.

address 0x24000108Node (8 bits) = 36Block (18 bits) = 4Offset (6 bits) = 8interconnection network (256 nodes)Node 20CPUcacheNode 82CPUblock 2Node 36RAM 16 MiB2¹⁸ lines × 64 Bdirectory0–1–2823–4–directory entry 4 empty → line not cached anywhere

Node 36 checks its directory: entry 4 is empty, so the line is in no cache. The current copy is in its own RAM.

address 0x24000108Node (8 bits) = 36Block (18 bits) = 4Offset (6 bits) = 8interconnection network (256 nodes)Node 20CPUblock 4Node 82CPUblock 2Node 36RAM 16 MiB2¹⁸ lines × 64 Bdirectory0–1–2823–420line sent to node 20 · entry 4 ← 20

The line goes to node 20 and directory entry 4 is set to 20: “a copy of this line is at 20”.

address 0x24000080Node (8 bits) = 36Block (18 bits) = 2Offset (6 bits) = 0interconnection network (256 nodes)Node 20CPUblock 4Node 82CPUblock 2Node 36RAM 16 MiB2¹⁸ lines × 64 Bdirectory0–1–2823–420new request: block 2 · directory: cached at node 82!

Now node 20 wants block 2 (0x24000080). The directory says the line is cached at node 82.

address 0x24000080Node (8 bits) = 36Block (18 bits) = 2Offset (6 bits) = 0interconnection network (256 nodes)Node 20CPUblocks 4, 2Node 82CPUblock 2 invalidNode 36RAM 16 MiB2¹⁸ lines × 64 Bdirectory0–1–2203–420invalidate at 82 → then send the line to 20 · entry 2 ← 20

Node 36 sends 82 an “invalidate” (with write-back if needed), then sends the line to 20; entry 2 ← 20. Point-to-point messages replace bus snooping.

In this simple design a line can be cached at only one node. Real systems track the set of copies with a bit vector or a limited pointer list; on multicore chips a “snoop filter” does the same job.

Multiprocessor Hardware

The Cache Coherence Problem

Core 0Core 1x = 5cache (stale)x = 7cache (new)bus / interconnectx = 5main memoryload x → 5 ? (expected 7)
  • Each core's private cache holds copies of the same memory. When one writes, the other's copy goes stale.
  • Coherence: all cores see writes to the same address in the same order, and a read returns the latest written value.
  • Snooping: every cache controller listens on a shared bus; fine for a few cores.
  • Directory: a list of copies is kept and messages go only to the nodes involved; scales to many cores and NUMA.
  • Two basic policies: write-invalidate (common) and write-update.

Coherence concerns a single address; the order of different addresses is the business of the memory consistency model (x86-TSO, ARM's weak model), which is why locks include memory barriers.

Multiprocessor Hardware

The MESI Protocol

statelocal readlocal writeother readother writeMMhitMhitSwrite backIwrite backEEhitMsilentS—I—SShitMBusUpgrS—I—IESBusRdMBusRdXI—I—dashed frame: state unchanged · E/S: E if no other copy
StateMeaning
M ModifiedOnly copy, dirty; memory is stale. Written back on eviction.
E ExclusiveOnly copy, clean. A write turns it into M without touching the bus.
S SharedThere may be several clean copies. To write, invalidate the others first.
I InvalidThe copy is invalid; an access is a cache miss.

Intel MESIF (F: the copy that responds) and AMD MOESI (O: dirty but shared) are variants of this protocol.

Multiprocessor Hardware

MESI Step by Step

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Iline xvalue = —Core 1Iline xvalue = —bus: —x = 0memory

Start: x is only in memory (0). The line is I in both caches.

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Eline xvalue = 0Core 1Iline xvalue = —bus: BusRdx = 0memory

C0 reads: BusRd, no other copy → E (only and clean).

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Sline xvalue = 0Core 1Sline xvalue = 0bus: BusRdx = 0memory

C1 reads: C0 snoops the request and says “I have it” → both S. Memory is still current.

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Iline xvalue = —Core 1Mline xvalue = 1bus: BusUpgrx = 0memory (stale)

C1 writes: to write in state S it broadcasts BusUpgr (invalidate) → C0 I, C1 M; memory is now stale.

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Sline xvalue = 1Core 1Sline xvalue = 1bus: BusRd + Flushx = 1memory

C0 reads: a miss. C1 flushes the dirty line (data to C0 and memory); both S, memory = 1.

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Mline xvalue = 2Core 1Iline xvalue = —bus: BusUpgrx = 1memory (stale)

C0 writes: another invalidation → C0 M, C1 I.

operation sequence1. C0: read x2. C1: read x3. C1: x = 14. C0: read x5. C0: x = 26. C0: x = 3Core 0Mline xvalue = 3Core 1Iline xvalue = —bus: —x = 1memory (stale)

C0 writes again: already M, so no bus traffic. That is why write-heavy private data is cheap.

Multiprocessor Hardware

False Sharing

struct counters {      /* bad */
  long a;   /* offset 0 */
  long b;   /* offset 8 */
};

struct padded {        /* good */
  _Alignas(64) long a;
  _Alignas(64) long b;
};

/* thread 0: for(..) s.a++;
   thread 1: for(..) s.b++; */
  • The unit of coherence is the line (64 B), not the variable.
  • Java: @Contended; Linux: ____cacheline_aligned.
  • DSM has the same problem at page granularity: if two independent variables share a page, the page bounces between machines.
64 B cache lineabCore 0: a++line: —Core 1: b++line: —no sharing in the logic, sharing in hardware

a and b are independent counters but sit in the same line.

64 B cache lineabCore 0: a++line: MCore 1: b++line: Ithe line moves across: ~50–150 ns

C0 a++: takes the line in M; C1's copy is invalidated.

64 B cache lineabCore 0: a++line: ICore 1: b++line: Mthe line moves across: ~50–150 ns

C1 b++: the line moves to C1 (M); C0's copy becomes invalid.

64 B cache lineabCore 0: a++line: MCore 1: b++line: Ithe line moves across: ~50–150 ns

When two cores write in quick alternation, the line moves every time: “ping-pong”. Two threads can run slower than one.

64 B cache line × 2abCore 0: a++line a: MCore 1: b++line b: Mseparate lines: both stay Mno coherence traffic

Fix: align or pad so each counter gets its own line. The memory cost is 56 B per counter; the gain is often a manyfold speedup.

Multiprocessor Hardware

Multicore and Heterogeneous Chips

multicore chip (homogeneous)Core 0L1L2Core 1L1L2Core 2L1L2Core 3L1L2L3 (shared, sliced) + on-chip networkmemory controllerPCIe / I/Oheterogeneous chipPbigPbigEEEEGPUNPUunified memory (LPDDR)
Multicore (CMP)

Private L1/L2, shared L3. A server socket: 192 cores (AMD EPYC 9005), 288 E-cores (Xeon 6900E).

Chiplets

Groups of cores sit on separate dies; even within one socket, L3 and memory distances differ.

Big–little cores

big.LITTLE, Apple and Intel P/E: the OS places work by energy (Linux EAS, Thread Director).

Accelerators

GPUs and NPUs on the same chip share the same memory; they are scheduled through the driver.

Multiprocessor Hardware

Simultaneous Multithreading (SMT)

(a) single threadt0t1t2t3t4t5(b) SMT: 2 hardware threadst0t1t2t3t4t5empty slot: cache miss, dependencyblue: thread A · orange: B
  • One physical core appears as two logical CPUs (Intel Hyper-Threading, AMD SMT): two register sets, shared execution units and L1/L2.
  • The gain is typically 10–30%, and zero when both threads want the same unit. The scheduler should fill idle physical cores first.
  • Security: a shared cache opens side channels (L1TF, MDS) → Linux core scheduling places only mutually trusted threads on one core. Intel Lunar Lake (2024) dropped SMT on its P-cores.

Multiprocessor Hardware

GPUs and Heterogeneous Computing

CPU: latency orientedcorecorecorecorelarge cacheGPU: throughput oriented (SIMT)SM 0SM 1SM 2SM 3SM 4SM 5HBM / GDDR memory: TB/s bandwidth
  • A GPU runs thousands of simple lanes in groups of 32 (warps/wavefronts) on the same instruction: data-parallel work (graphics, AI, scientific computing).
  • Programming: CUDA, ROCm/HIP, OpenCL, SYCL; the CPU launches a kernel, moves data and waits for the result.
  • The OS and driver manage GPU memory, time-share contexts, and provide unified/shared virtual memory (IOMMU, page faults).
  • Clusters put 4–8 GPUs per node linked by NVLink; nodes talk via RDMA over InfiniBand/RoCE.

Multiprocessor Operating Systems

Each CPU Has Its Own OS

CPU 1own OSCPU 2own OSCPU 3own OSCPU 4own OSbusmemorypart 1datapart 2datapart 3datapart 4dataOS code (read-only, shared)
  • Memory is split into one partition per CPU; each CPU runs an independent copy of the OS in its partition. The OS code can be shared; the data is separate.
  • A system call is caught and handled on the local CPU.

Pros

  • Simple: a uniprocessor OS runs almost unchanged.
  • No locks between partitions.

Cons

  • Processes are not shared: one CPU may be overloaded while another idles.
  • Pages are not shared; each OS keeps its own buffer cache → inconsistent disk blocks.

Today's equivalent: partitioned systems and separate VMs on a hypervisor; in research, the Barrelfish “multikernel” treats each core as a separate node that communicates by messages.

Multiprocessor Operating Systems

The Master-Slave Model

CPU 1master: OSCPU 2slave: processCPU 3slave: processCPU 4slave: processbusmemorymaster OSsingle ready listslave processes' memory
  • The only copy of the OS and its tables lives on the master CPU only; every system call is redirected to the master.
  • Slaves run user processes handed out by the master; with a single ready list, the load stays balanced.

Pros

  • Simple and consistent: only one CPU touches kernel data.
  • Load balancing comes for free.

Cons

  • The master becomes a bottleneck: with system-call-heavy work it does not scale past a few CPUs.
  • If the master crashes, the whole system stops.

This is the classic form of asymmetric multiprocessing (AMP). Asymmetry lives on in embedded chips today: a Cortex-M core running an RTOS beside Cortex-A cores running Linux (OpenAMP, remoteproc).

Multiprocessor Operating Systems

Symmetric Multiprocessors (SMP)

CPU 1OS + processCPU 2OS + processCPU 3OS + processCPU 4OS + processbusmemorysingle OS copyready list + lockfile table + lockuser processes
  • There is a single copy of the OS in memory; any CPU can run kernel code. A system call is handled on the CPU that made it.
  • Problem: two CPUs may touch the same kernel table at once → locks.
  • The first step was one big kernel lock (Linux BKL, 2.0), removed completely in 2.6.39 (2011). Today: thousands of fine-grained locks, RCU and per-CPU data.

Pros

  • No bottleneck; every CPU runs the kernel.
  • Memory and processes are balanced dynamically.

Cons

  • Kernel tables must be split into critical sections.
  • A lock-ordering bug → deadlock.

Multiprocessor Operating Systems

Comparing the Three Models

Separate OSMaster-slaveSMP
OS copiesone per CPUone (on the master)one (everyone runs it)
Where are system calls handled?local CPUmaster CPUlocal CPU
Ready listper CPUsingle, on the mastersingle, or per CPU + balancing
Load balancingnonegoodgood
Scalabilitygood, but no sharingmaster is a bottleneckdepends on lock design; the best
Complexitylowestlowhighest
TodayVMs, partitioningembedded AMP, legacy systemsLinux, Windows, macOS, BSD
separate operating systemsCPU1OS 1CPU2OS 2CPU3OS 3CPU4OS 4memory split into four partsmaster-slaveCPU1OSCPU2userCPU3userCPU4usermemorycalls only on CPU1SMPCPU1OSCPU2OSCPU3OSCPU4OSmemoryone kernel, everyone runs it
A multiprocessor OS differs from a multicomputer OS: in a multiprocessor all CPUs see the same physical memory; in a multicomputer each node has its own memory and communicates only by messages.

Synchronization

TSL and Bus Locking

CPU 1TSL R, LOCKCPU 2TSL R, LOCKbusLOCK = 0memory wordTSL = read + write (two separate bus cycles)two CPUs try at once

On a uniprocessor TSL (test and set lock) is an indivisible instruction. On a multiprocessor, reading and writing the memory word are separate bus cycles.

CPU 1TSL R, LOCKCPU 2TSL R, LOCKbusLOCK = 0memory word1. CPU 1 reads: 0

CPU 1 reads the lock: 0 (free).

CPU 1TSL R, LOCKCPU 2TSL R, LOCKbusLOCK = 0memory word1. CPU 1 reads: 02. CPU 2 reads: 0

Before CPU 1 writes, CPU 2 reads too: still 0.

CPU 1TSL R, LOCKCPU 2TSL R, LOCKbusLOCK = 1memory word1. CPU 1 reads: 02. CPU 2 reads: 03. CPU 1 writes: 14. CPU 2 writes: 1

CPU 1 writes 1, and CPU 2 writes 1.

CPU 1TSL R, LOCKCPU 2TSL R, LOCKbusLOCK = 1memory word1. CPU 1 reads: 02. CPU 2 reads: 03. CPU 1 writes: 14. CPU 2 writes: 1both read 0 → both in the critical section!

Result: both think “the lock is mine” and mutual exclusion breaks. Disabling interrupts does not help either: it stops only its own CPU.

CPU 1TSL R, LOCKCPU 2bus locked: waitsbusLOCK = 1memory word1. CPU 1 reads: 02. CPU 1 writes: 1 (same locked cycle)LOCK prefix / line lock

Fix: TSL locks the bus for the whole read–write. Today the x86 LOCK prefix usually just holds the cache line in state M; ARM uses LDXR/STXR or LSE CAS.

Synchronization

Ticket Lock

import java.util.concurrent.atomic.*;
public class Ticket {
  static AtomicInteger next = new AtomicInteger();
  static volatile int serving = 0;
  static int counter = 0;
  static void lock() {
    int my = next.getAndIncrement();
    while (serving != my) Thread.onSpinWait();
  }
  static void unlock() { serving++; }
  public static void main(String[] a)
      throws Exception {
    Runnable job = () -> {
      for (int i = 0; i < 100_000; i++) {
        lock(); counter++; unlock(); } };
    Thread t0 = new Thread(job);
    Thread t1 = new Thread(job);
    t0.start(); t1.start(); t0.join(); t1.join();
    System.out.println("counter = " + counter);
  }
}
next = 0next ticketserving = 0ticket being servedT0 · my = —outsideT1 · my = —outsideticket order = arrival order (FIFO)

Two counters: next hands out tickets, serving shows whose turn it is.

next = 1next ticketserving = 0ticket being servedT0 · my = 0spinningT1 · my = —outsideticket order = arrival order (FIFO)

T0 takes a ticket: getAndIncrement is atomic (x86 LOCK XADD) → my = 0.

next = 2next ticketserving = 0ticket being servedT0 · my = 0spinningT1 · my = 1spinningticket order = arrival order (FIFO)

T1 takes a ticket too: my = 1. Even if two CPUs call at the same moment, the tickets differ.

next = 2next ticketserving = 0ticket being servedT0 · my = 0critical sectionT1 · my = 1spinningticket order = arrival order (FIFO)

T0: serving == 0 → leaves the loop and does counter++.

next = 2next ticketserving = 0ticket being servedT0 · my = 0critical sectionT1 · my = 1spinningticket order = arrival order (FIFO)

T1 spins: it reads serving from the S copy in its own cache without loading the bus; onSpinWait = x86 PAUSE.

next = 2next ticketserving = 1ticket being servedT0 · my = 0outsideT1 · my = 1spinningticket order = arrival order (FIFO)

T0 releases: serving++. The write invalidates T1's copy (a single coherence message).

next = 2next ticketserving = 1ticket being servedT0 · my = 0outsideT1 · my = 1critical sectionticket order = arrival order (FIFO)

T1 reads the new value: 1 == my → it enters. The lock is fair: nobody starves.

next = 200000next ticketserving = 200000ticket being servedT0 · my = —doneT1 · my = —doneticket order = arrival order (FIFO)

2 × 100,000 increments, none lost. With an unlocked counter++, most runs end below 200,000.

Synchronization

Spinlocks and Cache Traffic

(a) spinning with TSL(b) read first, then TSL(c) MCS: each on its own flaglock lineCPU 0CPU 1CPU 2every try writes → the linebounces constantlylock lineS copyCPU 0S copyCPU 1S copyCPU 2local read: no trafficon release, everyone pouncestailCPU 2ownerCPU 0waitCPU 1waitqueuenodesreleaser writes only to the next waiter
  • (a) A plain TSL loop writes on every try: the lock line moves between CPUs nonstop, slowing the owner's work too.
  • (b) Test-and-test-and-set (TTAS): spin on the local copy; try TSL only when the lock looks free. Exponential backoff further reduces collisions.
  • (c) MCS (Mellor-Crummey & Scott, 1991): each waiter spins on its own flag in a separate line; a release is a single remote write. Linux qspinlock builds on this idea.

These locks do not prevent data inconsistency; they prevent coherence traffic (cache thrashing).

Synchronization · Step through the code

TTAS Lock: Code Trace

void lock(int *k) {
  for (;;) {
    while (*k == 1)   /* spin locally */
      ;
    if (tsl(k) == 0)  /* looked free */
      return;         /* acquired */
  }
}
void unlock(int *k) {
  *k = 0;
}
memory: kbus: 0 transactionsCPU 0M · k=1holds lockCPU 1I · k=–outsideCPU 2I · k=–outsidecache line state: M, E, S, I

CPU 0 holds the lock (k = 1, the line is M in its cache). CPU 1 and CPU 2 call lock.

memory: kbus: 1 transactionCPU 0S · k=1holds lockCPU 1S · k=1spinningCPU 2I · k=–outsidecache line state: M, E, S, I

CPU 1 reads *k: a miss, fetched over the bus. CPU 0's M copy is shared; both are S.

memory: kbus: 2 transactionsCPU 0S · k=1holds lockCPU 1S · k=1spinningCPU 2S · k=1spinningcache line state: M, E, S, I

CPU 2 reads too: three S copies. 2 bus transactions so far.

memory: kbus: 2 transactionsCPU 0S · k=1holds lockCPU 1S · k=1spinningCPU 2S · k=1spinningcache line state: M, E, S, I

As long as they see k = 1 they spin in their own caches: the bus counter does not move. A plain TSL loop would pull the line for writing on every try.

memory: kbus: 3 transactionsCPU 0M · k=0releasedCPU 1I · k=–spinningCPU 2I · k=–spinningcache line state: M, E, S, I

CPU 0 calls unlock: writing k = 0 invalidates the other copies (I); the line is M on CPU 0.

memory: kbus: 4 transactionsCPU 0S · k=0releasedCPU 1S · k=0spinningCPU 2I · k=–spinningcache line state: M, E, S, I

CPU 1's copy is invalid: it rereads (a miss), sees k = 0 and leaves the while.

memory: kbus: 5 transactionsCPU 0S · k=0releasedCPU 1S · k=0spinningCPU 2S · k=0spinningcache line state: M, E, S, I

CPU 2 also misses and sees k = 0. Both think the lock is “free” at the same time.

memory: kbus: 6 transactionsCPU 0I · k=–releasedCPU 1M · k=1holds lockCPU 2I · k=–spinningcache line state: M, E, S, I

CPU 1 does TSL first: it takes the line for writing (M), old value 0 → lock acquired.

memory: kbus: 7 transactionsCPU 0I · k=–releasedCPU 1I · k=–holds lockCPU 2M · k=1spinningcache line state: M, E, S, I

CPU 2's TSL also pulls the line (CPU 1's copy goes I), but the old value is 1: it fails. This race costs one more bus transaction.

memory: kbus: 7 transactionsCPU 0I · k=–releasedCPU 1I · k=–holds lockCPU 2M · k=1spinningcache line state: M, E, S, I

CPU 2 spins on its local copy again. 7 bus transactions in all, and the count stays put however long the wait.

Synchronization

Spin or Switch?

lock is released at t = 3 µs; context switch C = 5 µsspinspinning (3 µs)critical sectionswitchout Cother workback Ccrit. sec.036912151821µs
  • Spinning: the CPU is wasted, but work resumes the moment the lock frees. For short critical sections.
  • Switching (blocking): the CPU is given to other work; the cost is two context switches + a cold cache.
  • Rule: if the expected wait is shorter than 2C, spin; otherwise sleep.
  • Competitive strategy: spin for as long as a switch costs (2C), then sleep → at worst twice the optimal cost.
  • In practice: Linux futex + adaptive mutexes (spin while the owner runs), Windows CRITICAL_SECTION spin counts, lock inflation in Java.
  • In the kernel, only spinlocks may be used where interrupts are disabled.

Scheduling

Time Sharing: A Shared Ready Queue

16 CPUCPU 0PCPU 1QCPU 2RCPU 3idleCPU 4SCPU 5TCPU 6UCPU 7idleCPU 8VCPU 9WCPU 10XCPU 11YCPU 12ZCPU 13αCPU 14βCPU 15γpriority queues (one structure, one lock)priority 7ABCpriority 6DEpriority 5Fpriority 4GHIpriority 3JKpriority 2priority 1LMNpriority 0
  • With independent processes, all CPUs take work from one priority ready queue: the load balances itself, but the single lock becomes a bottleneck and a process may land on a CPU with a cold cache.
  • If a thread holding a spinlock runs out of quantum, the other CPUs spin in vain. Smart scheduling: the lock holder sets a flag, and the scheduler does not preempt a flagged thread but gives it a little extra time to finish its critical section.

Scheduling

Space Sharing

0A1A2B3B4B5C6C7C8A9A10B11B12B13C14C15C16D17D18D19D20D21D22C23C24D25D26D27D28D29D30idle31idleJob A: 4 CPUsJob B: 6 CPUsJob C: 8 CPUsJob D: 12 CPUs2 CPUs idle, the next job waits4 + 6 + 8 + 12 + 2 = 32
  • Related threads run at the same time on separate CPUs: when a job starts it gets as many CPUs as it asks for and keeps them until it finishes.
  • No multiprogramming per CPU → no context switches. If there are not enough CPUs, the job waits until enough are free (simplest: FCFS).
  • Drawback: when a thread blocks on I/O its CPU sits idle. HPC cluster job schedulers (Slurm, PBS) partition nodes this way today.

Scheduling

Gang Scheduling

two CPUs, 100 ms slices, jobs A and B time-sharedCPU 0A0B0A0B0CPU 1B1A1B1A1request (t = 0)A1 sees the request only at t = 100; A0 gets the reply at t = 200reply (t = 100)0 ms100 ms200 ms300 ms400 msif communicating threads run at different times, every message waits a slice

The problem: A0 and A1 communicate, but their slices are staggered. A1 sees A0's request from t = 0 only 100 ms later, and the reply waits 100 ms too.

CPU →CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5slice 0A0A1A2A3A4A5slice 1slice 2slice 3slice 4slice 5slice 6slice 7A: 6 threadsB, C: 3 eachD: 5 · E: 7

In gang scheduling, related threads form a gang. Slice 0: all 6 threads of A at once, on 6 CPUs.

CPU →CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5slice 0A0A1A2A3A4A5slice 1B0B1B2C0C1C2slice 2slice 3slice 4slice 5slice 6slice 7A: 6 threadsB, C: 3 eachD: 5 · E: 7

Slice 1: B (3) and C (3) together; all CPUs start and end slices in sync (synchronized time slicing).

CPU →CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5slice 0A0A1A2A3A4A5slice 1B0B1B2C0C1C2slice 2D0D1D2D3D4E0slice 3E1E2E3E4E5E6slice 4slice 5slice 6slice 7A: 6 threadsB, C: 3 eachD: 5 · E: 7

Slices 2–3: D (5) + E0, then E's remaining 6 threads. All members of a gang run in the same slice, so messages get immediate replies.

CPU →CPU 0CPU 1CPU 2CPU 3CPU 4CPU 5slice 0A0A1A2A3A4A5slice 1B0B1B2C0C1C2slice 2D0D1D2D3D4E0slice 3E1E2E3E4E5E6slice 4A0A1A2A3A4A5slice 5B0B1B2C0C1C2slice 6D0D1D2D3D4E0slice 7E1E2E3E4E5E6A: 6 threadsB, C: 3 eachD: 5 · E: 7

The cycle repeats. Two conditions: gang members start at the same time and all sit on different CPUs. Co-scheduling the vCPUs of a VM today is the same idea.

Multicomputers

Multicomputers and Network Interfaces

node 1CPUuser bufferkernel bufferNIC memorynetwork interface(NIC)node 2CPUuser bufferkernel bufferNIC memorynetwork interface(NIC)switchinterconnect12453: wireRDMA: the NIC writesstraight to the user buffer(no copies 1, 2, 4, 5)
  • A multicomputer (cluster): each node is CPU + RAM + network interface; memory is not shared and each node runs its own OS copy. Communication is by messages only.
  • In the worst case a message is copied 5 times, each copy costing microseconds. The fix: map NIC memory into user space and use DMA and RDMA (InfiniBand, RoCE), so the kernel never steps in.
  • Uses: large-scale simulation, HPC, AI training; TOP500 systems are multicomputers with tens of thousands of nodes.

Multicomputers

Interconnection Topologies

(a) single switch · diameter 2(b) ring (8) · diameter 4(c) grid 4×4 · diameter 6(d) double torus 4×4 · diameter 4(e) cube (8) · diameter 3(f) 4D hypercube (16) · diameter 4
n = 1,024 nodesringgrid 32×32torus 32×32hypercube (10D)
diameter (two farthest nodes)512623210
number of links1.0241.9842.0485.120

Diameter = worst-case hop count; log₂n for a hypercube. Today's supercomputers mostly use fat-tree and dragonfly topologies; Fugaku has a 6D torus (Tofu) network.

Multicomputers

Routing in a Hypercube

0000000100100011010001010110011110001001101010111100110111101111lower cube: first bit 0upper cube: first bit 1target 1011hops: 0

A 4D hypercube: 16 nodes, each linked to 4 neighbors that differ in exactly one bit. A message goes from 0000 to 1011.

0000000100100011010001010110011110001001101010111100110111101111lower cube: first bit 0upper cube: first bit 1target 1011hops: 1

Source ⊕ target = 1011: three bits differ. The leftmost differing bit is fixed first → 1000 (from the lower to the upper cube).

0000000100100011010001010110011110001001101010111100110111101111lower cube: first bit 0upper cube: first bit 1target 1011hops: 2

Next differing bit: 1010.

0000000100100011010001010110011110001001101010111100110111101111lower cube: first bit 0upper cube: first bit 1target 1011hops: 3

Last bit: 1011. Hops = number of differing bits (Hamming distance) = 3; the worst case is d = log₂16 = 4. The rule is local, so no routing table is needed.

Multicomputers · Step through the code

E-Cube Routing: Code Trace

void route(int src, int dst, int d) {
  int u = src;
  for (int b = d - 1; b >= 0; b--) {
    int m = 1 << b;
    if ((u ^ dst) & m) {  /* bit differs */
      u ^= m;             /* go to neighbor */
      printf("%d ", u);
    }
  }
}
0000000100100011010001010110011110001001101010111100110111101111u = 0110 · u ⊕ dst = 1011

A message from 0110 (6) to 1101 (13). u ⊕ dst = 1011: three bits differ, so the shortest path is 3 hops. Dashed edges are the 4th dimension (between the two 3-cubes).

0000000100100011010001010110011110001001101010111100110111101111u = 1110 · u ⊕ dst = 0011 · bit 3

b = 3: the bit differs → flip bit 3 of u; the message moves to neighbor 1110 (14).

0000000100100011010001010110011110001001101010111100110111101111u = 1110 · u ⊕ dst = 0011 · bit 2

b = 2: this bit is the same (1) in u and dst; no hop.

0000000100100011010001010110011110001001101010111100110111101111u = 1100 · u ⊕ dst = 0001 · bit 1

b = 1: the bit differs → flip bit 1 of u; the message moves to neighbor 1100 (12).

0000000100100011010001010110011110001001101010111100110111101111u = 1101 · u ⊕ dst = 0000 · bit 0

b = 0: the bit differs → flip bit 0 of u; the message moves to neighbor 1101 (13).

0000000100100011010001010110011110001001101010111100110111101111u = 1101 · u ⊕ dst = 0000

The target is reached in 3 hops: 0110 → 1110 → 1100 → 1101. Each node decides from its own address and the target alone; since bits are always fixed left to right, the path has no cycles.

Multicomputers

Store-and-Forward or Cut-Through

Aswitch 1switch 2Bstore-and-forward: each switch receives the whole packet, then forwards itlatency ≈ hops × (packet / bandwidth)t = 0 µs

Example: a 4 KiB packet over 10 Gb/s links; crossing one link takes 4096 × 8 / 10¹⁰ ≈ 3.28 µs.

Aswitch 1switch 2Bstore-and-forward: each switch receives the whole packet, then forwards itlatency ≈ hops × (packet / bandwidth)t = 3.28 µs

Store-and-forward: switch 1 buffers the whole packet, checks it, then forwards it.

Aswitch 1switch 2Bstore-and-forward: each switch receives the whole packet, then forwards itlatency ≈ hops × (packet / bandwidth)t = 9.83 µs

Three links → 3 × 3.28 = 9.83 µs. Latency grows linearly with the hop count.

Aswitch 1switch 2Bcut-through / wormhole: forwarded as soon as the header is readpieces of the packet (flits) on several links at oncet ≈ 3.38 µs (one packet time + small header delays)

Cut-through and wormhole routing: the path is set up when the header arrives and the packet flows like a pipe → ≈ 3.38 µs. HPC networks and data-center switches work this way.

Multicomputers

Blocking and Nonblocking Send

sending processrunningtrap, copyBLOCKED: message in transitrunningsend(m)message sent, returnsimple and safe, but the CPU idles for the whole transmission

(a) Blocking (synchronous) send: the call does not return until the message has been transmitted.

sending processrunningtrapkeeps running at oncesend(m)buffer m is still being sent!we get parallelism, but what if the sender changes m before transmission ends?

(b) Nonblocking (asynchronous) send: returns at once, but the buffer is off-limits until transmission ends.

OptionCost
Blocking sendCPU idles during transmission
Nonblocking + copy to kernelCPU time for the extra copy
Nonblocking + interrupt when donehard to program (race conditions)
Copy on write (CoW)an extra copy anyway if the buffer is written

Tanenbaum's four options. The receiver side is similar: a blocking receive, polling, or a pop-up thread / active message.

In MPI, MPI_Send/MPI_Recv are blocking and MPI_Isend/MPI_Irecv + MPI_Wait are nonblocking; on Linux, io_uring and zero-copy MSG_ZEROCOPY are modern answers to the same problem.

Multicomputers

Remote Procedure Call (RPC)

client machineClientClient stubOperating system (kernel)server machineServerServer stubOperating system (kernel)network12345goal: make a remote call look like a local one (Birrell and Nelson, 1984)

The client says result = add(3, 4); it looks like an ordinary local call.

client machineClientClient stubOperating system (kernel)server machineServerServer stubOperating system (kernel)network12345

1: The client calls the client stub as if it were a local procedure.

client machineClientClient stubOperating system (kernel)server machineServerServer stubOperating system (kernel)network12345add | 3 | 4

2: The stub marshals the parameters into a message and makes a system call into the kernel.

client machineClientClient stubOperating system (kernel)server machineServerServer stubOperating system (kernel)network12345add | 3 | 4

3: The kernel sends the message over the network to the server machine.

client machineClientClient stubOperating system (kernel)server machineServerServer stubOperating system (kernel)network12345

4: The server kernel hands the message to the server stub.

client machineClientClient stubOperating system (kernel)server machineServerServer stubOperating system (kernel)network12345reply (7) takes the reverse path: server → stub → kernel → network → client

5: The stub unpacks the parameters and calls the server procedure locally; the reply returns the same way.

Difficulties: pointers (another address space → copy/restore), arrays of unknown length, global variables, partial failures (at least once or at most once?). Stubs are generated automatically from an interface definition; today gRPC + Protocol Buffers, Thrift, Java RMI.

Multicomputers

Distributed Shared Memory (DSM)

ApplicationRuntimeOperating systemHardware(a) hardwareApplicationRuntimeOperating systemHardware(b) operating systemApplicationRuntimeOperating systemHardware(c) user softwarethe orange layer creates the illusion of a shared address space

DSM gives machines that share no memory the illusion of shared memory. The layer can be hardware (NUMA), the OS (page-based, Li and Hudak 1989) or a user library.

shared virtual address space: pages 0–150123456789101112131415CPU 0 memory0259CPU 1 memory136810CPU 2 memory47111214CPU 3 memory1315CPU 0 touches page 10 → page fault (not on CPU 0)

Pages are spread across the machines. When CPU 0 touches page 10, the MMU raises a page fault.

shared virtual address space: pages 0–150123456789101112131415CPU 0 memory025910CPU 1 memory1368CPU 2 memory47111214CPU 3 memory1315the OS fetches the page over the network; CPU 1 no longer has it

The OS finds who has the page and requests it; the page moves to CPU 0 and the instruction is restarted.

shared virtual address space: pages 0–150123456789101112131415CPU 0 memory025910CPU 1 memory136810CPU 2 memory47111214CPU 3 memory1315read-only page replicated: copies on CPU 0 and CPU 1; a write invalidates them

A read-only page is replicated: both CPUs read locally. On a write the other copies are invalidated; independent variables on the same page cause false sharing.

A 4 KiB page is expensive and coarse-grained on a network, so DSM gave way to message passing (MPI) and PGAS languages. Its modern counterpart is cross-machine memory pooling with CXL.

Multicomputers

Graph-Theoretic Deterministic Allocation

3223231425114321ABCDGHEFItotal traffic 39

9 processes (vertices) and the message traffic between them (edge weights). Assign 3 processes to each of 3 machines; goal: minimize network traffic.

3223231425114321ABCDGHEFInode 1: A, B, Cnode 2: D, E, Fnode 3: G, H, Inetwork traffic = 24

Alphabetical assignment: the cut (intermachine) edges total 24.

3223231425114321ABCDGHEFInode 1: A, B, Cnode 2: D, G, Hnode 3: E, F, Inetwork traffic = 15

The best partition (all 280 balanced options tried): 15. Chatty processes share a machine; the problem is NP-hard in general and is solved with heuristics.

Multicomputers

Distributed Heuristic Load Balancing

9367148load ≥ 7: overloadedrandom nodes are probed:“what is your load?”red: overloaded (≥ 7)green: lightly loaded (≤ 2)

Sender-initiated: when a new job arrives, a node over its threshold probes random nodes (Eager et al., 1986).

9367148probed: 3 and 1one below the threshold is foundred: overloaded (≥ 7)green: lightly loaded (≤ 2)

Probing is limited (e.g. N = 3); if a node below the threshold turns up, the job goes there, otherwise it runs locally.

8367248sender-initiated:job sent to the node with load 1red: overloaded (≥ 7)green: lightly loaded (≤ 2)

The job has moved. Drawback: when the system is fully loaded, every probe is wasted and adds load.

8367048receiver-initiated:an idle node looks for workred: overloaded (≥ 7)green: lightly loaded (≤ 2)

Receiver-initiated: a node that finishes a job and goes idle looks for work itself; the probing cost falls on the idle node.

7367148overloaded nodehands one job to the idle nodered: overloaded (≥ 7)green: lightly loaded (≤ 2)

A loaded node handed off a job. Works well under high load. Its modern form is work stealing: Cilk, Java ForkJoinPool, the Go and Tokio runtimes.

Virtualization

Hypervisors: Type 1 and Type 2

HardwareHypervisor (VMM)guest OSLinuxapplicationsguest OSWindowsapplicationsguest OSFreeBSDapplications(a) Type 1 (bare metal)HardwareHost OSapplicationHypervisor (process)guest OS + applications(b) Type 2 (hosted)
  • Type 1: VMware ESXi, Xen (XCP-ng), Microsoft Hyper-V; Linux KVM makes the kernel a hypervisor (most clouds: AWS Nitro, Google). With Hyper-V on, Windows itself runs in a root partition.
  • Type 2: VirtualBox, VMware Workstation/Fusion, Parallels, UTM. A pure type 2 interprets or binary-translates (QEMU without KVM); a practical one uses VT-x/AMD-V via a kernel module.
  • Popek–Goldberg (1974): a machine is virtualizable if all sensitive instructions are privileged (trap in user mode).
  • Classic x86 failed this test (e.g. POPF fails silently in user mode) → binary translation, then VT-x/AMD-V.

Uses: server consolidation, moving a running VM to another host (live migration), snapshots; the foundation of the cloud. In type 2, I/O goes through the host OS, so overhead is higher.

Virtualization

Trap and Emulate: VT-x / AMD-V

virtual machine (VMX non-root mode)guest applicationguest kernel: mov cr3, rax · out 0x3f8 · hltHypervisor (VMX root mode)hardwareVMCSguest statehost stateexit reasonall state lives here

The guest OS believes it runs in kernel mode, but the hypervisor is really in control. Intel VT-x / AMD-V (2005–06) add a new mode: root (hypervisor) and non-root (guest).

virtual machine (VMX non-root mode)guest applicationguest kernel: mov cr3, rax · out 0x3f8 · hltHypervisor (VMX root mode)hardwareVMCSguest statehost stateexit reasonsensitive instruction!

The guest kernel executes a sensitive instruction: changing the page table (CR3), an I/O port, HLT…

virtual machine (VMX non-root mode)guest applicationguest kernel: mov cr3, rax · out 0x3f8 · hltHypervisor (VMX root mode)hardwareVMCSguest statehost stateexit reasonVM exitreason: CR access

The hardware does not execute it: a VM exit traps to the hypervisor, and the reason is written to the VMCS.

virtual machine (VMX non-root mode)guest applicationguest kernel: mov cr3, rax · out 0x3f8 · hltHypervisor (VMX root mode)hardwareVMCSguest statehost stateexit reasonVM exitreason: CR accessVM entryemulated:shadow/EPT updated

The hypervisor emulates the instruction's effect, updates the virtual hardware state and returns to the guest with a VM entry. An exit costs ~1 µs; frequent exits hurt performance.

Before hardware support, VMware (1999) replaced sensitive instructions at run time through binary translation. Only the guest OS's sensitive instructions trap; the rest run directly on the hardware.

Virtualization

Paravirtualization

Hardware(a) bare metalLinuxstraight to hardwareHardware(b) full virtualizationunmodified Linuxsensitive instruction → trap / translationVMware (hypervisor)Hardware(c) paravirtualizationmodified Linuxno sensitive instructionshypercall interfaceXen (hypervisor)
  • The guest OS is modified: instead of sensitive instructions it makes hypercalls straight to the hypervisor — the one-layer-down analogue of a system call.
  • No trap-and-emulate cost, so it is faster; but it needs an OS whose source can be changed (Xen 2003).
  • VMI (Virtual Machine Interface) was proposed so one kernel could run on bare metal and on different hypervisors; it was removed from Linux in 2011. Paravirtualization lives on in drivers today: virtio disk/network, paravirtual clocks, Hyper-V enlightenments.

In today's clouds, CPU and memory are fully virtualized with hardware support; I/O uses virtio or direct device assignment with SR-IOV.

Virtualization

Memory Virtualization: Shadow Tables and EPT

guest virtual (GVA)guest physical (GPA)host physical (HPA)guest page table(managed by the guest OS)EPT / NPT(managed by the hypervisor)shadow page table: GVA → HPA directly (kept current in software)2D walk on a TLB miss: 4 levels → (4+1)² − 1 = 24 memory accesses · 5 levels → 35
  • Shadow page tables: the hypervisor makes the guest's tables read-only, traps every change and updates its GVA → HPA copy. Lots of VM exits.
  • EPT (Intel, 2008) / NPT (AMD): the hardware walks both tables together; the guest changes its own table without traps.
  • TLB misses get more expensive → huge pages (2 MiB / 1 GiB) and VM-tagged TLBs (VPID/ASID) are used.
  • Overcommit: a balloon driver reclaims memory from the guest; KSM merges identical pages (copy on write).

Virtualization

Containers and Virtual Machines

(a) virtual machinesHardwareHypervisorguest kernellibrariesapp 1guest kernellibrariesapp 2guest kernellibrariesapp 3(b) containersHardwareHost kernel: namespaces + cgroupscontainer runtime (containerd, runc)librariesapp 1librariesapp 2librariesapp 3
Virtual machineContainer
Isolation boundaryhardware level, separate kernelsame kernel: namespaces, cgroups, seccomp
Startup / overheadseconds, GBs of memorymilliseconds, MBs of memory
Different OSyes (Windows on Linux)no (same kernel family)
ExampleKVM, Hyper-V, ESXiDocker, Podman, Kubernetes pods

In between: Firecracker micro-VMs that boot in milliseconds (AWS Lambda), gVisor and Kata Containers combine container convenience with VM isolation. WSL 2 also runs a real Linux kernel in a lightweight Hyper-V VM; Linux containers on Windows run in that VM.

Distributed Systems

Distributed Systems and Middleware

WindowshardwareLinuxhardwaremacOShardwareLinuxhardwareCommon applicationMiddlewareuniform interface: a document, file, object or coordination modelnetwork
  • Distributed system: autonomous computers communicating over a network; each manages its own OS, memory and resources, yet the whole tries to look like one coherent system to the user.
  • Nodes may run different hardware and OSes, belong to different organizations and span the globe. The common ground is the middleware layer.
  • Four paradigms: document (the Web), file (NFS, SMB), object (CORBA, RMI, gRPC), coordination (Linda, publish-subscribe).

True distributed operating systems (Amoeba, Plan 9) stayed in research; today each node runs its own OS and distribution is handled by middleware and cluster managers (Kubernetes).

Distributed Systems

Comparing Three Kinds of Systems

ItemMultiprocessorMulticomputerDistributed system
Node configurationCPUCPU, RAM, net interfacecomplete computer
Peripheralsall sharedshared (maybe except disk)full set per node
Locationsame racksame roompossibly worldwide
Internode communicationshared RAMdedicated interconnecttraditional network
Operating systemone, sharedmany, the samemany, possibly different
File systemone, sharedone, sharedeach node has its own
Administrationone organizationone organizationmany organizations
multiprocessorRAMCCCCCCshared memorymulticomputerinterconnectC+MC+MC+MC+MC+MC+Mmessages, dedicated netdistributed systeminternetcomp.comp.comp.comp.complete computers, wide area

Tanenbaum (MOS 4e, Fig. 8-29). In a distributed system peripherals are not shared: each node has its own full set. In a multicomputer, the disk is not always shared either.

Distributed Systems

Network Hardware and Encapsulation

(a) classic: shared cableB1B2B3B4CSMA/CD: back off on collision(b) switched: starswitchB1B2B3B4full duplex, no collisionsencapsulation: each layer adds a headerapplicationdata (HTTP)transportTCPdatanetworkIPTCPdatalinkEthIPTCPdataFCSthe receiver strips headers in reverse order
  • Classic Ethernet (1976) shared one cable; today every computer connects to a switch, with full-duplex links of 1–400 Gb/s.
  • Routers join LANs to each other, and local networks connect to ISPs and the Internet backbone (high-speed fiber, exchange points).

Distributed Systems

Network Protocols and Service Types

ProtocolLayerServiceNote
IP (v4/v6)networkconnectionless, unreliableaddressing and routing
TCPtransportconnection-oriented, reliable byte streamacks, retransmission, flow and congestion control
UDPtransportconnectionless, unreliable datagramshas a checksum, no retransmission. DNS, games, voice
QUICtransport (over UDP)connection-oriented, reliable, encryptedused by HTTP/3 (2022)
DNSapplicationname → IP addresshierarchical, distributed, cached
HTTP/HTTPSapplicationWeb documentsHTTPS = HTTP + TLS, port 443
SMTPapplicationmail transferserver to server; reading via IMAP
FTP → SFTPapplicationfile transferFTP is unencrypted; replaced by SFTP/HTTPS
SNMPapplicationnetwork device monitoringv3 adds authentication and encryption

Reliable service: guaranteed delivery and order, with acknowledgments (ACK). Connection-oriented: set up, then data, then teardown; connectionless service sends each packet independently with no setup cost. UDP detects a corrupted packet with its checksum but neither fixes nor resends it.

Distributed Systems

Document-Based Middleware: The Web

BrowserclientDNS resolverwww.example.org203.0.113.10documents and links

The user types www.example.org/index.html. The Web is a huge directed graph of linked documents.

BrowserclientDNS resolverwww.example.org203.0.113.10A? www.example.org

1–2: The browser (through the OS resolver) asks DNS for www.example.org's address; answer: 203.0.113.10 (cached).

BrowserclientDNS resolverwww.example.org203.0.113.10SYNSYN-ACKACK + TLS ClientHelloTLS ServerHello, certificate

3: A TCP connection to the server's port 443 and the TLS handshake (one step in HTTP/3, with QUIC).

BrowserclientDNS resolverwww.example.org203.0.113.10GET /index.html HTTP/2200 OK + HTML

4–5: A GET /index.html request; the server returns the HTML and the browser starts rendering text.

BrowserclientDNS resolverwww.example.org203.0.113.10logo.png · style.css · app.js (parallel streams)

6: Images, CSS and scripts in the HTML are fetched over the same connection, as parallel streams in HTTP/2–3; the connection stays open for a while.

Port 80 and a new TCP connection per object belong to the HTTP/1.0 era; today the defaults are HTTPS (443), persistent connections and HTTP/2–3 multiplexing.

Distributed Systems

File-System-Based Middleware

ClientServerfilecopy1. the whole file is downloaded2. read/written locally3. new version uploaded back

Upload/download model: the whole file is copied to the client, used and sent back (FTP-like; early AFS). Simple, but costly for large files.

ClientServerfileread(fd, 4 KiB)datawrite(fd, ...)okthe file stays on the server

Remote access model: the file stays on the server; the client sends read/write requests over the network (NFS, SMB).

server 1server 2rootABCrootDEsquare: directory · circle: file

Two file servers, each with its own directory tree.

client 1roots1ABCs2DEclient 2roots1ABCs2DEsame namespace: /s1/A is the same file on every client

All clients mount the servers in the same place → a single shared namespace.

client 1roots1ABCs2DEclient 2roots2DEhomenotesclient 2 has no /s1, while /s2/D is the same on both

If clients mount different things, each sees a different view; the same path may name different files on different clients.

Naming: machine:/path (not location transparent) → mounting remote systems into the local tree → one namespace identical on all machines (location independence). Today: NFSv4, SMB 3, object storage (S3).

Distributed Systems

Semantics of File Sharing

single machinea b cA: write(c)B: read → abcevery read sees the latest write: UNIX semantics

On one processor operations are sequential: A appends “c”, then B reads and sees “abc” (sequential consistency).

File servera bClient 1cache: a b cClient 2reads → a b1. read a b2. c appended locally3. stale data!

In a distributed system client 1 changes the file in its cache; the server does not know yet. Client 2 reads from the server and sees the stale “ab”.

ApproachRuleExample
UNIX semanticsevery write is visible to all at oncesingle machine; remote access without caching
Session semanticschanges become visible when the file is closedAFS; NFS close-to-open consistency
Immutable filesfiles cannot be written; a new one is createdobject storage, Git objects
Transactionsall or nothing, serializabledatabases

Each solution trades consistency against performance.

Distributed Systems

Object-Based Middleware: CORBA

client machineClientclient stubORB (Object Request Broker)operating systemserver machineServer objectskeletonORB (Object Request Broker)operating systemIIOP
  • IDL: interfaces are defined independent of language; the compiler generates the client stub and the server skeleton.
  • ORB: locates the object, marshals the call and sends it over IIOP: network transparency.
  • SII (static invocation): through stubs generated from IDL at compile time; fast and type-checked.
  • DII (dynamic invocation): the interface is discovered at run time (interface repository).

SII and DII are separate mechanisms; SII is not a compiled form of DII. CORBA (OMG, 1991) survives in legacy systems; new systems prefer gRPC/Protobuf, REST and Java RMI.

Distributed Systems

Coordination-Based Middleware: Linda

tuple space("abc", 2, 5)("matrix-1", i, j, 0.5)("bit", 3)Process AProcess Bout · in · rd · eval

Linda (Gelernter, 1985): processes do not know each other; they communicate through a shared, persistent tuple space. A tuple is a record of typed fields.

tuple space("abc", 2, 5)("matrix-1", i, j, 0.5)("bit", 3)("abc", 2, 9)Process AProcess Bout("abc", 2, 9)

out puts a tuple into the space (nonblocking).

tuple space("abc", 2, 5)("matrix-1", i, j, 0.5)("bit", 3)("abc", 2, 9)Process AProcess Bin("abc", 2, ? i)template matches

in searches with a template: same number of fields, matching types field by field, equal constant fields. ? i is a formal parameter. With several matches, one is chosen arbitrarily.

tuple space("matrix-1", i, j, 0.5)("bit", 3)("abc", 2, 9)Process AProcess Bi = 5tuple taken from the space

The matching tuple is removed from the space and i = 5 is assigned. With no match, in blocks. rd reads without removing; eval starts a new process. Synchronization comes for free.

Distributed Systems · Step through the code

Linda in(): Code Trace

Tuple *in_find(Template *s) {
  for (Tuple *d = space; d; d = d->next) {
    if (d->n != s->n) continue;
    int ok = 1;
    for (int k = 0; ok && k < s->n; k++)
      if (d->type[k] != s->type[k] ||
          (s->fixed[k] && !equal(d, s, k)))
        ok = 0;          /* field mismatch */
    if (ok) { remove(d); return d; }
  }
  return NULL;           /* block */
}
template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d4"abc"25d5"abc"29

The space holds 5 tuples. A process calls in("abc", 2, ? int i): 3 fields, the first two constant, the third a formal parameter of type int.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d4"abc"25d5"abc"29

d1: 2 fields, but the template has 3 → skipped at once (continue).

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d4"abc"25d5"abc"29

d2, field 0: "xyz" ≠ "abc" → ok = 0, move to the next tuple.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d4"abc"25d5"abc"29

d3, field 1: 1 ≠ 2 → ok = 0, move to the next tuple.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d4"abc"25d5"abc"29

d4: field count and types fit, constants are equal; the formal field i gets 5. The tuple is removed from the space.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d5"abc"29

d4 is gone from the space; d5 would also have matched, but the first one found was taken (Linda does not define which is chosen). The same call is made again.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d5"abc"29

d1: 2 fields, but the template has 3 → skipped at once (continue).

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d5"abc"29

d2, field 0: "xyz" ≠ "abc" → ok = 0, move to the next tuple.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d5"abc"29

d3, field 1: 1 ≠ 2 → ok = 0, move to the next tuple.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14d5"abc"29

d5: field count and types fit, constants are equal; the formal field i gets 9. The tuple is removed from the space.

template:"abc"2? int ituple space:d1"abc"2d2"xyz"27d3"abc"14

A third in("abc", 2, ? int i) finds no match and returns NULL: the process blocks until someone outs a matching tuple.

Distributed Systems

Publish-Subscribe and Jini

sensor 1publishersensor 2publisherstock exchangepublisherbrokertemp/izmirtemp/giresunstock/XYZdashboardsubscriberalarmsubscribermobile appsubscriber
  • A publisher publishes on a topic and a subscriber subscribes to topics it cares about; neither knows the other's identity: decoupled in time and space.
  • Subscribers and topics are added dynamically → scalable and flexible. Delivery guarantees depend on broker settings (at most once / at least once / durable queue).
  • Today: MQTT (IoT), Apache Kafka, RabbitMQ, cloud queues.
  • Jini (Sun, 1998): Java devices discover each other on a network without prior configuration, register with a lookup service and use each other via Java RMI; leases make registrations expire on their own.
  • It continued as Apache River but is no longer active; its ideas live on in service discovery (mDNS/DNS-SD, Consul, Kubernetes services).

Distributed Systems

Cloud and Clusters

on-premisesapplicationdataruntimemiddlewareOSvirtualizationservers, storage, networkIaaSapplicationdataruntimemiddlewareOSvirtualizationservers, storage, networkPaaSapplicationdataruntimemiddlewareOSvirtualizationservers, storage, networkFaaSapplicationdataruntimemiddlewareOSvirtualizationservers, storage, networkSaaSapplicationdataruntimemiddlewareOSvirtualizationservers, storage, networkuser managesprovider manages
  • Cluster: servers linked by a fast network under one administration (HPC, data centers). Cloud: those resources offered over the network on demand, pay as you go (AWS, Azure, Google Cloud).
  • Infrastructure: KVM/Nitro virtualization, containers and Kubernetes (schedules pods onto nodes and restarts failed ones: today's counterpart of a distributed OS).
  • Every topic in this chapter comes together here: NUMA servers, virtual machines, RDMA networks, load balancing, publish-subscribe queues.

Summary

Summary

Hardware

UMA (bus, crossbar n², omega (n/2)log₂n) and NUMA; snooping or directories for coherence.

Caches

MESI transitions; writes generate invalidations; false sharing happens at line granularity and is fixed by padding.

OS and locks

Separate OS, master-slave, SMP; bus/line locking for TSL, TTAS, ticket and MCS; spin for short waits, sleep for long ones.

Scheduling

A shared queue means one lock and cold caches; space sharing and gang scheduling for related threads.

Multicomputers

Topology diameter, store-and-forward and cut-through, blocking/nonblocking send, RPC, DSM, load balancing.

Virtualization and distribution

Type 1/2, VT-x, EPT, containers; middleware: document, file, object, coordination; the cloud.

Summary

Review Questions

  1. How many crosspoints does a 64 × 64 crossbar have, and how many 2 × 2 switches does a 64-input omega network have?
  2. In the omega network, which switches and outputs does a CPU 011 → memory 110 request pass through?
  3. Local 80 ns, remote 140 ns, 75% of accesses local: what is the average access time?
  4. In MESI, what is the bus-level difference between writing a line in state E and in state S?
  5. Why can two threads' separate counters slow each other down? What is the fix?
  6. In which two ways is a ticket lock better than a plain TSL loop?
  7. What are the two conditions of gang scheduling, and what problem does it solve?
  8. What is the diameter of a 256-node hypercube? Of a 16 × 16 torus?
  9. With 4-level tables and EPT, how many memory accesses does a TLB miss need in the worst case?
  10. Which of the tuples ("abc", 2, 5) and ("abc", 2.0, 5) does in("abc", ? x, 5) (x int) match?
Answers
  1. 64² = 4,096 crosspoints; log₂64 = 6 stages × 32 = 192 switches.
  2. 1D (bit 1 → lower), 2D (bit 1 → lower), 3D (bit 0 → upper).
  3. 0.75 · 80 + 0.25 · 140 = 95 ns.
  4. E → M is silent (no bus traffic); S → M must broadcast an invalidation (BusUpgr).
  5. They share one 64 B line: every write steals the line from the other core (false sharing). Align or pad them onto separate lines.
  6. It is fair (FIFO, no starvation), and waiters only read, never write → less coherence traffic.
  7. Members run at the same time on different CPUs, and slices start in sync. Communicating threads no longer wait a whole slice for each other.
  8. log₂256 = 8; 2 · ⌊16/2⌋ = 16.
  9. (4 + 1)² − 1 = 24.
  10. Only ("abc", 2, 5): the second field must be of type int.

Summary

References

  • A. S. Tanenbaum, H. Bos, Modern Operating Systems, 4th ed. (2014) / 5th ed. (2023), Pearson — Chapter 8: Multiple Processor Systems; Chapter 7: Virtualization and the Cloud.
  • A. Silberschatz, P. B. Galvin, G. Gagne, Operating System Concepts, 10th ed., Wiley, 2018 — Section 5.5 (Multi-Processor Scheduling), Chapters 18 (Virtual Machines) and 19 (Networks and Distributed Systems).
  • R. H. Arpaci-Dusseau, A. C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces (OSTEP), ostep.org — “Multiprocessor Scheduling”, “Locks”, “Distributed Systems”, “Sun’s NFS”, “AFS”, Appendix: “Virtual Machine Monitors”.
  • J. L. Hennessy, D. A. Patterson, Computer Architecture: A Quantitative Approach, 6th ed., Morgan Kaufmann, 2017 — Chapter 5: Thread-Level Parallelism (coherence, MESI, directory protocols).
  • J. M. Mellor-Crummey, M. L. Scott, “Algorithms for Scalable Synchronization on Shared-Memory Multiprocessors”, ACM TOCS, 1991; A. D. Birrell, B. J. Nelson, “Implementing Remote Procedure Calls”, ACM TOCS, 1984; G. J. Popek, R. P. Goldberg, CACM, 1974.
  • Linux kernel documentation: “CFS/EEVDF Scheduler”, “Scheduler Domains”, “NUMA Memory Policy”, “Core Scheduling”; Intel SDM Vol. 3 (VMX, EPT).

The End

Chapter 12: Multiprocessor Systems
Previous: Chapter 11: Input/Output · Next: Chapter 13: Security
1 / 1