Chapter 11: Input/Output

Operating Systems
I/O hardware and busesDevice registers, port-mapped and memory-mapped I/OPolling (programmed I/O)Interrupts, vectors, MSI-XDMA, scatter-gather, IOMMUI/O software layersBuffering, caching, spoolingBlocking and asynchronous I/O, io_uringClocks and timersDisks and disk schedulingSSD, NVMe, blk-mqRAID and stable storage
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Hardware

Explain controllers, device registers, PCIe/USB/NVMe buses, and port-mapped versus memory-mapped I/O.

Transfer methods

Compare polling, interrupt-driven I/O and DMA; state the role of MSI-X and the IOMMU.

Software layers

Trace a read() through the interrupt handler, the driver, the device-independent layer and the user library.

Interfaces

Distinguish blocking, nonblocking and asynchronous I/O; tell epoll from io_uring.

Disks

Compute access time; find the total head movement of FCFS, SSTF, SCAN, C-SCAN, LOOK and C-LOOK.

SSD and RAID

Explain the FTL, wear leveling, TRIM and blk-mq; RAID 0/1/5/6/10 and recovery with XOR parity.

Introduction

I/O System Overview

Processor (SoC)cores · memory controllerPCIe root complex · APICDRAM (DDR5)main memorymemory busGPUPCIe 5.0 x16NVMe SSDPCIe 4.0/5.0 x4USB4 / Thunderbolt40–80 Gb/sChipset (PCH)I/O controllersDMI ≈ PCIe x4/x8SATAHDD / opticalUSB 3.xkeyboard, mouseEthernetNICWi-Fi / BTM.2 cardAudioHD Audiocontroller + device
  • The OS issues commands to devices, reads their status, catches interrupts and handles errors.
  • It gives applications a uniform, layered, device-independent interface.
  • Most kernel code is device drivers (about 60–70% of the Linux source lives in drivers/).

Introduction

Buses: PCIe, USB4, SATA

  • Bus: a set of wires carrying data, address and control signals between components. Old PCs used shared parallel buses (ISA, PCI); today we use point-to-point serial links (PCIe, USB, SATA).
  • A PCIe lane is full duplex; a link bundles x1, x4, x8 or x16 lanes. Bandwidth is counted per direction.
LinkRaw rateEffective data rateNote
PCIe 4.016 GT/s / lane≈ 1.97 GB/s / lane; x4 ≈ 7.9 GB/s128b/130b encoding; NVMe SSDs
PCIe 5.032 GT/s / lane≈ 3.94 GB/s / lane; x16 ≈ 63 GB/sGPUs, 14 GB/s-class SSDs
PCIe 6.064 GT/s (PAM4)≈ 7.6 GB/s / lane; x16 ≈ 121 GB/sFLIT + FEC; growing in servers
USB 2.0 / 3.2480 Mb/s / 5–20 Gb/s≤ 60 MB/s / ≤ 2.5 GB/sHost-polled protocol
USB4 v1 / v240 / 80 Gb/sPCIe and DisplayPort tunnelingThunderbolt 4/5 compatible
SATA III6 Gb/s≈ 600 MB/s (8b/10b)AHCI: one queue, 32 commands
Ethernet1 / 10 / 100 / 400 Gb/s125 MB/s … 50 GB/sData center: 400–800 GbE

Math: a PCIe 4.0 lane gives 16 GT/s × 128/130 ÷ 8 = 1.97 GB/s. FireWire, 56K modems and 10/100 Mb/s Ethernet are history. The Pentium-era northbridge is gone: the memory controller moved into the CPU (AMD 2003, Intel 2008); fast devices attach to the CPU over PCIe, slow ones to the chipset.

Introduction

Device Data Rates

1 B/s1 KB/s1 MB/s1 GB/s100 GB/sKeyboard10 B/sMouse (1 kHz gaming mouse)8 KB/sGigabit Ethernet125 MB/sHDD (7200 rpm, sequential)250 MB/sSATA SSD550 MB/sUSB 3.2 Gen 21.25 GB/sUSB4 (40 Gb/s)5 GB/sNVMe PCIe 4.0 x47 GB/sNVMe PCIe 5.0 x414 GB/s400 GbE50 GB/sDDR5-6400 (one channel)51.2 GB/sPCIe 5.0 x1663 GB/slog scale: each gridline is 1000×

A keyboard and PCIe 5.0 x16 differ by a factor of ≈ 6 × 109: the same OS must handle a few keystrokes per second and millions of NVMe requests per second. Values are typical upper limits; real rates depend on signal quality, protocol overhead and the device.

Introduction

Device Classes: Block, Character, Network

ClassExamplePropertyTypical interface
BlockHDD, SSD, USB stick, SD cardFixed-size, addressable blocks (512 B / 4 KiB); each block is read or written independentlyread, write, lseek; file system, mmap
CharacterKeyboard, mouse, serial port, printer, audioA byte stream; not addressable, no seekread/write, ioctl; /dev/ttyS0
NetworkEthernet, Wi-FiPacket based; neither block nor characterSocket API (send, recv); no file in /dev
OtherClocks, timers, GPUsNeither stream nor block: they raise interrupts or are memory-mappedtimerfd, DRM/KMS, mmap
$ ls -l /dev/nvme0n1 /dev/ttyS0
brw-rw---- 1 root disk    259, 0  nvme0n1   b = block, major/minor 259,0
crw-rw---- 1 root dialout   4, 64 ttyS0     c = character

Network interfaces are not character devices; in Unix/Linux networking is a class of its own. Block size today is mostly 512 B (emulated) or 4 KiB. The classification is not airtight: tapes and clocks do not fit neatly.

I/O Hardware

Device Controllers

Mechanical partplatter, head, motorController (electronics)Registersstatus, commandBuffersector / trackECC + DMAerror correctionSystem busPCIe / SATA / USBconnectorserial bit stream:preambledata: 512 B = 4096 bits (or 4 KiB)ECC
  • An I/O unit = a mechanical/device part + an electronic controller (a chip or card, linked to the device by a connector).
  • The preamble carries a sync pattern and the cylinder/sector number. The controller turns the bit stream into a block, corrects errors with ECC, keeps the block in its own buffer, then copies it to memory.
  • At boot and on hot-plug the OS enumerates the buses (PCIe config space, USB descriptors), reads each device ID and loads the matching driver: plug and play.
  • The OS talks to the controller, not the device; standard interfaces such as SATA and NVMe let one driver run devices from many vendors.
  • Modern disks use 4 KiB physical sectors (Advanced Format); drives that expose 512 B sectors are called 512e.

I/O Hardware

Device Registers: Status, Command, Data

StatusBUSYRDYERRDRQ––––CPU readsCommandWRITECMDRDYREADIRQEN––––CPU writesData-out01000001CPU → deviceData-in????????device → CPUbit 7bit 0
  • A controller typically has four registers: status (busy, ready, error), command (what to do), data-in and data-out.
  • Writing a register = issuing a command; reading one = learning the status. Devices may also have large data buffers (e.g. a frame buffer, NIC rings).
  • Register layout is device-specific; only the driver knows it. Bit names here are illustrative.

I/O Hardware

Port-Mapped and Memory-Mapped I/O

(a) Separate spaces(b) Memory-mapped(c) HybridMemoryI/Oports0 … 2ⁿ−10 … 0xFFFFdevice registersMemory← address rangeone address spacebuffersMemorycontrolportse.g. VGA on old PCs
Port (x86)Device
0x20–0x218259 interrupt controller
0x40–0x438254 timer (PIT)
0x60, 0x648042 keyboard controller
0x3F8–0x3FFCOM1 serial port
0xCF8–0xCFFPCI configuration
in  al, 0x60          ; port: special instruction
mov eax, [rbx+0x14]   ; MMIO: ordinary load

Linux: inb()/outb()  ·  ioremap() + readl()/writel()
/proc/ioports  ·  /proc/iomem

Port-mapped I/O uses special IN/OUT instructions and a separate address space; with memory-mapped I/O a device register is an ordinary memory address that the address decoder routes to the device. ARM and RISC-V use only MMIO; PCIe devices expose registers as MMIO via BARs.

I/O Hardware

Memory-Mapped I/O: Pros and Cons

Memory-mapped I/O: pros

  • No special instructions; a driver can be written entirely in C (a volatile pointer).
  • Protection comes from the page table: if the register page is not mapped into user space it is unreachable; it can also be handed to a single process (VFIO, DPDK).
  • Every memory instruction works on registers: TEST, AND and the like operate on a register directly.

Cons

  • Caching: a cached status register hides the current value → the page is marked uncacheable (PAT/MTRR, ioremap).
  • Multiple buses: if memory sits on its own fast bus, I/O devices never see the address → the bridge or root complex routes by address range.
  • The compiler may reorder accesses: volatile and memory barriers are required.
(a) single busCPUMemoryI/Oall addresses on one bus(b) dual busCPUMemoryI/Ofast memory busI/O bus

Programmed I/O · Step through the code

Polling: The Handshake Step by Step

while (status & BUSY)
    ;          /* busy-wait */
command = WRITE;
data_out = 'A';
command |= CMDRDY;
/* --- controller side --- */
/* sees CMDRDY → BUSY = 1 */
/* read command, send byte */
/* CMDRDY = ERR = BUSY = 0 */
CPUdriverController1BUSY0CMDRDY0WRITE0ERRdata-out = —controller registers

The driver reads the status register over and over: BUSY = 1 because the controller has not finished the previous job. Meanwhile the CPU can do nothing else.

CPUdriverController0BUSY0CMDRDY0WRITE0ERRdata-out = —controller registers

BUSY dropped to 0: the loop exits. How long this takes depends on the device, anywhere from a few cycles to milliseconds.

CPUdriverController0BUSY0CMDRDY1WRITE0ERRdata-out = —controller registers

The WRITE bit is set in the command register: the job is a write.

CPUdriverController0BUSY0CMDRDY1WRITE0ERRdata-out = 0x41controller registers

The byte to send ('A' = 0x41) goes into the data-out register.

CPUdriverController0BUSY1CMDRDY1WRITE0ERRdata-out = 0x41controller registers

CMDRDY = 1: “command ready”. This bit tells the controller it may start.

CPUdriverController1BUSY1CMDRDY1WRITE0ERRdata-out = 0x41controller registers

The controller sees CMDRDY and sets BUSY = 1, telling the host it is now busy.

CPUdriverController1BUSY1CMDRDY1WRITE0ERRdata-out = 0x41controller registers

The controller reads the command register (WRITE), takes the byte from data-out and sends it to the device.

CPUdriverController0BUSY0CMDRDY1WRITE0ERRdata-out = 0x41controller registers

Done: CMDRDY, ERR and BUSY are cleared (ERR = 0 → success). The loop starts over for the next byte.

Programmed I/O (PIO): the CPU runs this handshake itself for every byte. It is simple, but the CPU spins while it waits; if the CPU switches to another process and misses a byte (e.g. a buffer overrun), data can be lost.

Programmed I/O · Step through the code

Programmed I/O: Printer Example

copy_from_user(buffer, p, count);
for (i = 0; i < count; i++) {
    while (*printer_status_reg != READY)
        ;   /* busy-wait */
    *printer_data_register = p[i];
}
return_to_user();
CPUPIO loopprinter

The user's “ABCD” string is copied into a kernel buffer (copy_from_user); user memory is never used directly.

CPUPIO loopprinter

i = 0: the loop spins until the status is READY.

CPUPIO loopprinterA

'A' is written to the data register; the printer prints it and goes busy again.

CPUPIO loopprinterA

i = 1: the printer is busy, so the CPU polls the status register.

CPUPIO loopprinterAB

'B' is written to the data register; the printer prints it and goes busy again.

CPUPIO loopprinterAB

i = 2: the printer is busy, so the CPU polls the status register.

CPUPIO loopprinterABC

'C' is written to the data register; the printer prints it and goes busy again.

CPUPIO loopprinterABC

i = 3: the printer is busy, so the CPU polls the status register.

CPUPIO loopprinterABCD

'D' is written to the data register; the printer prints it and goes busy again.

CPUPIO loopprinterABCD

Loop done; control returns to the process. The CPU was tied to this process the whole time: a 4-character job runs at printer speed.

Tanenbaum's printer example. At 100 characters per second the CPU waits ≈ 10 ms per character, time in which it could run millions of instructions. In embedded systems where the CPU has nothing else to do, PIO still makes sense.

Interrupts · Step by step

Interrupt Hardware: From Lines to MSI

DiskcontrollerKeyboardcontrollerPrintercontrollerInterrupt controller(I/O APIC)CPUlocal APICMemory / PCIe root complex

Every device is wired to the interrupt controller; the controller shares the CPU's single interrupt input and sets priorities.

DiskcontrollerKeyboardcontrollerPrintercontrollerInterrupt controller(I/O APIC)CPUlocal APIC① done: IRQMemory / PCIe root complex

① When the disk controller finishes, it signals on its interrupt line (IRQ).

DiskcontrollerKeyboardcontrollerPrintercontrollerInterrupt controller(I/O APIC)CPUlocal APIC① done: IRQ② interruptMemory / PCIe root complex

② If no other interrupt is in progress, or this one has higher priority, the controller interrupts the CPU; otherwise the request waits (the line stays asserted until the CPU acknowledges).

DiskcontrollerKeyboardcontrollerPrintercontrollerInterrupt controller(I/O APIC)CPUlocal APIC① done: IRQ② interrupt③ ack + vectorMemory / PCIe root complex

③ The CPU accepts the interrupt; the controller puts the vector number on the bus. The CPU uses it as an index into the interrupt vector table.

DiskcontrollerKeyboardcontrollerPrintercontrollerInterrupt controller(I/O APIC)CPUlocal APICMemory / PCIe root complexMSI: vector 0x41 written to the 0xFEE0_0000 region

Modern PCIe uses MSI/MSI-X. No wire: the device sends a memory write containing the vector to a special address (the 0xFEE0_0000 region on x86), and the local APIC turns it into an interrupt.

Old PCs had two 8259 PICs and 15 IRQ lines. Today x86 has an I/O APIC plus a local APIC per core (x2APIC); ARM has the GIC. The “controller puts a number on the address lines” story applies only to that historical PIC setup.

Interrupts

Vectors, Priority, NMI and Exceptions

VectorEventType
0Divide errorfault
1 / 3Debug / breakpointtrap
2NMI (non-maskable interrupt)interrupt
6Invalid opcodefault
13General protection faultfault
14Page faultfault
18Machine checkabort
32–255Device interrupts, IPIs, APIC timermaskable
vector 14IDT (pointed to by IDTR)0divide_error…14page_fault…32device interrupthandler
  • Interrupt vector table (x86: the IDT, 256 entries): vector → handler address. Vectors 0–31 are reserved for processor exceptions.
  • Maskable interrupts can be deferred inside a critical section (cli/sti); an NMI cannot: hardware errors, memory ECC, watchdog.
  • Priority: the APIC orders interrupts by vector class; a high-priority interrupt can preempt a low-priority handler.
  • Interrupt chaining: with more devices than vectors, each entry points to a list of handlers (shared IRQ).
  • A fault (page fault) cannot complete the instruction; after the handler the same instruction runs again. A trap (system call, int3) resumes at the next instruction.
  • MSI-X: up to 2048 vectors per function; NVMe and NICs take one vector per queue and steer each interrupt to that queue's core (/proc/interrupts, irqbalance).

Interrupts · Step by step

Interrupt Handling Step by Step

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stack(empty)mode:userinterrupts:onrunning:Awaiting: B (read)

Process A is running; B is waiting for a disk read. A disk interrupt arrives. The CPU finishes the current instruction, then accepts the interrupt.

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stackRIP, CSRFLAGS (PSW)RSP, SSmode:kernelinterrupts:maskedrunning:—waiting: B (read)

The hardware pushes the PC (RIP), PSW (RFLAGS) and stack pointer onto the kernel stack, switches to kernel mode and masks interrupts.

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stackRIP, CSRFLAGS (PSW)RSP, SSother registersmode:kernelinterrupts:maskedrunning:entry codewaiting: B (read)

The vector number indexes the IDT → handler address. The entry code (assembly) saves the registers the hardware did not; context and stack are ready for the ISR.

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stackRIP, CSRFLAGS (PSW)RSP, SSother registersmode:kernelinterrupts:maskedrunning:ISRwaiting: B (read)

The ISR reads the device status, takes the data and sends the controller an acknowledgment (EOI); new interrupts may now arrive. Long work is deferred to the bottom half.

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stackRIP, CSRFLAGS (PSW)RSP, SSother registersmode:kernelinterrupts:maskedrunning:ISRB ready

The waiting process B moves from blocked to the ready queue.

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stackRIP, CSRFLAGS (PSW)RSP, SSother registersmode:kernelinterrupts:maskedrunning:schedulerB ready

The scheduler picks the next process (here B). B's page table is loaded into the MMU (x86: CR3) and its registers are restored from the process table / stack.

A runsuserHardwaresaves PC, PSWIDT → ISRentry codeISRdevice + EOIB wakesready queueSchedulerMMU, registersiretreturn to userkernel stackB's contextmode:userinterrupts:onrunning:BB ready

iret: the saved PC and PSW are restored, the CPU returns to user mode with interrupts enabled. B resumes where it left off.

This sequence follows Tanenbaum's interrupt-handler steps. The scheduler could also pick A; the choice depends on the policy.

Interrupts · Step by step

Precise and Imprecise Interrupts

(a) Precise interrupt: PC = 316300done304done308done312done316not started320not started324not started328not started332not startedPC

Precise interrupt: the PC is saved in a known place; every instruction before the PC has fully completed, none after it has started, and the state of the one at the PC is known.

(b) Imprecise: instructions in different stages300done30460%308done31230%31680%320not started32450%328not started33220%

Imprecise interrupt: with pipelining or out-of-order execution, instructions are half done. To restart, either the hardware rolls back or the OS must save and decode a lot of internal state.

  • A pipeline holds many instructions in different stages; a superscalar CPU completes them out of order. The “PC” alone does not describe the state.
  • Modern x86, ARM and RISC-V processors provide precise interrupts: the reorder buffer (ROB) retires instructions in program order, and on an interrupt the effects of later instructions are discarded. The price is chip area and complexity.

Interrupts · Step through the code

Interrupt-Driven I/O: Printer Example

/* system call */
copy_from_user(buffer, p, count);
enable_interrupts();
while (*printer_status_reg != READY) ;
*printer_data_register = p[0];
i = 1; count = count - 1;
scheduler();      /* block process */
/* interrupt service routine */
if (count == 0) {
    unblock_user();
} else {
    *printer_data_register = p[i];
    count = count - 1;
    i = i + 1;
}
acknowledge_interrupt();
return_from_interrupt();
printerprocess: runningCPU: process

“ABC” is copied into a kernel buffer and printer interrupts are enabled.

printerAprocess: runningCPU: process

When the printer is ready, only the first character is written.

printerAprocess: blockedCPU: other process

The counters are updated and the scheduler is called: the process blocks and the CPU runs another process.

printerABprocess: blockedCPU: ISR

Interrupt 1: 'A' printed. count ≠ 0 → p[1] = 'B' is written.

printerABprocess: blockedCPU: ISR

The interrupt is acknowledged and the interrupted process resumes. The CPU did not spin while the printer worked.

printerABCprocess: blockedCPU: ISR

Interrupt 2: 'B' printed → p[2] = 'C' is written.

printerABCprocess: readyCPU: ISR

Interrupt 3: count == 0 → all characters printed, so the user process moves to the ready queue.

One interrupt per character: 3 characters → 3 interrupts. Lines 13 and 14 of the ISR update i and count after each character. The cost: saving context and running the handler on every interrupt (≈ 1–2 µs).

Interrupts

Top Half, Bottom Half and Hybrid Polling

Top half (hardirq)

The real ISR: kept short, runs with interrupts partly disabled. It acknowledges the device, buffers the data and schedules the bottom half.

Bottom half

Deferred work: in Linux a softirq, tasklet, workqueue or threaded IRQ (request_threaded_irq); in Windows the ISR + DPC.

Interrupt coalescing

NICs and NVMe report several completions in one interrupt: fewer interrupts, slightly more latency.

NAPI: interrupts + polling

Under heavy network traffic the driver disables the interrupt and polls the queue, then returns to interrupts when traffic drops. The hybrid wins.

When pure polling?

When the event rate is very high and latency is critical: DPDK/SPDK, NVMe polled queues, io_uring IOPOLL. One core is dedicated to polling.

Interrupt storm

A faulty device that interrupts nonstop locks up the system; the kernel disables the line (“nobody cared”) and rate-limits interrupts.

DMA

Direct Memory Access (DMA)

  • Byte-by-byte transfer wastes CPU time; a DMA controller moves whole blocks without the CPU. The CPU only sets up the transfer and handles the completion interrupt.
  • Registers: memory address, byte count, control (I/O port, direction: read/write, unit: byte/word, burst size). A multichannel controller handles many transfers at once.
  • A single DMA controller on the motherboard is only the historical 8237 design. Today most devices have their own DMA engine and write to memory directly over PCIe.
(a) third-party DMA (8237, on board)CPUDMAchannels 0–7DeviceMemory(b) bus masterCPUNVMe / NIC / GPUown DMA engineMemory

DMA · Step by step

DMA Transfer Step by Step

CPUDMA controlleraddress0x7A000count4096controlread → memoryDisk controllerbuffer (sector)emptyDiskMain memorybuffer @ 0x7A000system bus① program

① The CPU programs the DMA controller (address = 0x7A000, count = 4096, direction) and the disk controller (read the sector), then goes back to other work.

CPUDMA controlleraddress0x7A000count4096controlread → memoryDisk controllerbuffer (sector)ECC verifiedDiskMain memorybuffer @ 0x7A000system busdisk to buffer

The disk controller reads the sector into its own buffer and checks it with ECC. Thanks to the buffer, no data is lost while the bus is busy.

CPUDMA controlleraddress0x7A000count4096controlread → memoryDisk controllerbuffer (sector)ECC verifiedDiskMain memorybuffer @ 0x7A000system bus② address + read request

② The DMA controller requests the bus, puts the memory address on it and sends a read request to the disk controller.

CPUDMA controlleraddress0x7A000count4096controlread → memoryDisk controllerbuffer (sector)ECC verifiedDiskMain memorybuffer @ 0x7A000system bus③ data straight to memory

③ The disk controller writes the word directly to memory; the CPU is not involved on the bus.

CPUDMA controlleraddress0x7A004count4092controlread → memoryDisk controllerbuffer (sector)ECC verifiedDiskMain memorybuffer @ 0x7A000system bus④ ack; address++, count−−

④ The disk controller sends an acknowledgment; the DMA controller increments the address and decrements the count. Steps ②–④ repeat until the count reaches 0.

CPUDMA controlleraddress0x7B000count0controlread → memoryDisk controllerbuffer (sector)ECC verifiedDiskMain memorybuffer @ 0x7A000system bus⑤ count = 0 → interrupt

⑤ Count is 0: the DMA controller interrupts the CPU. By the time the OS looks, the data is already in memory.

Cycle stealing: DMA grabs the bus briefly for each word, so the CPU slows a little when it accesses memory. In burst mode the bus is held for a whole block. In fly-by mode data goes straight from device to memory; otherwise the DMA controller reads first and then writes.

DMA

Scatter-Gather and the IOMMU

Scatter-gatherdescriptor list0x7A00040960x3100040960x9C0002048physical memory (not contiguous)0x7A0000x310000x9C000one command, three pieces: no extra copyIOMMU (Intel VT-d, AMD-Vi, Arm SMMU)DeviceIOVA 0x1000Rogue deviceIOVA 0xDEAD000IOMMUtableRAM0x7A000✗ blocked
Scatter-gather

The device gets a list of (address, length) pairs and writes to noncontiguous pages in one operation (readv, network packets, NVMe PRP/SGL).

IOMMU

Translates the device-visible address (IOVA) to a physical one through a page table: protection (Thunderbolt DMA attacks), assigning devices to VMs (VFIO), high memory for 32-bit devices.

DMA · Step through the code

I/O Using DMA: Printer Example

/* system call */
copy_from_user(buffer, p, count);
set_up_DMA_controller();
scheduler();     /* block process */

/* interrupt service routine */
acknowledge_interrupt();
unblock_user();
return_from_interrupt();
DMA → printerprocess: runningCPU: process

The data is copied into a kernel buffer.

DMA → printerprocess: runningCPU: process

The DMA controller is programmed: buffer address, count = 3, target printer port.

DMA → printerAprocess: blockedCPU: other process

The process blocks; the CPU does other work. DMA feeds the characters to the printer without the CPU.

DMA → printerABCprocess: blockedCPU: other process

All characters transferred, and the CPU saw no interrupt along the way.

DMA → printerABCprocess: blockedCPU: ISR

DMA raises a single interrupt; the handler acknowledges it.

DMA → printerABCprocess: readyCPU: ISR

The user process is woken and the handler returns. One interrupt per buffer, not per character.

DMA frees the CPU from I/O, but the DMA controller is usually slower than the CPU. If the device is very slow or the CPU has nothing else to do, interrupt-driven I/O or even PIO can be simpler and good enough (Tanenbaum).

DMA · Step-by-step calculation

Cost of the Three Methods

The printer prints 1000 characters per second; we print 4096 characters. Handling an interrupt costs 2 µs, setting up DMA 5 µs.

Polling: the CPU waits on every character: busy for 4096 × 1 ms = 4.096 s.

With programmed I/O the CPU is tied to the device's speed.

Interrupt-driven: 4096 interrupts × 2 µs = 8.19 ms (0.2% of the job's time).

The waiting is gone; the price is the number of interrupts.

DMA: 5 µs setup + 1 interrupt × 2 µs = ≈ 7 µs, about 1170 times less.

But DMA steals bus cycles, and its setup does not pay off for small transfers.

Polling (PIO)4.096 sInterrupt-driven8.19 msDMA≈ 7 µsCPU time (log): 1 µs … 10 s

More than four orders of magnitude apart. With short transfers, slow DMA or an idle CPU, the DMA advantage disappears.

I/O Software

Goals of I/O Software

Device independence

Programs should not care which device they use: sort < input works the same from a file, an SSD or a USB stick.

Uniform naming

Names should not depend on the device type: /dev/nvme0n1, /mnt/usb/report.txt; mounting builds a single tree.

Error handling

Errors should be handled as close to the hardware as possible: the controller corrects with ECC, the driver retries; upper layers hear about it only if they must.

Synchronous / asynchronous

Most hardware transfers are asynchronous (interrupt-driven), yet a blocking interface makes programming much easier for the user.

Buffering

An incoming packet must be stored somewhere before we know where it goes; real-time devices such as audio also rely on buffers.

Shared / dedicated

A disk serves many users at once; a printer or scanner is dedicated to one user (which opens the door to deadlock).

I/O Software

I/O Software Layers

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwarerequestreply

Each layer has a well-defined job and an interface to the layer above. Tanenbaum counts four layers: user-level I/O software, device-independent software, device drivers and interrupt handlers. In Linux the “device-independent” part splits into the VFS, the page cache, the file system and the block layer.

I/O Software · Step by step

The Path of a read() Call

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwaren = read(fd, buf, 4096)

The process calls read(fd, buf, 4096); the libc wrapper enters the kernel with the syscall instruction.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwarefd → struct file → inode

The VFS maps the descriptor to the open file and the file to its i-node; permissions were already checked at open time.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwarenot cached: miss

The page cache is checked: the page is not there. A cache page is allocated and a read is started.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwareoffset 0 → LBA 2048

The file system maps the file offset to a disk block (LBA); the block layer builds a bio/request and hands it to a hardware queue.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwarecommand to SQ + doorbell

The NVMe driver writes the read command into the submission queue and rings the doorbell register. The process blocks (state D).

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwareDMA → cache page, MSI-X

The SSD DMAs the data into the cache page, posts an entry to the completion queue and raises an MSI-X interrupt.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwaredone: wake process

The interrupt handler processes the completion: the page is marked “up to date” and the waiting process becomes ready.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwarecopy_to_user → n = 4096

When the process runs, the VFS copies the data from the cache into the user buffer and returns 4096.

User processread(), printf(), spoolingSystem call + VFSfd → file, permissions, namingPage cachebuffering, cachingFile system + block layeroffset → LBA, blk-mq, schedulerDevice driverbuild command, write registersInterrupt handlercompletion, wake processHardwarecontroller + device (DMA)userkernelhardwaresecond read: hit

If the same data is read again, it is a page-cache hit: the device is never touched and the call takes microseconds.

I/O Software

Device Drivers

  • A driver encapsulates device-specific code: only it knows the registers, commands and quirks. Most Linux drivers live in the kernel tree (in-tree); some come from vendors.
  • Its jobs: validate parameters, turn the request into device commands (LBA, queue entry), initialize the device if needed, write the registers, wait for the interrupt (block), check for errors, return the result.
  • It must be reentrant: a new request or interrupt can arrive while it runs. With hot-plug, the device can vanish while in use.
  • A buggy driver in the kernel crashes the whole system → user-space drivers: MINIX 3, FUSE, VFIO/UIO, DPDK/SPDK; Windows UMDF. Loadable modules: modprobe, lsmod.
User programkerneldevice-independent layerNVMedriverUSB/HIDdriverNetworkdriverPCIeNVMe ctrl.SSDxHCI ctrl.keyboardNICcable

I/O Software

Device-Independent I/O Software

  • A uniform interface for all drivers: every class offers the same functions, and the kernel calls them through a table of function pointers.
  • Naming: /dev/nvme0n1 → major number (driver) + minor number (unit). Protection: permissions on the device file.
  • A device-independent block size, buffering, error reporting, and allocating and releasing dedicated devices all live in this layer.
struct file_operations nvme_fops = {
  .open           = nvme_open,
  .release        = nvme_release,
  .read_iter      = nvme_read,
  .unlocked_ioctl = nvme_ioctl,
  .poll           = nvme_poll,
};
(a) no interfaceOperating systemSATANetworkUSB(b) uniform interfaceOperating systemopen/read/write/ioctlSATANetworkUSB

I/O Software · Step by step

Buffering

(a) no bufferuser processkernelmodem / deviceevery character:interrupt + wakeup

Unbuffered: every character raises an interrupt and wakes the process. Lots of context switches; inefficient.

(b) buffer in user spaceuser processkernelmodem / devicen charactersbuffer page must be pinned in memory

User buffer: the interrupt handler writes characters straight into the process's buffer. What if that page is swapped out? It must be pinned, and if many processes pin pages, memory runs short.

(c) kernel buffer + copyuser processkernelmodem / deviceuserkernelone copy when full

Kernel buffer: characters collect in the kernel and are copied to the user in one go when the buffer fills. But where does data go that arrives during the copy?

(d) double bufferuser processkernelmodem / deviceuserbuffer 1buffer 2fillingcopying

Double buffering: one buffer fills while the other is copied, and the roles alternate. The general form is the circular buffer.

(e) one packet over the network: five copiesuser (sender)kernel buffernetwork controller bufferuser (receiver)kernel buffernetwork controller buffer123: wire45zero copy: sendfile, splice, MSG_ZEROCOPY, io_uring

A packet may be copied 5 times on its way across the network; each copy costs memory bandwidth and latency. Hence zero-copy techniques.

Three reasons to buffer (Silberschatz): a speed mismatch between producer and consumer, a transfer-size mismatch (reassembling packets), and copy semantics (the user may change its buffer once write returns).

I/O Software · Step by step

Circular Buffer

01234567head = 0 tail = 0used slots: 0 / 8data (producer: ISR / DMA)tail: next slot to writehead: next slot to readstart: buffer emptyempty: head == tail full: (tail + 1) % 8 == head

Head and tail point to the same slot: the buffer is empty.

a0b1c234567head = 0 tail = 3used slots: 3 / 8data (producer: ISR / DMA)tail: next slot to writehead: next slot to readproducer wrote 3 bytesempty: head == tail full: (tail + 1) % 8 == head

The interrupt handler writes 3 incoming bytes at the tail, and the tail advances. The consumer has not read yet.

01c234567head = 2 tail = 3used slots: 1 / 8data (producer: ISR / DMA)tail: next slot to writehead: next slot to readconsumer read 2 bytesempty: head == tail full: (tail + 1) % 8 == head

The process reads 2 bytes and the head advances. Slots that were read can be reused.

01c2d3e4f5g6h7head = 2 tail = 0used slots: 6 / 8data (producer: ISR / DMA)tail: next slot to writehead: next slot to readwrap: tail 7 → 0empty: head == tail full: (tail + 1) % 8 == head

The producer writes 5 more bytes; at the end the tail wraps around (mod 8): tail = 0.

i01c2d3e4f5g6h7head = 2 tail = 1used slots: 7 / 8data (producer: ISR / DMA)tail: next slot to writehead: next slot to readone more byte: buffer fullempty: head == tail full: (tail + 1) % 8 == head

After 'i', tail = 1 and (tail + 1) mod 8 == head: the buffer is full. Leaving one slot empty tells full from empty. When full, the producer waits or drops the data (lost network packets).

Circular buffers are everywhere: keyboard and serial drivers, NIC receive/transmit rings, NVMe submission/completion queues, io_uring. With a single producer and a single consumer no lock is needed, only memory barriers.

I/O Software · Step through the code

Circular Buffer: Code Trace

int put(char c) {          /* producer */
  int s = (tail + 1) % N;
  if (s == head) return -1; /* full */
  buf[tail] = c;
  tail = s;
  return 0;
}
int get(char *c) {         /* consumer */
  if (head == tail) return -1; /* empty */
  *c = buf[head];
  head = (head + 1) % N;
  return 0;
}
N = 5 slots; holds at most N − 1 = 4 bytes·0·1·2·3·4headtail

Buffer empty: head = tail = 0. The interrupt handler writes with put, the process reads with get; each changes only its own index.

N = 5 slots; holds at most N − 1 = 4 bytes'a'0·1·2·3·4headtail

put('a'): buf[0] = 'a', tail = 1.

N = 5 slots; holds at most N − 1 = 4 bytes'a'0'b'1·2·3·4headtail

put('b'): buf[1] = 'b', tail = 2.

N = 5 slots; holds at most N − 1 = 4 bytes'a'0'b'1'c'2·3·4headtail

put('c'): buf[2] = 'c', tail = 3.

N = 5 slots; holds at most N − 1 = 4 bytes·0'b'1'c'2·3·4headtail

get: head ≠ tail, so the buffer is not empty. buf[0] = 'a' is read, head = 1. Slot 0 can be reused.

N = 5 slots; holds at most N − 1 = 4 bytes·0'b'1'c'2'd'3·4headtail

put('d'): buf[3] = 'd', tail = 4.

N = 5 slots; holds at most N − 1 = 4 bytes·0'b'1'c'2'd'3'e'4headtail

put('e'): buf[4] = 'e', tail = 0. The tail reached the end and wrapped around (mod 5).

N = 5 slots; holds at most N − 1 = 4 bytes·0'b'1'c'2'd'3'e'4headtail(tail + 1) mod 5 = 1 = head → full

put('f'): s = (0 + 1) mod 5 = 1 = head. One slot always stays empty; otherwise head == tail would mean both empty and full. The byte is rejected (−1).

N = 5 slots; holds at most N − 1 = 4 bytes·0·1'c'2'd'3'e'4headtail

get: head ≠ tail, so the buffer is not empty. buf[1] = 'b' is read, head = 2. Slot 1 can be reused.

I/O Software

Caching, Spooling and Error Handling

Cache ≠ buffer

A buffer temporarily holds the only copy of data; a cache is a fast copy of data that lives elsewhere. The Linux page cache combines both; free RAM goes to the cache (free -h: buff/cache).

Spooling

A queue for a dedicated device: users drop files into a spool directory and a single daemon (CUPS, /var/spool/cups) prints them in turn. No process ever holds the printer.

Error handling

A programming error (bad fd) → an error code (EBADF, EINVAL). A real I/O error → the driver retries; if it persists, EIO and a log entry (dmesg).

Allocation and protection

A dedicated device is claimed with open; if busy, the request is queued or EBUSY is returned. I/O instructions are privileged; users do I/O only through system calls.

User-space I/O software: library routines such as printf/scanf format data and buffer it (stdio line/block buffering), then call the write/read system calls.

I/O Interfaces

Blocking, Nonblocking and Asynchronous I/O

Blockingread() returns only when data arrivesrunsblocked: waiting for I/Ocontinues with dataNonblockingreturns at once: what is ready, or EAGAINrunsEAGAINother workEAGAINother workdataresumeAsynchronousrequest sent, result reported laterrunsrequestkeeps runningcompletedresumedevice data ready
InterfaceModelNote
select / pollreadiness notificationscans the whole fd list on every call: O(n)
epoll (Linux), kqueue (BSD)readiness notificationinterest list in the kernel; only ready fds return
io_uring, Windows IOCPcompletion notificationtrue asynchronous I/O, disks included

I/O Interfaces · Step through the code

Event Loop: epoll

int ep = epoll_create1(0);
for (k = 0; k < nsock; k++)   /* O_NONBLOCK */
    epoll_ctl(ep, EPOLL_CTL_ADD, s[k], &ev);
for (;;) {
  int n = epoll_wait(ep, evs, 64, -1);
  for (int i = 0; i < n; i++) {
    int fd = evs[i].data.fd;
    ssize_t r = read(fd, buf, sizeof buf);
    if (r < 0 && errno == EAGAIN) continue;
    handle(fd, buf, r);
  }
}
interest list (in kernel)ready list(empty)

An epoll instance is created: a kernel object holding the interest and ready lists.

interest list (in kernel)fd 5fd 6fd 7ready list(empty)

Three sockets (5, 6, 7) are added to the interest list in nonblocking mode. This happens once; unlike select, they are not passed again every round.

interest list (in kernel)fd 5fd 6fd 7ready list(empty)

epoll_wait blocks the process while no socket is ready; the CPU does not spin.

interest list (in kernel)fd 5fd 6fd 7ready listfd 7

A packet arrives on socket 7: NIC interrupt → network stack → fd 7 joins the ready list and the process wakes. n = 1.

interest list (in kernel)fd 5fd 6fd 7ready listfd 7

Only the ready fd is handled: 512 bytes are read. Cost grows with the number of ready fds, not the total.

interest list (in kernel)fd 5fd 6fd 7ready list(empty)

Data handled; when the next read returns EAGAIN there is nothing left. The loop goes back to epoll_wait.

One thread manages tens of thousands of connections (nginx, Node.js, Redis). epoll says “you may read”, but read still does the work; disk files always look “ready”, so true asynchrony for disks comes with io_uring.

I/O Interfaces · Step by step

io_uring: Shared Rings

Application (user space)Kernel: file system, block layer, NVMeSQ (submission ring)CQ (completion ring)shared memory (mmap)

io_uring (Linux 5.1, 2019): the application and the kernel see two rings in shared memory; requests and results are never copied.

Application (user space)Kernel: file system, block layer, NVMeSQ (submission ring)CQ (completion ring)READREADWRITEshared memory (mmap)SQEs written, tail advances

The application writes three submission entries (SQEs): two reads and a write. Each carries an opcode, fd, buffer and offset.

Application (user space)Kernel: file system, block layer, NVMeSQ (submission ring)CQ (completion ring)shared memory (mmap)io_uring_enter (or the SQPOLL kernel thread)

A single io_uring_enter call submits all three requests. In SQPOLL mode a kernel thread polls the ring: no system call at all.

Application (user space)Kernel: file system, block layer, NVMeSQ (submission ring)CQ (completion ring)40964096512shared memory (mmap)CQEs posted

As devices finish, the kernel posts completion entries (CQEs) to the ring; the result field holds bytes transferred or −errno.

Application (user space)Kernel: file system, block layer, NVMeSQ (submission ring)CQ (completion ring)shared memory (mmap)read with no system call

The application reads CQEs straight from memory and advances the head. Batched submission + batched reaping → very few context switches.

This is a completion model (like Windows IOCP): you are told “your job is done”. Hundreds of operation types exist, including network, file, accept and timers. Because of security bugs it is disabled in some environments (e.g. Android, some containers).

Clocks and Other Devices

Clock Hardware

Crystal oscillator1.193182 MHzpulseCounter−1 per pulseat 0InterruptIRQ 0 / vectorHolding register = 1193reload (square-wave mode)CPUclock handlermodes: one-shot · square-wave (periodic)
  • Example (8254 PIT): for a 1 kHz interrupt, counter = 1,193,182 / 1000 ≈ 1193 → actual frequency 1000.15 Hz.
  • Modern x86: the TSC (time-stamp counter), HPET and the local APIC timer (TSC-deadline mode); ARM: the generic timer. A battery-backed RTC keeps the date while powered off.
  • Clock software: time of day, quantum expiry, CPU usage accounting, alarm/timers, watchdogs, profiling.
  • Keep the time of day in a 64-bit counter: a 32-bit time_t overflows on 19 January 2038.

Clocks and Other Devices · Step by step

Clock Software: Ticks and Tickless Kernels

(a) periodic tick: HZ = 250 → 4 ms0481216202428323640time (ms)workCPU idle (wants a deep C-state)timer event6 useless wakeups while idle: each pulls the CPU out of deep sleephrtimer: ns-resolution timers kept sorted in a red-black tree

Periodic tick: the timer interrupts every 1/HZ seconds (Linux HZ = 100, 250 or 1000 → 10, 4, 1 ms). The handler advances time, charges the quantum and fires expired timers.

(b) tickless (NO_HZ): one-shot to the next event0481216202428323640time (ms)workCPU idle (wants a deep C-state)timer eventno tick while idle: the timer is programmed once, for 30 mshrtimer: ns-resolution timers kept sorted in a red-black tree

Tickless kernel (NO_HZ_IDLE, 2007): when the CPU is idle the tick stops and the clock is set one-shot to the next event. nohz_full removes the tick even while a single process runs on a core (HPC, real time).

(c) timer list: current time 4200Now4200Next signalin 3 ticks+3t = 4203+4t = 4207+6t = 4213+2t = 4215+1t = 4216delta list: each tick decrements only the head counter

Tanenbaum's example: pending signals at 4203, 4207, 4213, 4215 and 4216. The list stores only the differences; when the head reaches 0, its event fires.

Tickless operation saves energy and CPU time on laptops and VMs. For short (µs) waits, soft timers and polling are used together.

Clocks and Other Devices · Step through the code

Timer List: Code Trace

void add(Timer *z, int t) {  /* t: ticks */
  Timer **p = &head;
  while (*p && (*p)->delta <= t) {
    t -= (*p)->delta;
    p = &(*p)->next;
  }
  z->delta = t; z->next = *p;
  if (*p) (*p)->delta -= t;
  *p = z;
}
void tick(void) {
  if (head) head->delta--;
  while (head && head->delta == 0)
    head = run(head);   /* remove */
}
now = 4200in box: delta (after previous event)A34203B44207C64213D24215E14216*pbottom row: absolute event time

Pending events at 4203, 4207, 4213, 4215, 4216; the list stores only the deltas: 3, 4, 6, 2, 1. New timer X fires 10 ticks from now (4210): add(X, 10).

now = 4200in box: delta (after previous event)A34203B44207C64213D24215E14216*pbottom row: absolute event time

A's delta 3 ≤ t: X goes after it. t = 7, and *p moves to the next node.

now = 4200in box: delta (after previous event)A34203B44207C64213D24215E14216*pbottom row: absolute event time

B's delta 4 ≤ t: X goes after it. t = 3, and *p moves to the next node.

now = 4200in box: delta (after previous event)A34203B44207C64213D24215E14216*pbottom row: absolute event time

C's delta 6 > t = 3: X must go before C. The loop ends.

now = 4200in box: delta (after previous event)A34203B44207X34210C34213D24215E14216*pbottom row: absolute event time

X is spliced in with delta = 3; C's delta becomes 6 − 3 = 3. C's absolute time (4213) is unchanged; only one node was updated.

now = 4201in box: delta (after previous event)A24203B44207X34210C34213D24215E14216bottom row: absolute event time

Tick: only the head delta drops → 2.

now = 4202in box: delta (after previous event)A14203B44207X34210C34213D24215E14216bottom row: absolute event time

Tick: only the head delta drops → 1.

now = 4203in box: delta (after previous event)B44207X34210C34213D24215E14216bottom row: absolute event time

4203: the head delta hit 0; A runs and leaves the list. The work per tick is just decrementing the head counter: O(1), however long the list.

Clocks and Other Devices

Terminals, Input Devices and Power Management

Terminals and GUI input
  • tty + line discipline: canonical mode edits the line (backspace, deliver on Enter); raw mode delivers every key at once (stty -icanon, editors).
  • Pseudoterminal (pty): terminal emulators, ssh; screen control through ANSI escape sequences.
  • The keyboard sends scan codes (press/release); the driver maps them to characters with a keymap. USB HID devices are polled by the host at 125–1000 Hz.
  • Linux: input subsystem → /dev/input/event* (evdev) → libinput → Wayland compositor → window.
Power management
  • ACPI system states: S0 working (S0ix modern standby), S3 suspend to RAM, S4 hibernate (to disk), S5 off.
  • CPU: C-states (idle sleep depth), P-states (frequency/voltage; Linux schedutil).
  • Devices: D0–D3 states, runtime power management; NVMe APST, PCIe ASPM, Wi-Fi power saving, screen dimming.
  • The OS puts devices to sleep and wakes them in the right order, and chooses the wakeup sources (keyboard, network, clock). Battery life depends mostly on idle time.

Disks

Magnetic Disk Geometry

top viewsectortrackside viewcylinderarmspindleheadzoned recording (ZBR)32 sectors/track24 sectors/track16 sectors/trackvirtual geometry: flat LBA 0 … N−1
  • Platters spin at 5400–7200 rpm (servers: 10–15k), with one head per surface. Tracks at the same radius form a cylinder. Outer zones hold more sectors per track.
  • The OS sees LBAs, not CHS (cylinder, head, sector); the controller inside the disk maps them to the real geometry, spare sectors and cache. The old CHS limit: 1024 × 255 × 63 × 512 B ≈ 8.4 GB.

Disks · Step-by-step calculation

Access Time: An Example

Access time = seek + rotational latency + transfer. Disk: 7200 rpm, 8 ms average seek, 200 MB/s media rate; 4 KiB block.

All three components count, transfer time included.

One revolution = 60,000 / 7200 = 8.33 ms; average rotational latency = half a revolution = 4.17 ms.

On average the target sector is half a turn away.

Transfer = 4096 B ÷ 200 MB/s = 0.020 ms. Total = 8 + 4.17 + 0.02 = 12.19 ms.

Mechanical delay dominates: almost all the time goes to seek and rotation. The CPU could run millions of instructions meanwhile; the OS does not wait, it runs another process.

Random 4 KiB: 1000 / 12.19 ≈ 82 IOPS → 82 × 4 KiB ≈ 0.34 MB/s. Sequential 1 MiB: 8 + 4.17 + 5.24 = 17.41 ms → 60 MB/s.

The same disk is 180 times faster or slower depending on the access pattern, which is why locality and scheduling matter. (The math assumes a constant media rate and a fresh seek + rotation per chunk; a long sequential read seeks once and approaches the media rate.)

seek 8 msrotation 4.17 mstransfer 0.02 ms ← (0.2%)total 12.19 ms

Compare: an NVMe SSD does a random 4 KiB read in ≈ 50–100 µs and over a million IOPS with parallel queues.

Disks

Disk Formatting and Cylinder Skew

sector format (low-level format)preambledata 512 B / 4 KiBECCskew: don't miss sector 0 when switching trackstrack 00123456789101112131415track 11314150123456789101112track 21011121314150123456789
  • Low-level formatting is done at the factory: tracks, sectors, preamble, ECC and spare sectors. Formatted capacity is less than raw capacity. High-level formatting writes an empty file system to a partition (boot block, free-space management, root directory).
  • Skew math (Tanenbaum): 10,000 rpm → 6 ms per revolution; 300 sectors/track → 20 µs per sector. With an 800 µs track switch, skew = 800 / 20 = 40 sectors.
  • Interleaving once spaced consecutive sectors apart for slow controllers; today the controller buffers the whole track, so it is unnecessary.
  • Today's disks: 4 KiB physical sectors (Advanced Format); 512e disks emulate 512 B, and unaligned writes need read-modify-write.

Disks

Optical Disks (Briefly)

CD (1982)DVD (1996)Blu-ray (2006)
Capacity650–700 MB4.7 / 8.5 / 9.4 / 17 GB25 / 50 GB; BDXL 100–128 GB
Laser780 nm (infrared)650 nm (red)405 nm (blue-violet)
Smallest pit0.83 µm0.40 µm0.15 µm
Track pitch1.6 µm0.74 µm0.32 µm
  • A single spiral track; a transition between pit and land is a 1 bit. On pressed discs the pits are molded into polycarbonate, not burned by a laser. On CD-R/DVD-R the laser heats an organic dye layer to change its reflectivity.
  • On a CD-ROM each byte becomes a 14-bit channel symbol via EFM (a modulation code, not ECC); error correction uses CIRC. 98 frames × 24 data bytes = 2352 B; in mode 1, 2048 B of that is user data and the rest is sync, header and extra ECC/EDC.
  • A DVD's two layers are semitransparent; the laser reads the lower one by refocusing. Optical media now serve mainly archiving and distribution; their density trails magnetic and flash media.
optical: one spiral trackmagnetic: concentric

Disk Scheduling

Disk Arm Scheduling

  • The goal is to cut arm movement (seek time) and raise throughput; the driver keeps a list of (cylinder, request time). Metric: total head movement (cylinders).
  • Today the LBA → physical mapping lives inside the disk, and the disk itself reorders 32 requests by rotational position with NCQ. The OS schedules approximately, by “LBA proximity”.
seek: arm changes cylinderrotationred: the head's cylinder
AlgorithmIdeaProblem
FCFSArrival orderFair, but the arm wanders a lot
SSTFRequest closest to the head (greedy)Requests at the edges can starve
SCAN (elevator)Sweep to the end in one direction, then reverseEdges and middle do not wait equally
C-SCANSweep one way, jump from the end back to the startThe return trip is wasted
LOOK / C-LOOKSCAN/C-SCAN, but only up to the last requestThe forms used in practice

The examples use Silberschatz's queue: head at 53; requests 98, 183, 37, 122, 14, 124, 65, 67. Cylinders 0–199.

Disk Scheduling · Step by step

FCFS

First come, first served: requests in arrival order.

0143753656798122124183199total: 0 cylindersservedwaiting

The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.

0143753656798122124183199total: 45 cylindersservedwaiting

53 → 98: 45 cylinders · total 45

0143753656798122124183199total: 130 cylindersservedwaiting

98 → 183: 85 cylinders · total 130

0143753656798122124183199total: 276 cylindersservedwaiting

183 → 37: 146 cylinders · total 276

0143753656798122124183199total: 361 cylindersservedwaiting

37 → 122: 85 cylinders · total 361

0143753656798122124183199total: 469 cylindersservedwaiting

122 → 14: 108 cylinders · total 469

0143753656798122124183199total: 579 cylindersservedwaiting

14 → 124: 110 cylinders · total 579

0143753656798122124183199total: 638 cylindersservedwaiting

124 → 65: 59 cylinders · total 638

0143753656798122124183199total: 640 cylindersservedwaiting

65 → 67: 2 cylinders · total 640

640 cylinders in all. Fair and starvation-free, but swings like 122 → 14 → 124 are very costly.

Disk Scheduling · Step by step

SSTF

Shortest seek time first: at each step, the request closest to the head.

0143753656798122124183199total: 0 cylindersservedwaiting

The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.

0143753656798122124183199total: 12 cylindersservedwaiting

53 → 65: 12 cylinders · total 12

0143753656798122124183199total: 14 cylindersservedwaiting

65 → 67: 2 cylinders · total 14

0143753656798122124183199total: 44 cylindersservedwaiting

67 → 37: 30 cylinders · total 44

0143753656798122124183199total: 67 cylindersservedwaiting

37 → 14: 23 cylinders · total 67

0143753656798122124183199total: 151 cylindersservedwaiting

14 → 98: 84 cylinders · total 151

0143753656798122124183199total: 175 cylindersservedwaiting

98 → 122: 24 cylinders · total 175

0143753656798122124183199total: 177 cylindersservedwaiting

122 → 124: 2 cylinders · total 177

0143753656798122124183199total: 236 cylindersservedwaiting

124 → 183: 59 cylinders · total 236

236 in all (37% of FCFS). It is greedy and not optimal: 53 → 37 → 14 → 65 … is shorter (208). Under heavy load, requests in the middle can starve those at the edges.

Disk Scheduling · Step through the code

SSTF: Code Trace

int sstf(int head, int q[], int n) {
  int total = 0;
  for (int k = 0; k < n; k++) {
    int best = -1;
    for (int i = 0; i < n; i++)
      if (!done[i] && (best < 0 ||
        abs(q[i]-head) < abs(q[best]-head)))
        best = i;          /* closest */
    total += abs(q[best] - head);
    head = q[best];
    done[best] = 1;
  }
  return total;
}
05010015019950total = 0 cylinders

The head is at cylinder 50; queue (arrival order): 82, 170, 43, 140, 24, 16, 190. Each round, the inner loop looks at every waiting request.

0501001501995043total = 7 cylinders

k = 0: head at 50, nearest waiting request 43 (distance 7; runner-up 24, distance 26). total = 7.

050100150199504324total = 26 cylinders

k = 1: head at 43, nearest waiting request 24 (distance 19; runner-up 16, distance 27). total = 26.

05010015019950432416total = 34 cylinders

k = 2: head at 24, nearest waiting request 16 (distance 8; runner-up 82, distance 58). total = 34.

0501001501995043241682total = 100 cylinders

k = 3: head at 16, nearest waiting request 82 (distance 66; runner-up 140, distance 124). total = 100. The low requests are done, so the arm travels a long way back.

0501001501995043241682140total = 158 cylinders

k = 4: head at 82, nearest waiting request 140 (distance 58; runner-up 170, distance 88). total = 158.

0501001501995043241682140170total = 188 cylinders

k = 5: head at 140, nearest waiting request 170 (distance 30; runner-up 190, distance 50). total = 188.

0501001501995043241682140170190total = 208 cylinders

k = 6: head at 170, nearest waiting request 190 (distance 20). total = 208.

0501001501995043241682140170190total = 208 cylinders

208 cylinders in all (FCFS: 642 on this queue). The greedy choice takes the shortest move each time; if requests keep arriving in the middle, those at the edges may wait.

Disk Scheduling · Step by step

SCAN (Elevator)

SCAN: the head sweeps in one direction (here toward 0) to the end, then reverses, like an elevator.

0143753656798122124183199total: 0 cylindersservedwaiting

The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.

0143753656798122124183199total: 16 cylindersservedwaiting

53 → 37: 16 cylinders · total 16

0143753656798122124183199total: 39 cylindersservedwaiting

37 → 14: 23 cylinders · total 39

0143753656798122124183199total: 53 cylindersservedwaiting

14 → 0: 14 cylinders · total 53 (no request; the arm goes to the edge)

0143753656798122124183199total: 118 cylindersservedwaiting

0 → 65: 65 cylinders · total 118

0143753656798122124183199total: 120 cylindersservedwaiting

65 → 67: 2 cylinders · total 120

0143753656798122124183199total: 151 cylindersservedwaiting

67 → 98: 31 cylinders · total 151

0143753656798122124183199total: 175 cylindersservedwaiting

98 → 122: 24 cylinders · total 175

0143753656798122124183199total: 177 cylindersservedwaiting

122 → 124: 2 cylinders · total 177

0143753656798122124183199total: 236 cylindersservedwaiting

124 → 183: 59 cylinders · total 236

236 in all. Getting stuck in one region is SSTF's greedy flaw; SCAN prevents starvation. Going to the edge and back (14 → 0 → 14) wastes 28 cylinders.

Disk Scheduling · Step by step

C-SCAN

Circular SCAN: serve in the upward direction only; at the end, return to the start without serving.

0143753656798122124183199total: 0 cylindersservedwaiting

The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.

0143753656798122124183199total: 12 cylindersservedwaiting

53 → 65: 12 cylinders · total 12

0143753656798122124183199total: 14 cylindersservedwaiting

65 → 67: 2 cylinders · total 14

0143753656798122124183199total: 45 cylindersservedwaiting

67 → 98: 31 cylinders · total 45

0143753656798122124183199total: 69 cylindersservedwaiting

98 → 122: 24 cylinders · total 69

0143753656798122124183199total: 71 cylindersservedwaiting

122 → 124: 2 cylinders · total 71

0143753656798122124183199total: 130 cylindersservedwaiting

124 → 183: 59 cylinders · total 130

0143753656798122124183199total: 146 cylindersservedwaiting

183 → 199: 16 cylinders · total 146 (no request; the arm goes to the edge)

0143753656798122124183199total: 345 cylindersservedwaiting

199 → 0: return jump (199 cylinders), no requests served. Counted here · total 345

0143753656798122124183199total: 359 cylindersservedwaiting

0 → 14: 14 cylinders · total 359

0143753656798122124183199total: 382 cylindersservedwaiting

14 → 37: 23 cylinders · total 382

382 in all (183 without the return). The disk is treated as a ring; waiting times are more uniform than with SCAN.

Disk Scheduling · Step by step

LOOK and C-LOOK

0143753656798122124183199total: 0 cylindersservedwaiting

LOOK (downward): like SCAN, but it does not go to the edge; it reverses at the last request in that direction.

0143753656798122124183199total: 39 cylindersservedwaiting

53 → 37 → 14: the low requests are done, so no trip to 0. Total 39.

0143753656798122124183199total: 208 cylindersservedwaiting

14 → 65 … 183: direction reverses. Total 208 (28 less than SCAN). In practice, the “elevator” behavior is usually this LOOK.

0143753656798122124183199total: 130 cylindersservedwaiting

C-LOOK (upward): 53 → 65 … 183, the last request going up. Total 130.

0143753656798122124183199total: 299 cylindersservedwaiting

183 → 14: jumps to the lowest waiting request (169 cylinders), skipping 199 and 0.

0143753656798122124183199total: 322 cylindersservedwaiting

14 → 37: total 322 (153 without the jump). A one-way sweep evens out waiting times.

Linux's old deadline and CFQ schedulers and today's mq-deadline dispatch in LBA order, which behaves much like C-LOOK.

Disk Scheduling · Step through the code

LOOK: Code Trace

/* s[] sorted; go up first, then down */
int look(int head, int s[], int n) {
  int i = 0, total = 0;
  while (i < n && s[i] < head) i++;
  for (int j = i; j < n; j++) {
    total += s[j] - head; head = s[j];
  }
  for (int j = i - 1; j >= 0; j--) {
    total += head - s[j]; head = s[j];
  }
  return total;
}
05010015019950total = 0 cylinders

The same queue, sorted: s = 16, 24, 43, 82, 140, 170, 190. The head is at 50 and moving up.

05010015019950total = 0 cylinders

i is the index of the first request above 50: s[3] = 82. The 16, 24, 43 below are taken on the way back.

0501001501995082total = 32 cylinders

j = 3: 82 (+32), total = 32.

0501001501995082140total = 90 cylinders

j = 4: 140 (+58), total = 90.

0501001501995082140170total = 120 cylinders

j = 5: 170 (+30), total = 120.

0501001501995082140170190total = 140 cylinders

j = 6: 190 (+20), total = 140. The last request going up: the arm turns back without going to 199 (unlike SCAN).

050100150199508214017019043total = 287 cylinders

Direction reversed, j = 2: 190 → 43 (+147), total = 287. The longest move is on the way back.

05010015019950821401701904324total = 306 cylinders

j = 1: 24 (+19), total = 306.

0501001501995082140170190432416total = 314 cylinders

j = 0: 16 (+8), total = 314.

0501001501995082140170190432416total = 314 cylinders

314 in all; SSTF gets 208 on the same queue. Had the head been moving down, LOOK would also give 34 + 174 = 208: the direction decides the result, but no request starves.

Disk Scheduling

Comparing the Algorithms

FCFS640SSTF236SCAN236C-SCAN382 (183 without jump)LOOK208C-LOOK322 (153 without jump)
Tanenbaum's example: head 11; 1, 36, 16, 34, 9, 12Movement
FCFS: 10 + 35 + 20 + 18 + 25 + 3111
SSTF: 12, 9, 16, 1, 34, 36 → 1 + 3 + 7 + 15 + 33 + 261
Elevator (up): 12, 16, 34, 36, 9, 1 → 1 + 4 + 18 + 2 + 27 + 860

On the charted queue, SCAN starting upward would give 331: the direction changes the result. Tanenbaum's elevator does not go to the edge, so it is really LOOK.

Disk Scheduling · Step by step

SPTF: Counting Rotational Latency Too

rotationhead: track 10A (track 11)B (track 30)7200 rpm: 1 turn = 8.33 ms; the disk turns 43.2° per msA: 1 ms seek, sector reaches the head after 30°B: 3 ms seek, sector reaches the head after 150°0246810ms

The head is on track 10 with two requests waiting. SSTF, which looks only at track distance, sees A as far closer: one track away. But time is set by seek + rotation, not seek alone.

rotationhead: track 10A (track 11)B (track 30)7200 rpm: 1 turn = 8.33 ms; the disk turns 43.2° per msA: 1 ms seek, sector reaches the head after 30°B: 3 ms seek, sector reaches the head after 150°0246810msSSTF → A9.03 msseekrotational wait

SSTF picks A: during the 1 ms seek the disk turns 43.2°, so the sector that was 30° away is 13.2° past the head when the seek ends. Almost a full turn of waiting: 1 + 8.03 = 9.03 ms.

rotationhead: track 10A (track 11)B (track 30)7200 rpm: 1 turn = 8.33 ms; the disk turns 43.2° per msA: 1 ms seek, sector reaches the head after 30°B: 3 ms seek, sector reaches the head after 150°0246810msSSTF → A9.03 msSPTF → B3.47 msseekrotational wait

SPTF picks B: during the 3 ms seek the disk turns 129.6°, leaving 20.4° to B: 3 + 0.47 = 3.47 ms. The far track finished 2.6 times sooner than the near one.

SPTF (shortest positioning time first) minimizes positioning time, but the OS does not know sector angles, so it is implemented inside the disk via NCQ. Against starvation, F-SCAN freezes the queue when a sweep starts; new requests wait for the next sweep.

Disk Scheduling · Step by step

Controller Cache and Bad Sectors

Controller cache

The disk keeps a track cache in its own DRAM (≈ 256 MB): it also reads unrequested blocks passing under the head. This is separate from the OS cache.

Write cache and NCQ

Writes may sit in the cache: durability needs FLUSH/FUA. NCQ orders 32 requests by rotational position.

(a) track: sector 7 bad, two spares01234567 ✗89101112131415Y1Y2

A sector that fails in manufacturing or in use: written data cannot be read back. The controller tracks the factory list and those found in use (SMART: Reallocated Sectors).

(b) bad sector remapped to a spare01234567 ✗891011121314157Y2controller redirects

Method 1: the bad sector is remapped to a spare. Simple, but a sequential read through sector 7 may cost an extra revolution.

(c) sectors shifted, bad one skipped0123456✗789101112131415Y2sectors 8 … 15 shift by one, one spare used; sequential reads stay intact

Method 2: the sectors are shifted; the whole track must be rewritten, but order is preserved. The OS can also keep a bad-block list (badblocks, the file system).

SSD and NVMe

SSD: NAND Flash

SSD controllerFTLECC (LDPC)DRAM cachediediediediediediediediediediediedie4–8 channels × dies: parallelismblock (≈ 10–40 MiB) = erase unitpage (≈ 16 KiB) = read/write unit
CellBits/cellTypical P/E cyclesUse
SLC1≈ 50–100kcache region, industrial
MLC2≈ 3–10kolder enterprise
TLC3≈ 1–3kmost consumer and server SSDs
QLC4≈ 0.1–1khigh capacity, read-heavy
  • No moving parts: no seek and no rotational latency, so random access ≈ sequential access.
  • No in-place update: erase, then write; each cell survives a limited number of erases. 3D NAND has 200–300+ layers.
  • The price-per-GB gap has narrowed; HDDs are still cheaper for archives and capacity.

SSD and NVMe · Step by step

FTL, Garbage Collection, TRIM, Wear Leveling

mapping table (FTL)LBA 0B0.s0LBA 1B0.s1LBA 2B0.s2LBA 3—block B0 (erases: 0)s0valid: LBA 0s1valid: LBA 1s2valid: LBA 2s3emptyblock B1 (erases: 0)s0emptys1emptys2emptys3emptyLBAs 0, 1, 2 writtenwrites go only to erased (free) pages; erase is per block

Three logical blocks are written; the FTL places them in B0's free pages and records them in the table.

mapping table (FTL)LBA 0B0.s0LBA 1B0.s3LBA 2B0.s2LBA 3—block B0 (erases: 0)s0valid: LBA 0s1invalid (old LBA 1)s2valid: LBA 2s3valid: LBA 1block B1 (erases: 0)s0emptys1emptys2emptys3emptyLBA 1 updated: not in placewrites go only to erased (free) pages; erase is per block

LBA 1 is rewritten: the old page cannot be overwritten. The new data goes to a free page and the old one becomes invalid (out-of-place update).

mapping table (FTL)LBA 0B0.s0LBA 1B0.s3LBA 2—LBA 3—block B0 (erases: 0)s0valid: LBA 0s1invalid (old LBA 1)s2invalid (old LBA 2)s3valid: LBA 1block B1 (erases: 0)s0emptys1emptys2emptys3emptyTRIM: LBA 2 no longer usedwrites go only to erased (free) pages; erase is per block

When a file is deleted the file system sends TRIM (NVMe: Deallocate), so the SSD learns the page is no longer needed. Without TRIM, garbage collection would copy this data for nothing.

mapping table (FTL)LBA 0B0.s0LBA 1B0.s3LBA 2—LBA 3B1.s0block B0 (erases: 0)s0valid: LBA 0s1invalid (old LBA 1)s2invalid (old LBA 2)s3valid: LBA 1block B1 (erases: 0)s0valid: LBA 3s1emptys2emptys3emptyB0 full, LBA 3 goes to B1writes go only to erased (free) pages; erase is per block

B0 has no free pages left; new writes go to B1. Half of B0 is invalid.

mapping table (FTL)LBA 0B1.s1LBA 1B1.s2LBA 2—LBA 3B1.s0block B0 (erases: 1)s0emptys1emptys2emptys3emptyblock B1 (erases: 0)s0valid: LBA 3s1valid: LBA 0s2valid: LBA 1s3emptygarbage collection: 2 valid pages moved, B0 erasedwrites go only to erased (free) pages; erase is per block

Garbage collection: B0's valid pages (LBA 0, 1) are copied to B1 and B0 is erased. The 2 extra page writes are write amplification.

mapping table (FTL)LBA 0B1.s1LBA 1B1.s2LBA 2—LBA 3B1.s0block B0 (erases: 1)s0emptys1emptys2emptys3emptyblock B1 (erases: 0)s0valid: LBA 3s1valid: LBA 0s2valid: LBA 1s3emptywear leveling: rarely erased blocks get used toowrites go only to erased (free) pages; erase is per block

Wear leveling: the FTL evens out erase counts, moving long-unchanged (cold) data too so that all blocks age equally. Spare area (over-provisioning) makes garbage collection easier.

Windows “Optimize” does not defragment an SSD; it sends TRIM. Defragmenting an SSD only adds writes and wear. Aligning partitions to a 1 MiB boundary keeps logical blocks from straddling flash pages.

SSD and NVMe

NVMe and Multiple Queues

Core 0SQCQCore 1SQCQCore 2SQCQCore 3SQCQNVMe controller (PCIe x4)AHCI (SATA)one queue · 32 commands · lockNVMe65,535 queues × 65,536 commandsa queue per core: lock-free submission
  • NVMe is a protocol designed for flash on PCIe; M.2 is a form factor (an M.2 card can also be SATA). U.2/U.3 and E1.S/E3.S are cabled/slotted server forms.
  • Queue capacity: 64K queues × 64K commands per queue. Every OS has shipped a driver since 2013; the incompatibility with AHCI is at the protocol level only.
  • Latency ≈ 10–100 µs; PCIe 5.0 x4 drives reach ≈ 14 GB/s and > 2 million IOPS. NVMe-oF runs it over a network; ZNS adds zoned namespaces.

SSD and NVMe

Linux blk-mq and I/O Schedulers

CPU 0sw queueCPU 1sw queueCPU 2sw queueCPU 3sw queuescheduler (optional)hq 0hq 1hq 2hq 3hardware submission queuesNVMe
SchedulerIdeaBest for
noneNo sorting, dispatch directlyNVMe (default), fast SSDs
mq-deadlineSorted by LBA + read/write deadlinesSATA HDD/SSD (default on most distros)
bfqPer-process budgets, fair sharingDesktops, interactivity on slow devices
kyberThrottling against read/write latency targetsFast multiqueue devices
$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber bfq
$ echo bfq | sudo tee /sys/block/sda/queue/scheduler

Per-CPU software queues avoid contention on a single queue lock at millions of IOPS; Linux 5.0 removed the old single-queue block layer (noop, deadline, CFQ). An SSD has no seeks, so the scheduler's job is not ordering but protecting fairness and latency.

RAID and Stable Storage

RAID: Combining Disks

  • RAID (Patterson, Gibson, Katz, 1988) presents many disks to the OS as one disk; striping gives parallelism, mirroring and parity give redundancy. The opposite is SLED: a single large expensive disk.
  • Today the acronym reads Redundant Array of Independent Disks. SCSI disks were expensive; now SAS expanders attach hundreds of disks, and the standard levels are 0–6.
  • Hardware RAID cards or software RAID (Linux mdadm, ZFS RAID-Z, Btrfs, Windows Storage Spaces). RAID is not a backup: a deleted file is deleted on every disk.
RAID 0: stripingdisk 0A0A2A4A6disk 1A1A3A5A7RAID 1: mirroringdisk 0A0A1A2A3disk 1A0A1A2A3parallel, no redundancyeach block twice
4 × 4 TB disksUsableSurvivesSmall random-write cost
RAID 0 (striping)16 TB0 disks1 I/O
RAID 1 (4-way mirror)4 TB3 disks4 I/Os (one per copy)
RAID 10 (mirror + stripe)8 TB1 (2 if in different pairs)2 I/Os
RAID 5 (distributed parity)12 TB1 disk4 I/Os (2 reads + 2 writes)
RAID 6 (dual parity)8 TB2 disks6 I/Os (3 reads + 3 writes)

RAID and Stable Storage · Step by step

RAID Levels

RAID 0: striping, no redundancydisk 0s0s4s8s12disk 1s1s5s9s13disk 2s2s6s10s14disk 3s3s7s11s15

RAID 0: consecutive stripes go to different disks; n times the bandwidth for large requests. Lose one disk and you lose everything.

RAID 1: mirroringdisk 0s0s1s2s3disk 1s0s1s2s3

RAID 1: every write happens twice; a read comes from either disk (load is spread). Half the capacity is redundancy.

RAID 10: striping over mirrored pairsdisk 0s0s2s4s6disk 1s0s2s4s6disk 2s1s3s5s7disk 3s1s3s5s7

RAID 10: mirror first, then stripe. Common for databases: fast writes and fast rebuilds.

RAID 2/3: bit/byte striping + a parity diskdisk 0b0b4b8b12disk 1b1b5b9b13disk 2b2b6b10b14disk 3b3b7b11b15disk 4P0P1P2P3

RAID 2 uses words + a Hamming code, RAID 3 bytes + a single parity disk; the disks must spin in sync. Neither is used today (disks do their own ECC).

RAID 4: block striping + a dedicated parity diskdisk 0s0s4s8s12disk 1s1s5s9s13disk 2s2s6s10s14disk 3s3s7s11s15disk 4P0P1P2P3

RAID 4: block stripes + a dedicated parity disk. Every small write updates the parity disk, which becomes the bottleneck.

RAID 5: parity spread across the disksdisk 0s0s4s8s12disk 1s1s5s9P3disk 2s2s6P2s13disk 3s3P1s10s14disk 4P0s7s11s15

RAID 5: parity blocks rotate across the disks, removing the bottleneck. Capacity of n − 1 disks; survives one disk failure.

RAID 6: P and Q parity spread across the disksdisk 0Q0s3s6s9disk 1s0s4s7P3disk 2s1s5P2Q3disk 3s2P1Q2s10disk 4P0Q1s8s11

RAID 6: P (XOR) + Q (Reed–Solomon, GF(2⁸)) parity; survives two disk failures. Preferred for large disks, where a rebuild takes hours.

RAID and Stable Storage · Step by step

Parity and XOR: An Example

D0 (disk 0)1011 0010D1 (disk 1)0101 1100D2 (disk 2)1110 0111P (disk 3)0000 0000

Three data blocks in a stripe (an 8-bit example here). The parity block is not computed yet.

D0 (disk 0)1011 0010D1 (disk 1)0101 1100D2 (disk 2)1110 0111P (disk 3)0000 1001P = D0 ⊕ D1 ⊕ D2 = 0000 1001

Parity is a bitwise XOR: each bit position is completed so the number of 1s is even.

D0 (disk 0)1011 0010D1 (disk 1)✗ failedD2 (disk 2)1110 0111P (disk 3)0000 1001D1 = D0 ⊕ D2 ⊕ P = ?

Disk 1 failed. Since x ⊕ x = 0, the lost block is the XOR of the others.

D0 (disk 0)1011 0010D1 (disk 1)0101 1100D2 (disk 2)1110 0111P (disk 3)0000 1001D1 = 1011 0010 ⊕ 1110 0111 ⊕ 0000 1001 = 0101 1100

The recovered value 0101 1100 equals the original D1. Every stripe is rewritten this way onto a new disk (the rebuild).

D0 (disk 0)1011 0010D1 (disk 1)0000 1111D2 (disk 2)1110 0111P (disk 3)0101 1010P' = P ⊕ D1 ⊕ D1' = 0101 1010

Small write: D1' = 0000 1111. Read the old D1 and P, compute P', write D1' and P': 4 I/Os (read-modify-write).

Rebuild risk: a consumer disk has an unrecoverable read error (URE) about once per 1014 bits. Rebuilding 12 TB gives 12×1012×8 / 1014 ≈ 0.96 expected errors, a ≈ 62% chance of at least one: a RAID 5 rebuild will likely fail, so large arrays use RAID 6.

RAID and Stable Storage · Step through the code

RAID 5 Small Write and Recovery: Code Trace

void small_write(int d, uint8_t nv) {
  uint8_t ov = rd(d);      /* I/O 1 */
  uint8_t p = rd(P);       /* I/O 2 */
  p = p ^ ov ^ nv;
  wr(d, nv);               /* I/O 3 */
  wr(P, p);                /* I/O 4 */
}
uint8_t rebuild(int bad) {
  uint8_t x = 0;
  for (int d = 0; d <= P; d++)
    if (d != bad)
      x ^= rd(d);          /* incl. P */
  return x;
}
disk 0D01011 0010disk 1D10110 1001disk 2D21100 0101disk 3P0001 1110disk I/Os: 0

In the stripe P = D0 ⊕ D1 ⊕ D2 = 0001 1110. D1 is to be overwritten with 0011 0011: small_write(1, ...).

disk 0D01011 0010disk 1D10110 1001disk 2D21100 0101disk 3P0001 1110disk I/Os: 1

The old D1 is read. The other data disks (D0, D2) are not: only 1 of the 3 data disks in the stripe changes.

disk 0D01011 0010disk 1D10110 1001disk 2D21100 0101disk 3P0001 1110disk I/Os: 2

The old parity is read: p = 0001 1110.

disk 0D01011 0010disk 1D10110 1001disk 2D21100 0101disk 3P0001 11100001 1110 ⊕ 0110 1001 ⊕ 0011 0011 = 0100 0100disk I/Os: 2

p ⊕ ov removes the old D1 from the parity (x ⊕ x = 0); ⊕ nv adds the new value. p = 0100 0100.

disk 0D01011 0010disk 1D10011 0011disk 2D21100 0101disk 3P0001 1110disk I/Os: 3

The new D1 is written.

disk 0D01011 0010disk 1D10011 0011disk 2D21100 0101disk 3P0100 0100disk I/Os: 4

The new P is written: 4 I/Os (2 reads + 2 writes). A crash between these two writes leaves the stripe inconsistent (the “write hole”).

disk 0D01011 0010disk 1D10011 0011disk 2D2????disk 3P0100 0100disk I/Os: 0

Disk 2 failed: D2 cannot be read. rebuild(2) computes it as the XOR of the rest.

disk 0D01011 0010disk 1D10011 0011disk 2D2????disk 3P0100 0100x = 1011 0010disk I/Os: 1

d = 0: x ^= D0 → x = 1011 0010.

disk 0D01011 0010disk 1D10011 0011disk 2D2????disk 3P0100 0100x = 1000 0001disk I/Os: 2

d = 1: x ^= D1 → x = 1000 0001.

disk 0D01011 0010disk 1D10011 0011disk 2D2????disk 3P0100 0100x = 1100 0101disk I/Os: 3

d = 3: x ^= P → x = 1100 0101. (d = 2 is skipped.)

disk 0D01011 0010disk 1D10011 0011disk 2D2????disk 3P0100 0100D2 = 1100 0101disk I/Os: 3

x = 1100 0101, the same as the lost D2. Each stripe is rebuilt onto a new disk with 3 reads; the array is unprotected meanwhile.

RAID and Stable Storage · Calculation

RAID Performance: An Example

Single disk (7200 rpm, 8 ms seek, 200 MB/s): random 4 KiB = 12.19 ms → R ≈ 82 IOPS (0.34 MB/s); a sequential 10 MiB chunk = 8 + 4.17 + 52.43 = 64.6 ms → S ≈ 162 MB/s. Array: N = 4 disks. Assumes a constant media rate and a fresh seek + rotation per chunk; a long sequential read seeks once and S approaches the media rate (200 MB/s).
LevelCapacitySequential read/writeRandom readRandom writeWrite latency
RAID 0N = 4N·S ≈ 649 MB/sN·R ≈ 328N·R ≈ 328D
RAID 1 (two-way mirror)N/2 = 2(N/2)·S ≈ 325 MB/sN·R ≈ 328(N/2)·R ≈ 164D (slowest copy)
RAID 4N − 1 = 3(N−1)·S ≈ 487 MB/s(N−1)·R ≈ 246R/2 ≈ 412D
RAID 5N − 1 = 3(N−1)·S ≈ 487 MB/sN·R ≈ 328(N/4)·R ≈ 822D
  • Sequential reads on a mirror get only (N/2)·S: the two copies split the blocks, but each disk still rotates under the blocks it skips.
  • RAID 4: every small write needs 1 read + 1 write on the parity disk; the array drops to half the speed of one disk, and adding disks does not help.
  • RAID 5: the same 4 I/Os spread over all disks, so N·R/4 grows with the disk count. A full-stripe write reads no old data: (N−1)·S.
random 4 KiB writes (IOPS), N = 4RAID 0328RAID 1164RAID 441RAID 582

RAID and Stable Storage · Step by step

Stable Storage

  • Stable storage: a write either completes correctly or leaves the old data in place. Assumptions: the same block failing on both disks at once is negligible; bad data is detected by ECC; if the CPU crashes, the write in progress stops.
  • Stable write: write disk 1, read it back, verify (retry n times, use a spare sector if needed); then do the same on disk 2. Stable read: if disk 1 cannot be read with valid ECC, read disk 2.
disk 1olddisk 2oldrecovery:do nothing (old stays)

(a) Crash before writing disk 1: both copies are old. The write never happened.

disk 1ECC errordisk 2oldrecovery:copy disk 2 → disk 1

(b) Crash while writing disk 1: disk 1 is bad (ECC). The old value is restored from disk 2.

disk 1newdisk 2oldrecovery:copy disk 1 → disk 2

(c) Disk 1 written, crash before disk 2: the new value is completed.

disk 1newdisk 2ECC errorrecovery:copy disk 1 → disk 2

(d) Crash while writing disk 2: disk 2 is bad, so the new value is copied from disk 1.

disk 1newdisk 2newrecovery:do nothing

(e) Both copies new: the write is complete. In every case the result is either entirely old or entirely new.

RAID protects against failing sectors, not against a crash mid-write. Today the same idea appears as journaling, write barriers/FUA and the power-loss capacitors in enterprise SSDs.

Summary

Summary

TopicKey ideaToday
Hardware accessController registers; port- or memory-mappedMMIO through PCIe BARs
TransferPolling → interrupts → DMAMSI-X, bus-master DMA, IOMMU
Software layersUser / independent / driver / interruptVFS + page cache + blk-mq
InterfaceBlocking, nonblocking, asynchronousepoll, io_uring
BufferingSpeed/size mismatch, double and circular buffersNIC/NVMe rings, zero copy
HDDSeek + rotation + transfer; schedulingNCQ, mq-deadline, BFQ
SSDNo seeks; FTL, garbage collection, TRIMNVMe, multiqueue, none
ReliabilityRAID 0/1/5/6/10, stable storagemdadm, ZFS, journaling

The essence of I/O design: decouple slow, diverse devices from the fast CPU (buffers, interrupts, DMA), hide their differences behind a uniform interface (drivers, VFS), and choose the order wisely (scheduling, batching).

Summary

Review Questions

  1. With port-mapped and with memory-mapped I/O, which instruction accesses a device register? How is the caching problem solved for MMIO?
  2. 4096 characters go to a 1000-character-per-second printer using interrupt-driven I/O, at 2 µs per interrupt. What is the CPU cost?
  3. What is cycle stealing? When does DMA not pay off?
  4. How long does a random 4 KiB read take on a disk with 7200 rpm, 8 ms average seek and a 200 MB/s media rate?
  5. Head at 53, queue 98, 183, 37, 122, 14, 124, 65, 67 (0–199). What is the total movement for SSTF and C-LOOK (upward)?
  6. Head at 11, requests 1, 36, 16, 34, 9, 12. How many cylinders do FCFS, SSTF and the elevator (upward) move?
  7. In RAID 5, D0 = 1011 0010, D2 = 1110 0111, P = 0000 1001. What is the failed D1? If one disk gives 100 IOPS, how many IOPS do 6-disk RAID 5 and RAID 4 give for random small writes?
  8. Why does an SSD need TRIM? Which I/O scheduler is the Linux default for NVMe devices, and why?
  9. What is the fundamental difference in model between epoll and io_uring?
Answers
  1. Port: IN/OUT; MMIO: an ordinary MOV. Register pages are marked uncacheable (PAT/MTRR, ioremap).
  2. 4096 × 2 µs = 8.19 ms (4.096 s with polling).
  3. DMA briefly takes the bus for each word, so the CPU waits on memory accesses. It does not pay off for small transfers, a slow DMA controller, or a CPU with nothing else to do.
  4. 8 + 4.17 + 0.02 = 12.19 ms (≈ 82 IOPS).
  5. SSTF 236; C-LOOK 322 (153 without the jump).
  6. 111, 61, 60.
  7. D0 ⊕ D2 ⊕ P = 0101 1100. RAID 5: 6 × 100 / 4 = 150; RAID 4: the parity disk is the bottleneck, 100 / 2 = 50.
  8. The SSD learns which pages are no longer valid; garbage collection skips them and write amplification drops. none: no seeks, multiqueue is already parallel, and sorting costs CPU.
  9. epoll reports readiness (you then call read); io_uring reports completion and submits requests in batches through shared rings.

Summary

References

  • A. Silberschatz, P. B. Galvin, G. Gagne, Operating System Concepts, 10th ed., Wiley, 2018 — Chapter 11: Mass-Storage Structure; Chapter 12: I/O Systems.
  • A. S. Tanenbaum, H. Bos, Modern Operating Systems, 4th ed. (2014) / 5th ed. (2023), Pearson — Chapter 5: Input/Output.
  • R. H. Arpaci-Dusseau, A. C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces (OSTEP), ostep.org — “I/O Devices”, “Hard Disk Drives”, “RAID”, “Flash-based SSDs”.
  • D. A. Patterson, G. Gibson, R. H. Katz, “A Case for Redundant Arrays of Inexpensive Disks (RAID)”, SIGMOD 1988.
  • NVM Express Base Specification 2.x; PCI-SIG PCIe 5.0/6.0; USB4 Specification; UEFI Specification (GPT).
  • Linux kernel documentation: “Multi-Queue Block IO Queueing Mechanism (blk-mq)”, “NO_HZ: Reducing Scheduling-Clock Ticks”, io_uring (J. Axboe, “Efficient IO with io_uring”), “Linux Device Drivers” (Corbet, Rubini, Kroah-Hartman, 3rd ed.).

The End

Chapter 11: Input/Output
Previous: Chapter 10: File Systems · Next: Chapter 12: Multiprocessor Systems
1 / 1