Chapter 11: Input/Output
Introduction
Learning Outcomes
Explain controllers, device registers, PCIe/USB/NVMe buses, and port-mapped versus memory-mapped I/O.
Compare polling, interrupt-driven I/O and DMA; state the role of MSI-X and the IOMMU.
Trace a read() through the interrupt handler, the driver, the device-independent layer and the user library.
Distinguish blocking, nonblocking and asynchronous I/O; tell epoll from io_uring.
Compute access time; find the total head movement of FCFS, SSTF, SCAN, C-SCAN, LOOK and C-LOOK.
Explain the FTL, wear leveling, TRIM and blk-mq; RAID 0/1/5/6/10 and recovery with XOR parity.
Introduction
I/O System Overview
- The OS issues commands to devices, reads their status, catches interrupts and handles errors.
- It gives applications a uniform, layered, device-independent interface.
- Most kernel code is device drivers (about 60–70% of the Linux source lives in
drivers/).
Introduction
Buses: PCIe, USB4, SATA
- Bus: a set of wires carrying data, address and control signals between components. Old PCs used shared parallel buses (ISA, PCI); today we use point-to-point serial links (PCIe, USB, SATA).
- A PCIe lane is full duplex; a link bundles x1, x4, x8 or x16 lanes. Bandwidth is counted per direction.
| Link | Raw rate | Effective data rate | Note |
|---|---|---|---|
| PCIe 4.0 | 16 GT/s / lane | ≈ 1.97 GB/s / lane; x4 ≈ 7.9 GB/s | 128b/130b encoding; NVMe SSDs |
| PCIe 5.0 | 32 GT/s / lane | ≈ 3.94 GB/s / lane; x16 ≈ 63 GB/s | GPUs, 14 GB/s-class SSDs |
| PCIe 6.0 | 64 GT/s (PAM4) | ≈ 7.6 GB/s / lane; x16 ≈ 121 GB/s | FLIT + FEC; growing in servers |
| USB 2.0 / 3.2 | 480 Mb/s / 5–20 Gb/s | ≤ 60 MB/s / ≤ 2.5 GB/s | Host-polled protocol |
| USB4 v1 / v2 | 40 / 80 Gb/s | PCIe and DisplayPort tunneling | Thunderbolt 4/5 compatible |
| SATA III | 6 Gb/s | ≈ 600 MB/s (8b/10b) | AHCI: one queue, 32 commands |
| Ethernet | 1 / 10 / 100 / 400 Gb/s | 125 MB/s … 50 GB/s | Data center: 400–800 GbE |
Math: a PCIe 4.0 lane gives 16 GT/s × 128/130 ÷ 8 = 1.97 GB/s. FireWire, 56K modems and 10/100 Mb/s Ethernet are history. The Pentium-era northbridge is gone: the memory controller moved into the CPU (AMD 2003, Intel 2008); fast devices attach to the CPU over PCIe, slow ones to the chipset.
Introduction
Device Data Rates
A keyboard and PCIe 5.0 x16 differ by a factor of ≈ 6 × 109: the same OS must handle a few keystrokes per second and millions of NVMe requests per second. Values are typical upper limits; real rates depend on signal quality, protocol overhead and the device.
Introduction
Device Classes: Block, Character, Network
| Class | Example | Property | Typical interface |
|---|---|---|---|
| Block | HDD, SSD, USB stick, SD card | Fixed-size, addressable blocks (512 B / 4 KiB); each block is read or written independently | read, write, lseek; file system, mmap |
| Character | Keyboard, mouse, serial port, printer, audio | A byte stream; not addressable, no seek | read/write, ioctl; /dev/ttyS0 |
| Network | Ethernet, Wi-Fi | Packet based; neither block nor character | Socket API (send, recv); no file in /dev |
| Other | Clocks, timers, GPUs | Neither stream nor block: they raise interrupts or are memory-mapped | timerfd, DRM/KMS, mmap |
$ ls -l /dev/nvme0n1 /dev/ttyS0 brw-rw---- 1 root disk 259, 0 nvme0n1 b = block, major/minor 259,0 crw-rw---- 1 root dialout 4, 64 ttyS0 c = character
Network interfaces are not character devices; in Unix/Linux networking is a class of its own. Block size today is mostly 512 B (emulated) or 4 KiB. The classification is not airtight: tapes and clocks do not fit neatly.
I/O Hardware
Device Controllers
- An I/O unit = a mechanical/device part + an electronic controller (a chip or card, linked to the device by a connector).
- The preamble carries a sync pattern and the cylinder/sector number. The controller turns the bit stream into a block, corrects errors with ECC, keeps the block in its own buffer, then copies it to memory.
- At boot and on hot-plug the OS enumerates the buses (PCIe config space, USB descriptors), reads each device ID and loads the matching driver: plug and play.
- The OS talks to the controller, not the device; standard interfaces such as SATA and NVMe let one driver run devices from many vendors.
- Modern disks use 4 KiB physical sectors (Advanced Format); drives that expose 512 B sectors are called 512e.
I/O Hardware
Device Registers: Status, Command, Data
- A controller typically has four registers: status (busy, ready, error), command (what to do), data-in and data-out.
- Writing a register = issuing a command; reading one = learning the status. Devices may also have large data buffers (e.g. a frame buffer, NIC rings).
- Register layout is device-specific; only the driver knows it. Bit names here are illustrative.
I/O Hardware
Port-Mapped and Memory-Mapped I/O
| Port (x86) | Device |
|---|---|
| 0x20–0x21 | 8259 interrupt controller |
| 0x40–0x43 | 8254 timer (PIT) |
| 0x60, 0x64 | 8042 keyboard controller |
| 0x3F8–0x3FF | COM1 serial port |
| 0xCF8–0xCFF | PCI configuration |
in al, 0x60 ; port: special instruction mov eax, [rbx+0x14] ; MMIO: ordinary load Linux: inb()/outb() · ioremap() + readl()/writel() /proc/ioports · /proc/iomem
Port-mapped I/O uses special IN/OUT instructions and a separate address space; with memory-mapped I/O a device register is an ordinary memory address that the address decoder routes to the device. ARM and RISC-V use only MMIO; PCIe devices expose registers as MMIO via BARs.
I/O Hardware
Memory-Mapped I/O: Pros and Cons
Memory-mapped I/O: pros
- No special instructions; a driver can be written entirely in C (a
volatilepointer). - Protection comes from the page table: if the register page is not mapped into user space it is unreachable; it can also be handed to a single process (VFIO, DPDK).
- Every memory instruction works on registers:
TEST,ANDand the like operate on a register directly.
Cons
- Caching: a cached status register hides the current value → the page is marked uncacheable (PAT/MTRR,
ioremap). - Multiple buses: if memory sits on its own fast bus, I/O devices never see the address → the bridge or root complex routes by address range.
- The compiler may reorder accesses:
volatileand memory barriers are required.
Programmed I/O · Step through the code
Polling: The Handshake Step by Step
while (status & BUSY)
; /* busy-wait */
command = WRITE;
data_out = 'A';
command |= CMDRDY;
/* --- controller side --- */
/* sees CMDRDY → BUSY = 1 */
/* read command, send byte */
/* CMDRDY = ERR = BUSY = 0 */The driver reads the status register over and over: BUSY = 1 because the controller has not finished the previous job. Meanwhile the CPU can do nothing else.
BUSY dropped to 0: the loop exits. How long this takes depends on the device, anywhere from a few cycles to milliseconds.
The WRITE bit is set in the command register: the job is a write.
The byte to send ('A' = 0x41) goes into the data-out register.
CMDRDY = 1: “command ready”. This bit tells the controller it may start.
The controller sees CMDRDY and sets BUSY = 1, telling the host it is now busy.
The controller reads the command register (WRITE), takes the byte from data-out and sends it to the device.
Done: CMDRDY, ERR and BUSY are cleared (ERR = 0 → success). The loop starts over for the next byte.
Programmed I/O (PIO): the CPU runs this handshake itself for every byte. It is simple, but the CPU spins while it waits; if the CPU switches to another process and misses a byte (e.g. a buffer overrun), data can be lost.
Programmed I/O · Step through the code
Programmed I/O: Printer Example
copy_from_user(buffer, p, count); for (i = 0; i < count; i++) { while (*printer_status_reg != READY) ; /* busy-wait */ *printer_data_register = p[i]; } return_to_user();
The user's “ABCD” string is copied into a kernel buffer (copy_from_user); user memory is never used directly.
i = 0: the loop spins until the status is READY.
'A' is written to the data register; the printer prints it and goes busy again.
i = 1: the printer is busy, so the CPU polls the status register.
'B' is written to the data register; the printer prints it and goes busy again.
i = 2: the printer is busy, so the CPU polls the status register.
'C' is written to the data register; the printer prints it and goes busy again.
i = 3: the printer is busy, so the CPU polls the status register.
'D' is written to the data register; the printer prints it and goes busy again.
Loop done; control returns to the process. The CPU was tied to this process the whole time: a 4-character job runs at printer speed.
Tanenbaum's printer example. At 100 characters per second the CPU waits ≈ 10 ms per character, time in which it could run millions of instructions. In embedded systems where the CPU has nothing else to do, PIO still makes sense.
Interrupts · Step by step
Interrupt Hardware: From Lines to MSI
Every device is wired to the interrupt controller; the controller shares the CPU's single interrupt input and sets priorities.
① When the disk controller finishes, it signals on its interrupt line (IRQ).
② If no other interrupt is in progress, or this one has higher priority, the controller interrupts the CPU; otherwise the request waits (the line stays asserted until the CPU acknowledges).
③ The CPU accepts the interrupt; the controller puts the vector number on the bus. The CPU uses it as an index into the interrupt vector table.
Modern PCIe uses MSI/MSI-X. No wire: the device sends a memory write containing the vector to a special address (the 0xFEE0_0000 region on x86), and the local APIC turns it into an interrupt.
Old PCs had two 8259 PICs and 15 IRQ lines. Today x86 has an I/O APIC plus a local APIC per core (x2APIC); ARM has the GIC. The “controller puts a number on the address lines” story applies only to that historical PIC setup.
Interrupts
Vectors, Priority, NMI and Exceptions
| Vector | Event | Type |
|---|---|---|
| 0 | Divide error | fault |
| 1 / 3 | Debug / breakpoint | trap |
| 2 | NMI (non-maskable interrupt) | interrupt |
| 6 | Invalid opcode | fault |
| 13 | General protection fault | fault |
| 14 | Page fault | fault |
| 18 | Machine check | abort |
| 32–255 | Device interrupts, IPIs, APIC timer | maskable |
- Interrupt vector table (x86: the IDT, 256 entries): vector → handler address. Vectors 0–31 are reserved for processor exceptions.
- Maskable interrupts can be deferred inside a critical section (
cli/sti); an NMI cannot: hardware errors, memory ECC, watchdog. - Priority: the APIC orders interrupts by vector class; a high-priority interrupt can preempt a low-priority handler.
- Interrupt chaining: with more devices than vectors, each entry points to a list of handlers (shared IRQ).
- A fault (page fault) cannot complete the instruction; after the handler the same instruction runs again. A trap (system call,
int3) resumes at the next instruction. - MSI-X: up to 2048 vectors per function; NVMe and NICs take one vector per queue and steer each interrupt to that queue's core (
/proc/interrupts,irqbalance).
Interrupts · Step by step
Interrupt Handling Step by Step
Process A is running; B is waiting for a disk read. A disk interrupt arrives. The CPU finishes the current instruction, then accepts the interrupt.
The hardware pushes the PC (RIP), PSW (RFLAGS) and stack pointer onto the kernel stack, switches to kernel mode and masks interrupts.
The vector number indexes the IDT → handler address. The entry code (assembly) saves the registers the hardware did not; context and stack are ready for the ISR.
The ISR reads the device status, takes the data and sends the controller an acknowledgment (EOI); new interrupts may now arrive. Long work is deferred to the bottom half.
The waiting process B moves from blocked to the ready queue.
The scheduler picks the next process (here B). B's page table is loaded into the MMU (x86: CR3) and its registers are restored from the process table / stack.
iret: the saved PC and PSW are restored, the CPU returns to user mode with interrupts enabled. B resumes where it left off.
This sequence follows Tanenbaum's interrupt-handler steps. The scheduler could also pick A; the choice depends on the policy.
Interrupts · Step by step
Precise and Imprecise Interrupts
Precise interrupt: the PC is saved in a known place; every instruction before the PC has fully completed, none after it has started, and the state of the one at the PC is known.
Imprecise interrupt: with pipelining or out-of-order execution, instructions are half done. To restart, either the hardware rolls back or the OS must save and decode a lot of internal state.
- A pipeline holds many instructions in different stages; a superscalar CPU completes them out of order. The “PC” alone does not describe the state.
- Modern x86, ARM and RISC-V processors provide precise interrupts: the reorder buffer (ROB) retires instructions in program order, and on an interrupt the effects of later instructions are discarded. The price is chip area and complexity.
Interrupts · Step through the code
Interrupt-Driven I/O: Printer Example
/* system call */ copy_from_user(buffer, p, count); enable_interrupts(); while (*printer_status_reg != READY) ; *printer_data_register = p[0]; i = 1; count = count - 1; scheduler(); /* block process */ /* interrupt service routine */ if (count == 0) { unblock_user(); } else { *printer_data_register = p[i]; count = count - 1; i = i + 1; } acknowledge_interrupt(); return_from_interrupt();
“ABC” is copied into a kernel buffer and printer interrupts are enabled.
When the printer is ready, only the first character is written.
The counters are updated and the scheduler is called: the process blocks and the CPU runs another process.
Interrupt 1: 'A' printed. count ≠ 0 → p[1] = 'B' is written.
The interrupt is acknowledged and the interrupted process resumes. The CPU did not spin while the printer worked.
Interrupt 2: 'B' printed → p[2] = 'C' is written.
Interrupt 3: count == 0 → all characters printed, so the user process moves to the ready queue.
One interrupt per character: 3 characters → 3 interrupts. Lines 13 and 14 of the ISR update i and count after each character. The cost: saving context and running the handler on every interrupt (≈ 1–2 µs).
Interrupts
Top Half, Bottom Half and Hybrid Polling
The real ISR: kept short, runs with interrupts partly disabled. It acknowledges the device, buffers the data and schedules the bottom half.
Deferred work: in Linux a softirq, tasklet, workqueue or threaded IRQ (request_threaded_irq); in Windows the ISR + DPC.
NICs and NVMe report several completions in one interrupt: fewer interrupts, slightly more latency.
Under heavy network traffic the driver disables the interrupt and polls the queue, then returns to interrupts when traffic drops. The hybrid wins.
When the event rate is very high and latency is critical: DPDK/SPDK, NVMe polled queues, io_uring IOPOLL. One core is dedicated to polling.
A faulty device that interrupts nonstop locks up the system; the kernel disables the line (“nobody cared”) and rate-limits interrupts.
DMA
Direct Memory Access (DMA)
- Byte-by-byte transfer wastes CPU time; a DMA controller moves whole blocks without the CPU. The CPU only sets up the transfer and handles the completion interrupt.
- Registers: memory address, byte count, control (I/O port, direction: read/write, unit: byte/word, burst size). A multichannel controller handles many transfers at once.
- A single DMA controller on the motherboard is only the historical 8237 design. Today most devices have their own DMA engine and write to memory directly over PCIe.
DMA · Step by step
DMA Transfer Step by Step
① The CPU programs the DMA controller (address = 0x7A000, count = 4096, direction) and the disk controller (read the sector), then goes back to other work.
The disk controller reads the sector into its own buffer and checks it with ECC. Thanks to the buffer, no data is lost while the bus is busy.
② The DMA controller requests the bus, puts the memory address on it and sends a read request to the disk controller.
③ The disk controller writes the word directly to memory; the CPU is not involved on the bus.
④ The disk controller sends an acknowledgment; the DMA controller increments the address and decrements the count. Steps ②–④ repeat until the count reaches 0.
⑤ Count is 0: the DMA controller interrupts the CPU. By the time the OS looks, the data is already in memory.
Cycle stealing: DMA grabs the bus briefly for each word, so the CPU slows a little when it accesses memory. In burst mode the bus is held for a whole block. In fly-by mode data goes straight from device to memory; otherwise the DMA controller reads first and then writes.
DMA
Scatter-Gather and the IOMMU
The device gets a list of (address, length) pairs and writes to noncontiguous pages in one operation (readv, network packets, NVMe PRP/SGL).
Translates the device-visible address (IOVA) to a physical one through a page table: protection (Thunderbolt DMA attacks), assigning devices to VMs (VFIO), high memory for 32-bit devices.
DMA · Step through the code
I/O Using DMA: Printer Example
/* system call */ copy_from_user(buffer, p, count); set_up_DMA_controller(); scheduler(); /* block process */ /* interrupt service routine */ acknowledge_interrupt(); unblock_user(); return_from_interrupt();
The data is copied into a kernel buffer.
The DMA controller is programmed: buffer address, count = 3, target printer port.
The process blocks; the CPU does other work. DMA feeds the characters to the printer without the CPU.
All characters transferred, and the CPU saw no interrupt along the way.
DMA raises a single interrupt; the handler acknowledges it.
The user process is woken and the handler returns. One interrupt per buffer, not per character.
DMA frees the CPU from I/O, but the DMA controller is usually slower than the CPU. If the device is very slow or the CPU has nothing else to do, interrupt-driven I/O or even PIO can be simpler and good enough (Tanenbaum).
DMA · Step-by-step calculation
Cost of the Three Methods
Polling: the CPU waits on every character: busy for 4096 × 1 ms = 4.096 s.
With programmed I/O the CPU is tied to the device's speed.
Interrupt-driven: 4096 interrupts × 2 µs = 8.19 ms (0.2% of the job's time).
The waiting is gone; the price is the number of interrupts.
DMA: 5 µs setup + 1 interrupt × 2 µs = ≈ 7 µs, about 1170 times less.
But DMA steals bus cycles, and its setup does not pay off for small transfers.
More than four orders of magnitude apart. With short transfers, slow DMA or an idle CPU, the DMA advantage disappears.
I/O Software
Goals of I/O Software
Programs should not care which device they use: sort < input works the same from a file, an SSD or a USB stick.
Names should not depend on the device type: /dev/nvme0n1, /mnt/usb/report.txt; mounting builds a single tree.
Errors should be handled as close to the hardware as possible: the controller corrects with ECC, the driver retries; upper layers hear about it only if they must.
Most hardware transfers are asynchronous (interrupt-driven), yet a blocking interface makes programming much easier for the user.
An incoming packet must be stored somewhere before we know where it goes; real-time devices such as audio also rely on buffers.
A disk serves many users at once; a printer or scanner is dedicated to one user (which opens the door to deadlock).
I/O Software
I/O Software Layers
Each layer has a well-defined job and an interface to the layer above. Tanenbaum counts four layers: user-level I/O software, device-independent software, device drivers and interrupt handlers. In Linux the “device-independent” part splits into the VFS, the page cache, the file system and the block layer.
I/O Software · Step by step
The Path of a read() Call
The process calls read(fd, buf, 4096); the libc wrapper enters the kernel with the syscall instruction.
The VFS maps the descriptor to the open file and the file to its i-node; permissions were already checked at open time.
The page cache is checked: the page is not there. A cache page is allocated and a read is started.
The file system maps the file offset to a disk block (LBA); the block layer builds a bio/request and hands it to a hardware queue.
The NVMe driver writes the read command into the submission queue and rings the doorbell register. The process blocks (state D).
The SSD DMAs the data into the cache page, posts an entry to the completion queue and raises an MSI-X interrupt.
The interrupt handler processes the completion: the page is marked “up to date” and the waiting process becomes ready.
When the process runs, the VFS copies the data from the cache into the user buffer and returns 4096.
If the same data is read again, it is a page-cache hit: the device is never touched and the call takes microseconds.
I/O Software
Device Drivers
- A driver encapsulates device-specific code: only it knows the registers, commands and quirks. Most Linux drivers live in the kernel tree (in-tree); some come from vendors.
- Its jobs: validate parameters, turn the request into device commands (LBA, queue entry), initialize the device if needed, write the registers, wait for the interrupt (block), check for errors, return the result.
- It must be reentrant: a new request or interrupt can arrive while it runs. With hot-plug, the device can vanish while in use.
- A buggy driver in the kernel crashes the whole system → user-space drivers: MINIX 3, FUSE, VFIO/UIO, DPDK/SPDK; Windows UMDF. Loadable modules:
modprobe,lsmod.
I/O Software
Device-Independent I/O Software
- A uniform interface for all drivers: every class offers the same functions, and the kernel calls them through a table of function pointers.
- Naming:
/dev/nvme0n1→ major number (driver) + minor number (unit). Protection: permissions on the device file. - A device-independent block size, buffering, error reporting, and allocating and releasing dedicated devices all live in this layer.
struct file_operations nvme_fops = {
.open = nvme_open,
.release = nvme_release,
.read_iter = nvme_read,
.unlocked_ioctl = nvme_ioctl,
.poll = nvme_poll,
};I/O Software · Step by step
Buffering
Unbuffered: every character raises an interrupt and wakes the process. Lots of context switches; inefficient.
User buffer: the interrupt handler writes characters straight into the process's buffer. What if that page is swapped out? It must be pinned, and if many processes pin pages, memory runs short.
Kernel buffer: characters collect in the kernel and are copied to the user in one go when the buffer fills. But where does data go that arrives during the copy?
Double buffering: one buffer fills while the other is copied, and the roles alternate. The general form is the circular buffer.
A packet may be copied 5 times on its way across the network; each copy costs memory bandwidth and latency. Hence zero-copy techniques.
Three reasons to buffer (Silberschatz): a speed mismatch between producer and consumer, a transfer-size mismatch (reassembling packets), and copy semantics (the user may change its buffer once write returns).
I/O Software · Step by step
Circular Buffer
Head and tail point to the same slot: the buffer is empty.
The interrupt handler writes 3 incoming bytes at the tail, and the tail advances. The consumer has not read yet.
The process reads 2 bytes and the head advances. Slots that were read can be reused.
The producer writes 5 more bytes; at the end the tail wraps around (mod 8): tail = 0.
After 'i', tail = 1 and (tail + 1) mod 8 == head: the buffer is full. Leaving one slot empty tells full from empty. When full, the producer waits or drops the data (lost network packets).
Circular buffers are everywhere: keyboard and serial drivers, NIC receive/transmit rings, NVMe submission/completion queues, io_uring. With a single producer and a single consumer no lock is needed, only memory barriers.
I/O Software · Step through the code
Circular Buffer: Code Trace
int put(char c) { /* producer */ int s = (tail + 1) % N; if (s == head) return -1; /* full */ buf[tail] = c; tail = s; return 0; } int get(char *c) { /* consumer */ if (head == tail) return -1; /* empty */ *c = buf[head]; head = (head + 1) % N; return 0; }
Buffer empty: head = tail = 0. The interrupt handler writes with put, the process reads with get; each changes only its own index.
put('a'): buf[0] = 'a', tail = 1.
put('b'): buf[1] = 'b', tail = 2.
put('c'): buf[2] = 'c', tail = 3.
get: head ≠ tail, so the buffer is not empty. buf[0] = 'a' is read, head = 1. Slot 0 can be reused.
put('d'): buf[3] = 'd', tail = 4.
put('e'): buf[4] = 'e', tail = 0. The tail reached the end and wrapped around (mod 5).
put('f'): s = (0 + 1) mod 5 = 1 = head. One slot always stays empty; otherwise head == tail would mean both empty and full. The byte is rejected (−1).
get: head ≠ tail, so the buffer is not empty. buf[1] = 'b' is read, head = 2. Slot 1 can be reused.
I/O Software
Caching, Spooling and Error Handling
A buffer temporarily holds the only copy of data; a cache is a fast copy of data that lives elsewhere. The Linux page cache combines both; free RAM goes to the cache (free -h: buff/cache).
A queue for a dedicated device: users drop files into a spool directory and a single daemon (CUPS, /var/spool/cups) prints them in turn. No process ever holds the printer.
A programming error (bad fd) → an error code (EBADF, EINVAL). A real I/O error → the driver retries; if it persists, EIO and a log entry (dmesg).
A dedicated device is claimed with open; if busy, the request is queued or EBUSY is returned. I/O instructions are privileged; users do I/O only through system calls.
User-space I/O software: library routines such as printf/scanf format data and buffer it (stdio line/block buffering), then call the write/read system calls.
I/O Interfaces
Blocking, Nonblocking and Asynchronous I/O
| Interface | Model | Note |
|---|---|---|
select / poll | readiness notification | scans the whole fd list on every call: O(n) |
epoll (Linux), kqueue (BSD) | readiness notification | interest list in the kernel; only ready fds return |
io_uring, Windows IOCP | completion notification | true asynchronous I/O, disks included |
I/O Interfaces · Step through the code
Event Loop: epoll
int ep = epoll_create1(0); for (k = 0; k < nsock; k++) /* O_NONBLOCK */ epoll_ctl(ep, EPOLL_CTL_ADD, s[k], &ev); for (;;) { int n = epoll_wait(ep, evs, 64, -1); for (int i = 0; i < n; i++) { int fd = evs[i].data.fd; ssize_t r = read(fd, buf, sizeof buf); if (r < 0 && errno == EAGAIN) continue; handle(fd, buf, r); } }
An epoll instance is created: a kernel object holding the interest and ready lists.
Three sockets (5, 6, 7) are added to the interest list in nonblocking mode. This happens once; unlike select, they are not passed again every round.
epoll_wait blocks the process while no socket is ready; the CPU does not spin.
A packet arrives on socket 7: NIC interrupt → network stack → fd 7 joins the ready list and the process wakes. n = 1.
Only the ready fd is handled: 512 bytes are read. Cost grows with the number of ready fds, not the total.
Data handled; when the next read returns EAGAIN there is nothing left. The loop goes back to epoll_wait.
One thread manages tens of thousands of connections (nginx, Node.js, Redis). epoll says “you may read”, but read still does the work; disk files always look “ready”, so true asynchrony for disks comes with io_uring.
I/O Interfaces · Step by step
io_uring: Shared Rings
io_uring (Linux 5.1, 2019): the application and the kernel see two rings in shared memory; requests and results are never copied.
The application writes three submission entries (SQEs): two reads and a write. Each carries an opcode, fd, buffer and offset.
A single io_uring_enter call submits all three requests. In SQPOLL mode a kernel thread polls the ring: no system call at all.
As devices finish, the kernel posts completion entries (CQEs) to the ring; the result field holds bytes transferred or −errno.
The application reads CQEs straight from memory and advances the head. Batched submission + batched reaping → very few context switches.
This is a completion model (like Windows IOCP): you are told “your job is done”. Hundreds of operation types exist, including network, file, accept and timers. Because of security bugs it is disabled in some environments (e.g. Android, some containers).
Clocks and Other Devices
Clock Hardware
- Example (8254 PIT): for a 1 kHz interrupt, counter = 1,193,182 / 1000 ≈ 1193 → actual frequency 1000.15 Hz.
- Modern x86: the TSC (time-stamp counter), HPET and the local APIC timer (TSC-deadline mode); ARM: the generic timer. A battery-backed RTC keeps the date while powered off.
- Clock software: time of day, quantum expiry, CPU usage accounting,
alarm/timers, watchdogs, profiling. - Keep the time of day in a 64-bit counter: a 32-bit
time_toverflows on 19 January 2038.
Clocks and Other Devices · Step by step
Clock Software: Ticks and Tickless Kernels
Periodic tick: the timer interrupts every 1/HZ seconds (Linux HZ = 100, 250 or 1000 → 10, 4, 1 ms). The handler advances time, charges the quantum and fires expired timers.
Tickless kernel (NO_HZ_IDLE, 2007): when the CPU is idle the tick stops and the clock is set one-shot to the next event. nohz_full removes the tick even while a single process runs on a core (HPC, real time).
Tanenbaum's example: pending signals at 4203, 4207, 4213, 4215 and 4216. The list stores only the differences; when the head reaches 0, its event fires.
Tickless operation saves energy and CPU time on laptops and VMs. For short (µs) waits, soft timers and polling are used together.
Clocks and Other Devices · Step through the code
Timer List: Code Trace
void add(Timer *z, int t) { /* t: ticks */ Timer **p = &head; while (*p && (*p)->delta <= t) { t -= (*p)->delta; p = &(*p)->next; } z->delta = t; z->next = *p; if (*p) (*p)->delta -= t; *p = z; } void tick(void) { if (head) head->delta--; while (head && head->delta == 0) head = run(head); /* remove */ }
Pending events at 4203, 4207, 4213, 4215, 4216; the list stores only the deltas: 3, 4, 6, 2, 1. New timer X fires 10 ticks from now (4210): add(X, 10).
A's delta 3 ≤ t: X goes after it. t = 7, and *p moves to the next node.
B's delta 4 ≤ t: X goes after it. t = 3, and *p moves to the next node.
C's delta 6 > t = 3: X must go before C. The loop ends.
X is spliced in with delta = 3; C's delta becomes 6 − 3 = 3. C's absolute time (4213) is unchanged; only one node was updated.
Tick: only the head delta drops → 2.
Tick: only the head delta drops → 1.
4203: the head delta hit 0; A runs and leaves the list. The work per tick is just decrementing the head counter: O(1), however long the list.
Clocks and Other Devices
Terminals, Input Devices and Power Management
- tty + line discipline: canonical mode edits the line (backspace, deliver on Enter); raw mode delivers every key at once (
stty -icanon, editors). - Pseudoterminal (pty): terminal emulators,
ssh; screen control through ANSI escape sequences. - The keyboard sends scan codes (press/release); the driver maps them to characters with a keymap. USB HID devices are polled by the host at 125–1000 Hz.
- Linux: input subsystem →
/dev/input/event*(evdev) → libinput → Wayland compositor → window.
- ACPI system states: S0 working (S0ix modern standby), S3 suspend to RAM, S4 hibernate (to disk), S5 off.
- CPU: C-states (idle sleep depth), P-states (frequency/voltage; Linux
schedutil). - Devices: D0–D3 states, runtime power management; NVMe APST, PCIe ASPM, Wi-Fi power saving, screen dimming.
- The OS puts devices to sleep and wakes them in the right order, and chooses the wakeup sources (keyboard, network, clock). Battery life depends mostly on idle time.
Disks
Magnetic Disk Geometry
- Platters spin at 5400–7200 rpm (servers: 10–15k), with one head per surface. Tracks at the same radius form a cylinder. Outer zones hold more sectors per track.
- The OS sees LBAs, not CHS (cylinder, head, sector); the controller inside the disk maps them to the real geometry, spare sectors and cache. The old CHS limit: 1024 × 255 × 63 × 512 B ≈ 8.4 GB.
Disks · Step-by-step calculation
Access Time: An Example
All three components count, transfer time included.
One revolution = 60,000 / 7200 = 8.33 ms; average rotational latency = half a revolution = 4.17 ms.
On average the target sector is half a turn away.
Transfer = 4096 B ÷ 200 MB/s = 0.020 ms. Total = 8 + 4.17 + 0.02 = 12.19 ms.
Mechanical delay dominates: almost all the time goes to seek and rotation. The CPU could run millions of instructions meanwhile; the OS does not wait, it runs another process.
Random 4 KiB: 1000 / 12.19 ≈ 82 IOPS → 82 × 4 KiB ≈ 0.34 MB/s. Sequential 1 MiB: 8 + 4.17 + 5.24 = 17.41 ms → 60 MB/s.
The same disk is 180 times faster or slower depending on the access pattern, which is why locality and scheduling matter. (The math assumes a constant media rate and a fresh seek + rotation per chunk; a long sequential read seeks once and approaches the media rate.)
Compare: an NVMe SSD does a random 4 KiB read in ≈ 50–100 µs and over a million IOPS with parallel queues.
Disks
Disk Formatting and Cylinder Skew
- Low-level formatting is done at the factory: tracks, sectors, preamble, ECC and spare sectors. Formatted capacity is less than raw capacity. High-level formatting writes an empty file system to a partition (boot block, free-space management, root directory).
- Skew math (Tanenbaum): 10,000 rpm → 6 ms per revolution; 300 sectors/track → 20 µs per sector. With an 800 µs track switch, skew = 800 / 20 = 40 sectors.
- Interleaving once spaced consecutive sectors apart for slow controllers; today the controller buffers the whole track, so it is unnecessary.
- Today's disks: 4 KiB physical sectors (Advanced Format); 512e disks emulate 512 B, and unaligned writes need read-modify-write.
Disks
Optical Disks (Briefly)
| CD (1982) | DVD (1996) | Blu-ray (2006) | |
|---|---|---|---|
| Capacity | 650–700 MB | 4.7 / 8.5 / 9.4 / 17 GB | 25 / 50 GB; BDXL 100–128 GB |
| Laser | 780 nm (infrared) | 650 nm (red) | 405 nm (blue-violet) |
| Smallest pit | 0.83 µm | 0.40 µm | 0.15 µm |
| Track pitch | 1.6 µm | 0.74 µm | 0.32 µm |
- A single spiral track; a transition between pit and land is a 1 bit. On pressed discs the pits are molded into polycarbonate, not burned by a laser. On CD-R/DVD-R the laser heats an organic dye layer to change its reflectivity.
- On a CD-ROM each byte becomes a 14-bit channel symbol via EFM (a modulation code, not ECC); error correction uses CIRC. 98 frames × 24 data bytes = 2352 B; in mode 1, 2048 B of that is user data and the rest is sync, header and extra ECC/EDC.
- A DVD's two layers are semitransparent; the laser reads the lower one by refocusing. Optical media now serve mainly archiving and distribution; their density trails magnetic and flash media.
Disk Scheduling
Disk Arm Scheduling
- The goal is to cut arm movement (seek time) and raise throughput; the driver keeps a list of (cylinder, request time). Metric: total head movement (cylinders).
- Today the LBA → physical mapping lives inside the disk, and the disk itself reorders 32 requests by rotational position with NCQ. The OS schedules approximately, by “LBA proximity”.
| Algorithm | Idea | Problem |
|---|---|---|
| FCFS | Arrival order | Fair, but the arm wanders a lot |
| SSTF | Request closest to the head (greedy) | Requests at the edges can starve |
| SCAN (elevator) | Sweep to the end in one direction, then reverse | Edges and middle do not wait equally |
| C-SCAN | Sweep one way, jump from the end back to the start | The return trip is wasted |
| LOOK / C-LOOK | SCAN/C-SCAN, but only up to the last request | The forms used in practice |
The examples use Silberschatz's queue: head at 53; requests 98, 183, 37, 122, 14, 124, 65, 67. Cylinders 0–199.
Disk Scheduling · Step by step
FCFS
First come, first served: requests in arrival order.
The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.
53 → 98: 45 cylinders · total 45
98 → 183: 85 cylinders · total 130
183 → 37: 146 cylinders · total 276
37 → 122: 85 cylinders · total 361
122 → 14: 108 cylinders · total 469
14 → 124: 110 cylinders · total 579
124 → 65: 59 cylinders · total 638
65 → 67: 2 cylinders · total 640
640 cylinders in all. Fair and starvation-free, but swings like 122 → 14 → 124 are very costly.
Disk Scheduling · Step by step
SSTF
Shortest seek time first: at each step, the request closest to the head.
The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.
53 → 65: 12 cylinders · total 12
65 → 67: 2 cylinders · total 14
67 → 37: 30 cylinders · total 44
37 → 14: 23 cylinders · total 67
14 → 98: 84 cylinders · total 151
98 → 122: 24 cylinders · total 175
122 → 124: 2 cylinders · total 177
124 → 183: 59 cylinders · total 236
236 in all (37% of FCFS). It is greedy and not optimal: 53 → 37 → 14 → 65 … is shorter (208). Under heavy load, requests in the middle can starve those at the edges.
Disk Scheduling · Step through the code
SSTF: Code Trace
int sstf(int head, int q[], int n) { int total = 0; for (int k = 0; k < n; k++) { int best = -1; for (int i = 0; i < n; i++) if (!done[i] && (best < 0 || abs(q[i]-head) < abs(q[best]-head))) best = i; /* closest */ total += abs(q[best] - head); head = q[best]; done[best] = 1; } return total; }
The head is at cylinder 50; queue (arrival order): 82, 170, 43, 140, 24, 16, 190. Each round, the inner loop looks at every waiting request.
k = 0: head at 50, nearest waiting request 43 (distance 7; runner-up 24, distance 26). total = 7.
k = 1: head at 43, nearest waiting request 24 (distance 19; runner-up 16, distance 27). total = 26.
k = 2: head at 24, nearest waiting request 16 (distance 8; runner-up 82, distance 58). total = 34.
k = 3: head at 16, nearest waiting request 82 (distance 66; runner-up 140, distance 124). total = 100. The low requests are done, so the arm travels a long way back.
k = 4: head at 82, nearest waiting request 140 (distance 58; runner-up 170, distance 88). total = 158.
k = 5: head at 140, nearest waiting request 170 (distance 30; runner-up 190, distance 50). total = 188.
k = 6: head at 170, nearest waiting request 190 (distance 20). total = 208.
208 cylinders in all (FCFS: 642 on this queue). The greedy choice takes the shortest move each time; if requests keep arriving in the middle, those at the edges may wait.
Disk Scheduling · Step by step
SCAN (Elevator)
SCAN: the head sweeps in one direction (here toward 0) to the end, then reverses, like an elevator.
The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.
53 → 37: 16 cylinders · total 16
37 → 14: 23 cylinders · total 39
14 → 0: 14 cylinders · total 53 (no request; the arm goes to the edge)
0 → 65: 65 cylinders · total 118
65 → 67: 2 cylinders · total 120
67 → 98: 31 cylinders · total 151
98 → 122: 24 cylinders · total 175
122 → 124: 2 cylinders · total 177
124 → 183: 59 cylinders · total 236
236 in all. Getting stuck in one region is SSTF's greedy flaw; SCAN prevents starvation. Going to the edge and back (14 → 0 → 14) wastes 28 cylinders.
Disk Scheduling · Step by step
C-SCAN
Circular SCAN: serve in the upward direction only; at the end, return to the start without serving.
The head is at 53. Queue: 98, 183, 37, 122, 14, 124, 65, 67 (cylinders 0–199). Red: waiting, green: served.
53 → 65: 12 cylinders · total 12
65 → 67: 2 cylinders · total 14
67 → 98: 31 cylinders · total 45
98 → 122: 24 cylinders · total 69
122 → 124: 2 cylinders · total 71
124 → 183: 59 cylinders · total 130
183 → 199: 16 cylinders · total 146 (no request; the arm goes to the edge)
199 → 0: return jump (199 cylinders), no requests served. Counted here · total 345
0 → 14: 14 cylinders · total 359
14 → 37: 23 cylinders · total 382
382 in all (183 without the return). The disk is treated as a ring; waiting times are more uniform than with SCAN.
Disk Scheduling · Step by step
LOOK and C-LOOK
LOOK (downward): like SCAN, but it does not go to the edge; it reverses at the last request in that direction.
53 → 37 → 14: the low requests are done, so no trip to 0. Total 39.
14 → 65 … 183: direction reverses. Total 208 (28 less than SCAN). In practice, the “elevator” behavior is usually this LOOK.
C-LOOK (upward): 53 → 65 … 183, the last request going up. Total 130.
183 → 14: jumps to the lowest waiting request (169 cylinders), skipping 199 and 0.
14 → 37: total 322 (153 without the jump). A one-way sweep evens out waiting times.
Linux's old deadline and CFQ schedulers and today's mq-deadline dispatch in LBA order, which behaves much like C-LOOK.
Disk Scheduling · Step through the code
LOOK: Code Trace
/* s[] sorted; go up first, then down */ int look(int head, int s[], int n) { int i = 0, total = 0; while (i < n && s[i] < head) i++; for (int j = i; j < n; j++) { total += s[j] - head; head = s[j]; } for (int j = i - 1; j >= 0; j--) { total += head - s[j]; head = s[j]; } return total; }
The same queue, sorted: s = 16, 24, 43, 82, 140, 170, 190. The head is at 50 and moving up.
i is the index of the first request above 50: s[3] = 82. The 16, 24, 43 below are taken on the way back.
j = 3: 82 (+32), total = 32.
j = 4: 140 (+58), total = 90.
j = 5: 170 (+30), total = 120.
j = 6: 190 (+20), total = 140. The last request going up: the arm turns back without going to 199 (unlike SCAN).
Direction reversed, j = 2: 190 → 43 (+147), total = 287. The longest move is on the way back.
j = 1: 24 (+19), total = 306.
j = 0: 16 (+8), total = 314.
314 in all; SSTF gets 208 on the same queue. Had the head been moving down, LOOK would also give 34 + 174 = 208: the direction decides the result, but no request starves.
Disk Scheduling
Comparing the Algorithms
| Tanenbaum's example: head 11; 1, 36, 16, 34, 9, 12 | Movement |
|---|---|
| FCFS: 10 + 35 + 20 + 18 + 25 + 3 | 111 |
| SSTF: 12, 9, 16, 1, 34, 36 → 1 + 3 + 7 + 15 + 33 + 2 | 61 |
| Elevator (up): 12, 16, 34, 36, 9, 1 → 1 + 4 + 18 + 2 + 27 + 8 | 60 |
On the charted queue, SCAN starting upward would give 331: the direction changes the result. Tanenbaum's elevator does not go to the edge, so it is really LOOK.
Disk Scheduling · Step by step
SPTF: Counting Rotational Latency Too
The head is on track 10 with two requests waiting. SSTF, which looks only at track distance, sees A as far closer: one track away. But time is set by seek + rotation, not seek alone.
SSTF picks A: during the 1 ms seek the disk turns 43.2°, so the sector that was 30° away is 13.2° past the head when the seek ends. Almost a full turn of waiting: 1 + 8.03 = 9.03 ms.
SPTF picks B: during the 3 ms seek the disk turns 129.6°, leaving 20.4° to B: 3 + 0.47 = 3.47 ms. The far track finished 2.6 times sooner than the near one.
SPTF (shortest positioning time first) minimizes positioning time, but the OS does not know sector angles, so it is implemented inside the disk via NCQ. Against starvation, F-SCAN freezes the queue when a sweep starts; new requests wait for the next sweep.
Disk Scheduling · Step by step
Controller Cache and Bad Sectors
The disk keeps a track cache in its own DRAM (≈ 256 MB): it also reads unrequested blocks passing under the head. This is separate from the OS cache.
Writes may sit in the cache: durability needs FLUSH/FUA. NCQ orders 32 requests by rotational position.
A sector that fails in manufacturing or in use: written data cannot be read back. The controller tracks the factory list and those found in use (SMART: Reallocated Sectors).
Method 1: the bad sector is remapped to a spare. Simple, but a sequential read through sector 7 may cost an extra revolution.
Method 2: the sectors are shifted; the whole track must be rewritten, but order is preserved. The OS can also keep a bad-block list (badblocks, the file system).
SSD and NVMe
SSD: NAND Flash
| Cell | Bits/cell | Typical P/E cycles | Use |
|---|---|---|---|
| SLC | 1 | ≈ 50–100k | cache region, industrial |
| MLC | 2 | ≈ 3–10k | older enterprise |
| TLC | 3 | ≈ 1–3k | most consumer and server SSDs |
| QLC | 4 | ≈ 0.1–1k | high capacity, read-heavy |
- No moving parts: no seek and no rotational latency, so random access ≈ sequential access.
- No in-place update: erase, then write; each cell survives a limited number of erases. 3D NAND has 200–300+ layers.
- The price-per-GB gap has narrowed; HDDs are still cheaper for archives and capacity.
SSD and NVMe · Step by step
FTL, Garbage Collection, TRIM, Wear Leveling
Three logical blocks are written; the FTL places them in B0's free pages and records them in the table.
LBA 1 is rewritten: the old page cannot be overwritten. The new data goes to a free page and the old one becomes invalid (out-of-place update).
When a file is deleted the file system sends TRIM (NVMe: Deallocate), so the SSD learns the page is no longer needed. Without TRIM, garbage collection would copy this data for nothing.
B0 has no free pages left; new writes go to B1. Half of B0 is invalid.
Garbage collection: B0's valid pages (LBA 0, 1) are copied to B1 and B0 is erased. The 2 extra page writes are write amplification.
Wear leveling: the FTL evens out erase counts, moving long-unchanged (cold) data too so that all blocks age equally. Spare area (over-provisioning) makes garbage collection easier.
Windows “Optimize” does not defragment an SSD; it sends TRIM. Defragmenting an SSD only adds writes and wear. Aligning partitions to a 1 MiB boundary keeps logical blocks from straddling flash pages.
SSD and NVMe
NVMe and Multiple Queues
- NVMe is a protocol designed for flash on PCIe; M.2 is a form factor (an M.2 card can also be SATA). U.2/U.3 and E1.S/E3.S are cabled/slotted server forms.
- Queue capacity: 64K queues × 64K commands per queue. Every OS has shipped a driver since 2013; the incompatibility with AHCI is at the protocol level only.
- Latency ≈ 10–100 µs; PCIe 5.0 x4 drives reach ≈ 14 GB/s and > 2 million IOPS. NVMe-oF runs it over a network; ZNS adds zoned namespaces.
SSD and NVMe
Linux blk-mq and I/O Schedulers
| Scheduler | Idea | Best for |
|---|---|---|
none | No sorting, dispatch directly | NVMe (default), fast SSDs |
mq-deadline | Sorted by LBA + read/write deadlines | SATA HDD/SSD (default on most distros) |
bfq | Per-process budgets, fair sharing | Desktops, interactivity on slow devices |
kyber | Throttling against read/write latency targets | Fast multiqueue devices |
$ cat /sys/block/nvme0n1/queue/scheduler [none] mq-deadline kyber bfq $ echo bfq | sudo tee /sys/block/sda/queue/scheduler
Per-CPU software queues avoid contention on a single queue lock at millions of IOPS; Linux 5.0 removed the old single-queue block layer (noop, deadline, CFQ). An SSD has no seeks, so the scheduler's job is not ordering but protecting fairness and latency.
RAID and Stable Storage
RAID: Combining Disks
- RAID (Patterson, Gibson, Katz, 1988) presents many disks to the OS as one disk; striping gives parallelism, mirroring and parity give redundancy. The opposite is SLED: a single large expensive disk.
- Today the acronym reads Redundant Array of Independent Disks. SCSI disks were expensive; now SAS expanders attach hundreds of disks, and the standard levels are 0–6.
- Hardware RAID cards or software RAID (Linux
mdadm, ZFS RAID-Z, Btrfs, Windows Storage Spaces). RAID is not a backup: a deleted file is deleted on every disk.
| 4 × 4 TB disks | Usable | Survives | Small random-write cost |
|---|---|---|---|
| RAID 0 (striping) | 16 TB | 0 disks | 1 I/O |
| RAID 1 (4-way mirror) | 4 TB | 3 disks | 4 I/Os (one per copy) |
| RAID 10 (mirror + stripe) | 8 TB | 1 (2 if in different pairs) | 2 I/Os |
| RAID 5 (distributed parity) | 12 TB | 1 disk | 4 I/Os (2 reads + 2 writes) |
| RAID 6 (dual parity) | 8 TB | 2 disks | 6 I/Os (3 reads + 3 writes) |
RAID and Stable Storage · Step by step
RAID Levels
RAID 0: consecutive stripes go to different disks; n times the bandwidth for large requests. Lose one disk and you lose everything.
RAID 1: every write happens twice; a read comes from either disk (load is spread). Half the capacity is redundancy.
RAID 10: mirror first, then stripe. Common for databases: fast writes and fast rebuilds.
RAID 2 uses words + a Hamming code, RAID 3 bytes + a single parity disk; the disks must spin in sync. Neither is used today (disks do their own ECC).
RAID 4: block stripes + a dedicated parity disk. Every small write updates the parity disk, which becomes the bottleneck.
RAID 5: parity blocks rotate across the disks, removing the bottleneck. Capacity of n − 1 disks; survives one disk failure.
RAID 6: P (XOR) + Q (Reed–Solomon, GF(2⁸)) parity; survives two disk failures. Preferred for large disks, where a rebuild takes hours.
RAID and Stable Storage · Step by step
Parity and XOR: An Example
Three data blocks in a stripe (an 8-bit example here). The parity block is not computed yet.
Parity is a bitwise XOR: each bit position is completed so the number of 1s is even.
Disk 1 failed. Since x ⊕ x = 0, the lost block is the XOR of the others.
The recovered value 0101 1100 equals the original D1. Every stripe is rewritten this way onto a new disk (the rebuild).
Small write: D1' = 0000 1111. Read the old D1 and P, compute P', write D1' and P': 4 I/Os (read-modify-write).
Rebuild risk: a consumer disk has an unrecoverable read error (URE) about once per 1014 bits. Rebuilding 12 TB gives 12×1012×8 / 1014 ≈ 0.96 expected errors, a ≈ 62% chance of at least one: a RAID 5 rebuild will likely fail, so large arrays use RAID 6.
RAID and Stable Storage · Step through the code
RAID 5 Small Write and Recovery: Code Trace
void small_write(int d, uint8_t nv) { uint8_t ov = rd(d); /* I/O 1 */ uint8_t p = rd(P); /* I/O 2 */ p = p ^ ov ^ nv; wr(d, nv); /* I/O 3 */ wr(P, p); /* I/O 4 */ } uint8_t rebuild(int bad) { uint8_t x = 0; for (int d = 0; d <= P; d++) if (d != bad) x ^= rd(d); /* incl. P */ return x; }
In the stripe P = D0 ⊕ D1 ⊕ D2 = 0001 1110. D1 is to be overwritten with 0011 0011: small_write(1, ...).
The old D1 is read. The other data disks (D0, D2) are not: only 1 of the 3 data disks in the stripe changes.
The old parity is read: p = 0001 1110.
p ⊕ ov removes the old D1 from the parity (x ⊕ x = 0); ⊕ nv adds the new value. p = 0100 0100.
The new D1 is written.
The new P is written: 4 I/Os (2 reads + 2 writes). A crash between these two writes leaves the stripe inconsistent (the “write hole”).
Disk 2 failed: D2 cannot be read. rebuild(2) computes it as the XOR of the rest.
d = 0: x ^= D0 → x = 1011 0010.
d = 1: x ^= D1 → x = 1000 0001.
d = 3: x ^= P → x = 1100 0101. (d = 2 is skipped.)
x = 1100 0101, the same as the lost D2. Each stripe is rebuilt onto a new disk with 3 reads; the array is unprotected meanwhile.
RAID and Stable Storage · Calculation
RAID Performance: An Example
| Level | Capacity | Sequential read/write | Random read | Random write | Write latency |
|---|---|---|---|---|---|
| RAID 0 | N = 4 | N·S ≈ 649 MB/s | N·R ≈ 328 | N·R ≈ 328 | D |
| RAID 1 (two-way mirror) | N/2 = 2 | (N/2)·S ≈ 325 MB/s | N·R ≈ 328 | (N/2)·R ≈ 164 | D (slowest copy) |
| RAID 4 | N − 1 = 3 | (N−1)·S ≈ 487 MB/s | (N−1)·R ≈ 246 | R/2 ≈ 41 | 2D |
| RAID 5 | N − 1 = 3 | (N−1)·S ≈ 487 MB/s | N·R ≈ 328 | (N/4)·R ≈ 82 | 2D |
- Sequential reads on a mirror get only (N/2)·S: the two copies split the blocks, but each disk still rotates under the blocks it skips.
- RAID 4: every small write needs 1 read + 1 write on the parity disk; the array drops to half the speed of one disk, and adding disks does not help.
- RAID 5: the same 4 I/Os spread over all disks, so N·R/4 grows with the disk count. A full-stripe write reads no old data: (N−1)·S.
RAID and Stable Storage · Step by step
Stable Storage
- Stable storage: a write either completes correctly or leaves the old data in place. Assumptions: the same block failing on both disks at once is negligible; bad data is detected by ECC; if the CPU crashes, the write in progress stops.
- Stable write: write disk 1, read it back, verify (retry n times, use a spare sector if needed); then do the same on disk 2. Stable read: if disk 1 cannot be read with valid ECC, read disk 2.
(a) Crash before writing disk 1: both copies are old. The write never happened.
(b) Crash while writing disk 1: disk 1 is bad (ECC). The old value is restored from disk 2.
(c) Disk 1 written, crash before disk 2: the new value is completed.
(d) Crash while writing disk 2: disk 2 is bad, so the new value is copied from disk 1.
(e) Both copies new: the write is complete. In every case the result is either entirely old or entirely new.
RAID protects against failing sectors, not against a crash mid-write. Today the same idea appears as journaling, write barriers/FUA and the power-loss capacitors in enterprise SSDs.
Summary
Summary
| Topic | Key idea | Today |
|---|---|---|
| Hardware access | Controller registers; port- or memory-mapped | MMIO through PCIe BARs |
| Transfer | Polling → interrupts → DMA | MSI-X, bus-master DMA, IOMMU |
| Software layers | User / independent / driver / interrupt | VFS + page cache + blk-mq |
| Interface | Blocking, nonblocking, asynchronous | epoll, io_uring |
| Buffering | Speed/size mismatch, double and circular buffers | NIC/NVMe rings, zero copy |
| HDD | Seek + rotation + transfer; scheduling | NCQ, mq-deadline, BFQ |
| SSD | No seeks; FTL, garbage collection, TRIM | NVMe, multiqueue, none |
| Reliability | RAID 0/1/5/6/10, stable storage | mdadm, ZFS, journaling |
The essence of I/O design: decouple slow, diverse devices from the fast CPU (buffers, interrupts, DMA), hide their differences behind a uniform interface (drivers, VFS), and choose the order wisely (scheduling, batching).
Summary
Review Questions
- With port-mapped and with memory-mapped I/O, which instruction accesses a device register? How is the caching problem solved for MMIO?
- 4096 characters go to a 1000-character-per-second printer using interrupt-driven I/O, at 2 µs per interrupt. What is the CPU cost?
- What is cycle stealing? When does DMA not pay off?
- How long does a random 4 KiB read take on a disk with 7200 rpm, 8 ms average seek and a 200 MB/s media rate?
- Head at 53, queue 98, 183, 37, 122, 14, 124, 65, 67 (0–199). What is the total movement for SSTF and C-LOOK (upward)?
- Head at 11, requests 1, 36, 16, 34, 9, 12. How many cylinders do FCFS, SSTF and the elevator (upward) move?
- In RAID 5, D0 = 1011 0010, D2 = 1110 0111, P = 0000 1001. What is the failed D1? If one disk gives 100 IOPS, how many IOPS do 6-disk RAID 5 and RAID 4 give for random small writes?
- Why does an SSD need TRIM? Which I/O scheduler is the Linux default for NVMe devices, and why?
- What is the fundamental difference in model between
epollandio_uring?
Answers
- Port:
IN/OUT; MMIO: an ordinaryMOV. Register pages are marked uncacheable (PAT/MTRR,ioremap). - 4096 × 2 µs = 8.19 ms (4.096 s with polling).
- DMA briefly takes the bus for each word, so the CPU waits on memory accesses. It does not pay off for small transfers, a slow DMA controller, or a CPU with nothing else to do.
- 8 + 4.17 + 0.02 = 12.19 ms (≈ 82 IOPS).
- SSTF 236; C-LOOK 322 (153 without the jump).
- 111, 61, 60.
- D0 ⊕ D2 ⊕ P = 0101 1100. RAID 5: 6 × 100 / 4 = 150; RAID 4: the parity disk is the bottleneck, 100 / 2 = 50.
- The SSD learns which pages are no longer valid; garbage collection skips them and write amplification drops.
none: no seeks, multiqueue is already parallel, and sorting costs CPU. - epoll reports readiness (you then call
read); io_uring reports completion and submits requests in batches through shared rings.
Summary
References
- A. Silberschatz, P. B. Galvin, G. Gagne, Operating System Concepts, 10th ed., Wiley, 2018 — Chapter 11: Mass-Storage Structure; Chapter 12: I/O Systems.
- A. S. Tanenbaum, H. Bos, Modern Operating Systems, 4th ed. (2014) / 5th ed. (2023), Pearson — Chapter 5: Input/Output.
- R. H. Arpaci-Dusseau, A. C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces (OSTEP), ostep.org — “I/O Devices”, “Hard Disk Drives”, “RAID”, “Flash-based SSDs”.
- D. A. Patterson, G. Gibson, R. H. Katz, “A Case for Redundant Arrays of Inexpensive Disks (RAID)”, SIGMOD 1988.
- NVM Express Base Specification 2.x; PCI-SIG PCIe 5.0/6.0; USB4 Specification; UEFI Specification (GPT).
- Linux kernel documentation: “Multi-Queue Block IO Queueing Mechanism (blk-mq)”, “NO_HZ: Reducing Scheduling-Clock Ticks”,
io_uring(J. Axboe, “Efficient IO with io_uring”), “Linux Device Drivers” (Corbet, Rubini, Kroah-Hartman, 3rd ed.).