Chapter 10: File Systems

Operating Systems
The file concept and attributesFile operationsOpen-file tables and fdsAccess methodsDirectory structures and path resolutionHard and symbolic linksMounting and protection (rwx)MBR, GPT, and the superblockContiguous, linked, FAT, i-nodeFree-space managementCaching and performancefsck, journaling, CoW, LFSNTFS, ext4, FAT32, exFATVFS, NFS, and SSDs
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

The file abstraction

Define a file by its name, type, attributes, and operations; trace open-file tables and file descriptors.

Directories

Compare directory structures, resolve a path step by step, and explain hard/symbolic links and mounting.

Protection

Read rwx permission bits and the special bits; tell what r, w, and x mean for a file and for a directory.

Allocation

Apply contiguous, linked, FAT, and i-node allocation to the same file; compute the maximum file size.

Free space and performance

Compare bitmap, list, grouping, and counting methods; explain the buffer/page cache and read-ahead.

Reliability and real systems

Compare fsck, journaling, CoW, and LFS; recognize NTFS, ext4, FAT32/exFAT, VFS, NFS, and SSD TRIM.

Introduction

Why a File System?

  • Three requirements: very large data (it won't fit in an address space), persistence (it outlives the process), and concurrent access (many processes use the same data).
  • A disk offers only numbered blocks: "read / write block k". The file system builds the file and directory abstractions on top.
  • A file is a logical unit, much like an address space in memory; processes create files, and the OS manages naming, structure, access, protection, and placement.
  • Two viewpoints:
    • User: name, type, directory, permissions, operations: what is visible?
    • Implementer: how are blocks allocated, free space tracked, and consistency kept after a crash?
Applicationopen · read · write · closeFile abstractionname · byte sequence · attributesFile systemfile offset → block numberBlock device (SSD / HDD)LBA 0 … N−1 · 512 B / 4 KiBdisk = numbered fixed-size blocks

The File Concept

File Naming

File systemName lengthCaseNot
FAT (MS-DOS)8.3 (e.g., REPORT~1.TXT)insensitiveVFAT long names in extra entries: 255 UTF-16
NTFS255 UTF-16 unitspreserving, insensitive\ / : * ? " < > | forbidden
ext4, XFS255 bytessensitiveonly / and NUL forbidden
APFS255 UTF-8 charactersinsensitive by default on macOSUnicode normalization
Extension (suffix)

The part of the name after the dot: thesis.pdf. On UNIX it is just a convention; the kernel gives it no meaning, but programs may (gcc tells .c from .o).

Windows

The extension is associated with a program in the registry; double-clicking opens that program. The hidden-extension trap: invoice.pdf.exe.

Older Windows versions used FAT16/FAT32, current Windows uses NTFS; FAT32 and exFAT live on today on USB sticks and SD cards. Names are stored as Unicode; non-ASCII characters are no problem.

The File Concept

Extensions and Magic Numbers

GroupExtensions and what they stand for
Documentdocx Word (Office Open XML), doc legacy binary Word, pdf Portable Document Format, odt OpenDocument, rtf, txt
Imagejpg JPEG, png Portable Network Graphics, gif, bmp bitmap, tiff; new: webp, avif, heic
Audio / videomp3 MPEG-1 Audio Layer III, wav, m4a, flac; mp4 MPEG-4 Part 14, mkv, webm, mov; flv is obsolete
Spreadsheet / slidesxlsx, csv Comma-Separated Values, pptx, odp
Webhtml HyperText Markup Language, css Cascading Style Sheets, js JavaScript
Executableexe Windows PE, msi Windows Installer; on Linux ELF has no extension, libraries are .so
Archivezip, 7z, rar, tar Tape Archive, gz gzip, zst Zstandard
the file's first bytesPNG89 50 4E 47‰PNGPDF25 50 44 46%PDFELF7F 45 4C 46.ELFZIP · docx · pptx50 4B 03 04PK..Windows PE4D 5AMZgzip1F 8B..

A file's type can also be recognized from its magic number, regardless of extension: the file report command does this. docx/xlsx/pptx are really ZIP archives.

The File Concept

File Structure

(a) Byte sequence1 byte = the smallest unitUNIX, Windows, macOSthe application gives it meaning(b) Fixed-length recordsrecord 1 · 80 bytesrecord 2 · 80 bytesrecord 3 · 80 bytesrecord 4 · 80 bytesrecord 5 · 80 bytesrecord 6 · 80 bytespunched-card image (80 columns)today: mainframe data sets(c) Tree of keyed recordsDog · PonyAnt Cat CowFox Hen LionRat Worma record is found by its key fieldtoday: database / B-tree files
  • Maximum flexibility: the OS sees a file only as a sequence of bytes; anything can go in it, and the application decides the structure.
  • With records, the unit of reading/writing is a record; with a tree, requests like "get the record whose key is Pony" are supported directly.
  • Today structured access is implemented in libraries and databases (SQLite, B+ trees), not in the OS.

The File Concept

File Types

  • Regular files hold user data: text (lines of printable characters, ending in LF or CRLF) or binary (the program knows the structure).
  • Directories are system files that hold the structure of the file system.
  • Special files model I/O devices: character devices (printer, terminal) and block devices (disk).
  • "Everything is a file": the same open/read/write interface works for devices, pipes, and sockets. The letter in the table is the first character of ls -l output.
LetterTypeExample
-regular file (text or binary)notes.txt, /bin/ls
ddirectory/home
lsymbolic link/bin → usr/bin
ccharacter device (serial, byte by byte)/dev/tty, /dev/null
bblock device (block by block, random)/dev/nvme0n1, /dev/sda
pnamed pipe (FIFO)mkfifo chan
sUNIX socket/run/systemd/notify

The File Concept

Binary Files: Executables and Archives

(a) Old UNIX a.outmagic numbertext sizedata sizeBSS sizesymbol table sizeentry pointtext (code)datarelocation bitssymbol tableheader + sections(b) Today: ELF (Linux)ELF header · 7F 45 4C 46program headers.text (code).rodata.data.symtab · .strtabsection headersWindows: PE · macOS: Mach-O(c) Archive (.a / .lib)!<arch> headermember header: name · sizeobject module (.o)member header: name · sizeobject module (.o)member header: name · sizeobject module (.o)compiled, unlinked modules

Each OS recognizes its own executable format; the kernel runs a file only if it starts with the right magic number. The UNIX file in the figure is the old a.out format (Tanenbaum); the archive extension is .a, and the modules inside are .o.

The File Concept

File Attributes

AttributeMeaningWhere / how
Name, type, locationthe name is in the directory entry; the type and block addresses are in the i-node/MFT recordall systems
Sizelogical size (st_size) and allocated space (st_blocks)they differ in a sparse file
Owner, groupUID / GID; SID on Windowschown, icacls
Protectionrwx bits, ACLschmod, setfacl
Time stampsaccess (atime), content change (mtime), i-node change (ctime), creation (btime)stat; relatime on Linux
Link countnumber of names that refer to the fileblocks freed when it reaches 0
Read-only, Hidden, Systemread-only, hidden, system fileWindows attrib; on UNIX, hidden = the name starts with .
Archivechanged since the last backup, to be backed upWindows; the backup tool clears it
Compressed, Encryptedcompressed / encryptedNTFS, Btrfs, APFS; fscrypt, BitLocker
Sparse, Temporaryzero blocks absent on disk; temporary (a cache hint)NTFS flags
Reparse pointspecial handling: symbolic link, junction, cloud placeholderNTFS
Extended attributesname = value pairsxattr (SELinux label), NTFS ADS

Symlink, Device, Junction are file types, not attributes; Temporary doesn't delete the file by itself (it's only a hint to delay writes); Creator and Max size belong to old systems.

The File Concept

The stat Structure and ls -l

struct stat {
  dev_t     st_dev;      /* device (file system)   */
  ino_t     st_ino;      /* i-node number          */
  mode_t    st_mode;     /* type + permission bits */
  nlink_t   st_nlink;    /* hard link count        */
  uid_t     st_uid;      /* owner user             */
  gid_t     st_gid;      /* owner group            */
  dev_t     st_rdev;     /* no. if a device file   */
  off_t     st_size;     /* size (bytes)           */
  blksize_t st_blksize;  /* preferred I/O size     */
  blkcnt_t  st_blocks;   /* 512 B units allocated  */
  struct timespec st_atim; /* last access  */
  struct timespec st_mtim; /* last content */
  struct timespec st_ctim; /* last i-node  */
};
$ ls -l b.txt-rwxr-x---typeownergroupother1→ hard link count (st_nlink)root root→ owner and group81→ size in bytes (st_size)May 26 19:20→ last modification (mtime)b.txt→ name: comes from the directory entrythe name is in the directory, not the i-node!

Since POSIX.1-2008 the times are struct timespec fields with nanoseconds (st_atime remains as a macro). On Linux the creation time is read with statx().

The File Concept

File Operations and System Calls

OperationPOSIX callWhat it does
Createopen(O_CREAT), creatcreates an empty file and directory entry, sets the attributes
Deleteunlinkremoves the name; the space is freed when the link count is 0 and the last descriptor closes
Openopenresolves the path, brings the i-node into memory, returns a descriptor (fd)
Closeclosefrees the fd and, if needed, the table entries
Read / Writeread, writereads / writes from the file position and advances it
Appendopen(O_APPEND)each write goes atomically to the end of the file
Seeklseekchanges the position (random access); does no I/O
Get / Set attributesstat, chmod, chown, utimensatreads / changes attributes
Renamerenamechanges the name; atomic within one file system
Truncate, Mapftruncate, mmapshortens/extends the file; maps it into the address space
1 · write report.tmpreport: old2 · fsync(report.tmp)data on disk3 · rename → reportreport: new (atomic)

The safe-save pattern: write the new contents to a temporary file → fsync → rename(tmp, real). After a crash either the old or the new file is visible, never a half-written one.

The File Concept

Open-File Tables

Process A · fd table0 stdin1 stdout2 stderr3Process B (A's child)0 stdin1 stdout2 stderr34System-wide open-file tablepos = 4096O_RDONLYref = 2pos = 0O_RDWRref = 1In-memory i-node tablei-node 4711size = 6000ref = 2on-disk i-node + blocksopen()fork() shares
  • fd: an index into the kernel's per-process table; 0, 1, 2 are standard input/output/error, and open returns the lowest free number.
  • An open-file entry holds the position (offset) and access mode: fork and dup share the entry (a common position); a second open creates a new entry (a separate position).
  • The i-node is in memory once per file; further opens don't reread it from disk. The Windows equivalent is the handle table.

The File Concept · Step through the code

open, read, close, Step by Step

#include <fcntl.h>
#include <unistd.h>

int main(void) {
  char buf[4096];
  int fd = open("notes.txt", O_RDONLY);
  ssize_t n = read(fd, buf, sizeof buf);
  n = read(fd, buf, sizeof buf);
  n = read(fd, buf, sizeof buf);
  close(fd);
  return 0;
}
fd table0 tty1 tty2 tty3notes.txt (6000 bytes)0 – 40954096 – 5999

Initially only 0, 1, 2 are open (inherited from the shell: the terminal).

fd table0 tty1 tty2 tty3open-file entrypos = 0O_RDONLYref = 1i-node 4711size 6000ref = 1notes.txt (6000 bytes)0 – 40954096 – 5999position

open: the path is resolved, i-node 4711 is brought into memory, and an open-file entry with position 0 is set up; the lowest free number, 3, is returned.

fd table0 tty1 tty2 tty3open-file entrypos = 4096O_RDONLYref = 1i-node 4711size 6000ref = 1notes.txt (6000 bytes)0 – 40954096 – 5999position

First read: the block(s) are read/cached, 4096 bytes are copied to buf, and the position becomes 4096.

fd table0 tty1 tty2 tty3open-file entrypos = 6000O_RDONLYref = 1i-node 4711size 6000ref = 1notes.txt (6000 bytes)0 – 40954096 – 5999position

Second read: only 1904 bytes are left in the file; a short read is not an error. Position 6000.

fd table0 tty1 tty2 tty3open-file entrypos = 6000O_RDONLYref = 1i-node 4711size 6000ref = 1notes.txt (6000 bytes)0 – 40954096 – 5999position

The third read returns 0: end of file (EOF). On an error it would return −1 and set errno.

fd table0 tty1 tty2 tty3entry freedref = 0notes.txt (6000 bytes)0 – 40954096 – 5999

close: fd 3 is freed; the entry's reference count drops to 0 → it is deleted, and the i-node's to 0 → it can be evicted from memory.

The File Concept · Step through the code

Copying a File: copy abc xyz

#define BUF 4096

int copy(const char *src, const char *dst) {
  char buf[BUF];
  int in  = open(src, O_RDONLY);
  int out = open(dst, O_WRONLY | O_CREAT
                      | O_TRUNC, 0644);
  ssize_t n;
  while ((n = read(in, buf, BUF)) > 0)
    write(out, buf, n);
  close(in);
  close(out);
  return 0;
}
abc (10000 bytes)buffer(4096 B)xyz

in = 3; out = 4. O_TRUNC: if xyz exists its contents are erased; otherwise O_CREAT creates it with mode 0644.

abc (10000 bytes)buffer(4096 B)4096 bytesxyzread

First read: 4096 bytes into the buffer.

abc (10000 bytes)buffer(4096 B)4096 bytesxyzwrite

write writes the same 4096 bytes to xyz (to the page cache first; it reaches the disk later).

abc (10000 bytes)buffer(4096 B)4096 bytesxyzwrite

Second round: 8192 bytes copied.

abc (10000 bytes)buffer(4096 B)1808 bytesxyzwrite

The third read returns only 1808 bytes; write writes exactly n bytes (not BUF!).

abc (10000 bytes)buffer(4096 B)xyz

read returns 0 → the loop ends; both descriptors are closed. In total 4 read + 3 write calls.

Real code must check every call for a −1 return and complete a short write in a loop; with O_CREAT the third argument (the mode) must be given.

The File Concept

Access Methods

Sequential access0123456current positionrewindread next · write nexttape, logs, streamsDirect (random) access0123456read 5lseek(fd, 5·B, SEEK_SET)disks, databases, mmapIndexed accessindexAdaKaanSelinZeyneprecord blocksblock 1block 4block 7block 10key → index → blockISAM, B+ trees
  • Sequential: from start to end in order; rewinding instead of jumping. Tapes and cassettes worked this way; it is still the most common pattern for logs and streams.
  • Direct: the file is a sequence of numbered blocks/records; you start where you like (lseek, pread). Disks and SSDs support this.
  • Indexed: a small index (and if needed an index of the index) finds the right block by key; it is built on top of direct access.
  • With memory-mapped access (mmap) the file's pages are mapped into the address space; reads and writes are ordinary memory accesses.

Directories

Directory Operations

CallFunction
mkdircreates an empty directory (containing only . and ..)
rmdirdeletes a directory; it must be empty
opendir / closediropens / closes a directory stream
readdirreturns the next entry (name, i-node no., type) in the open directory
renamerenames or moves a file or directory (atomic within one file system)
linkadds a new name (a hard link) for an existing file; the i-node's link count goes up
symlinkcreates a symbolic link containing the target path
unlinkremoves a name; the file is deleted when the link count reaches 0
  • A directory is a file too: its contents are (name → file) mappings; but users can change it only through these calls, not with write.
  • link doesn't attach a directory to another directory; it gives a file a second name; hard links to directories are forbidden to users (to prevent cycles).
/home (i-node 20).20..2ali31ali (i-node 31).31..20link counts: 20 → 3 (new ..), 31 → 2 (ali and .)

Directories · Step by step

Directory Structures

single directory (root)catboatestdatamailconthexrecordsAll users' files are in one place: two users can't both have a file named "test",and files can't be grouped. Early systems (CDC 6600) and some embedded devices worked this way.

Single-level: simple, but has naming and grouping problems.

master file directory (MFD)aliayşecanececatbotestadatatestxdataaA user file directory (UFD) per user: no name clashes (ali/test ≠ can/test), but sharing and grouping are hard.

Two-level: path = user/file. System programs need a separate "user" directory; the idea of a search path comes from here.

/binhomeusraliayşeliblsnotes.txthwphoto.pngproject.c

Tree (hierarchical): users create as many subdirectories as they like; every file has a single absolute path. The basis of all general-purpose systems today.

/binhomeusraliayşeliblsnotes.txthwphoto.pngsharedproject.cone file, two names

Acyclic graph (DAG): project.c appears under both ali/hw and ayşe/shared (a link). Problems: a traversal may count it twice, and deletion becomes ambiguous.

/binhomeusraliayşeliblsnotes.txthwphoto.pngsharedproject.cshared/back → homecycle!

General graph: linking a directory creates a cycle; searches may never end, and unreachable cycles need garbage collection. That's why UNIX forbids hard links to directories.

Directories

Path Names

  • An absolute path starts at the root: /home/sercan/os/slides. Windows: C:\Users\sercan\os; a network share \\server\share.
  • A relative path starts at the working directory (pwd, cd): from /home/sercan/course, ../os/slides.
  • . is the current directory, .. the parent; at the root, .. leads back to the root.
  • The separator is / on UNIX and \ on Windows (the API accepts / too).
/binetchomeusrdevsercanlibcourseosslidespasswdworking directory../os/slides

In historical UNIX, user directories were under /usr; today they are under /home (/Users on macOS), and /usr is for read-only system software.

Directories · Step by step

Path Resolution, Step by Step

root directory1.1..4bin7dev14lib9etc6usr8tmp

Absolute path: the root directory's i-node is known (1 in this example; 2 in V7 UNIX and ext4). usr is looked up in the root directory → i-node 6.

root directory1.1..4bin7dev14lib9etc6usr8tmpi-node 6mode, size,times …→ block 132

i-node 6 is read: /usr is a directory whose contents are in block 132. (Permissions are checked here too: x on the directory is required.)

root directory1.1..4bin7dev14lib9etc6usr8tmpi-node 6mode, size,times …→ block 132block 132: /usr6.1..19dick30erik51jim26ast45bal

Block 132 is read and ast is looked up → i-node 26.

root directory1.1..4bin7dev14lib9etc6usr8tmpi-node 6mode, size,times …→ block 132block 132: /usr6.1..19dick30erik51jim26ast45bali-node 26mode, size,times …→ block 406

i-node 26: the contents of the /usr/ast directory are in block 406.

root directory1.1..4bin7dev14lib9etc6usr8tmpi-node 6mode, size,times …→ block 132block 132: /usr6.1..19dick30erik51jim26ast45bali-node 26mode, size,times …→ block 406block 406: /usr/ast26.6..64grants92books60mbox81minix17srci-node 60/usr/ast/mbox

In block 406, mbox → i-node 60. Without caching that's 5 disk reads; that's why Linux keeps name → i-node mappings in the dentry cache.

The example is Tanenbaum's figure: /usr/ast/mbox is looked up. For a relative path the search starts at the working directory's i-node instead of the root.

Directories · Step by step

Hard and Symbolic Links

directory /home/alia.txt812i-node 812type: regular filelink count = 1"hello\n"

echo hello > a.txt: directory entry a.txt → i-node 812, link count 1.

directory /home/alia.txt812b.txt812i-node 812type: regular filelink count = 2"hello\n"

ln a.txt b.txt (a hard link): the new entry points to the same i-node; link count 2. The two names are equal; neither is the "original".

directory /home/alia.txt812b.txt812c.txt905i-node 812type: regular filelink count = 2"hello\n"i-node 905type: symbolic link"a.txt"resolved again through the name

ln -s a.txt c.txt (a symbolic link): a small file with its own i-node (905), whose contents are the target's path.

directory /home/alia.txtremoved (rm)b.txt812c.txt905i-node 812type: regular filelink count = 1"hello\n"i-node 905type: symbolic link"a.txt"a.txt gone → ENOENT (dangling link)

rm a.txt: link count 1 → the data stays, and b.txt works. c.txt dangles. (Tanenbaum: if owner C deletes the file, B's link remains but C still owns the i-node, so it counts against C's quota.)

Hard links: within one file system, not to directories, no extra cost. Symbolic links: can point to directories, files on other file systems, even network paths; each access needs extra path resolution, and the kernel limits loops (ELOOP). Windows: mklink /H, mklink, junctions.

Directories · Step by step

Mounting

root file system (/dev/nvme0n1p2, ext4)/binhomemntusbold.txtUSB stick (/dev/sdb1, exFAT)rootphotosdocsnot in the tree yet

Two separate file systems. /mnt/usb is an ordinary directory containing old.txt.

root file system (/dev/nvme0n1p2, ext4)/binhomemntusbphotosdocsthe USB root took /mnt/usb's place/mnt/usb/photos → photos on the USBold.txt temporarily hidden

mount /dev/sdb1 /mnt/usb: the kernel adds an entry to the mount table; when path resolution reaches /mnt/usb it jumps to the USB's root i-node. The old contents are hidden, not deleted.

root file system (/dev/nvme0n1p2, ext4)/binhomemntusbold.txtUSB stick (/dev/sdb1, exFAT)rootphotosdocsdetached (umount)

umount /mnt/usb: detached if no files are open (otherwise EBUSY); old.txt is visible again.

At boot the root file system is mounted, the others via /etc/fstab or systemd units; findmnt shows the tree. Windows gives each volume a drive letter (C:, D:), but NTFS can also mount on an empty folder. Crossing a mount point applies permissions and options such as nosuid, noexec, ro.

Protection

Access Rights: rwx

setuid1×4setgid0×2sticky0×1r1×4w1×2x1×1r1×4w0×2x1×1r1×4w0×2x1×1special bits= 4owner (u)= 7group (g)= 5other (o)= 5example: chmod 4755 → -rwsr-xr-x (setuid + rwx r-x r-x)
RightOn a fileOn a directory
rread the contentslist the entries (ls)
wchange the contentsadd/delete/rename entries (the file's own permission doesn't matter!)
xrun as a programpass through (path resolution, cd)
  • Order of checks: owner? → owner rights; if not, in the group? → group rights; otherwise, other. The first match is used.
  • setuid: the program runs with the identity of the file's owner (passwd). sticky on a directory: anyone can write, but only the owner can delete (/tmp: drwxrwxrwt). Example drwxrwx--T: sticky on, no x for others → a capital T.
  • Owner/group/other is coarse-grained; for named users and groups an access-control list (ACL) is used, shown as + in ls output.

Disk Layout

Disk Partitioning: MBR and GPT

MBR (BIOS, 1983)LBA 0512 Bcode 446 Bboot loader4 × 16 Bpartition table55 AAPartition 1 (active)Partition 2extended → logicalat most 4 primary partitions; 32-bit LBA × 512 B ⇒ a 2 TiB limitGPT (UEFI, 2000s)LBA 0protective MBRLBA 1GPT headerLBA 2–33128 entries × 128 BESPFAT32, 100+ MiBWindowsNTFSLinuxext4 / Btrfsbackup entrieslast LBAsbackup headerlast LBA64-bit LBA, 128 partitions, CRC32 checksums, a backup at the end of the disk; each partition has a GUID and a type GUID
  • A disk is divided into one or more partitions; each can hold a separate file system (or swap space, LVM, an encrypted container).
  • The partition table is at the end of the MBR: each entry holds a start and a length; the BIOS boots only from the partition marked active.
  • GPT is part of the UEFI standard; it is required for disks over 2 TiB and for Windows 11. The protective MBR stops old tools from treating the disk as "empty".

Disk Layout · Step by step

Booting: BIOS/MBR and UEFI/GPT

BIOS + MBR (legacy)UEFI + GPT (today)BIOS / POSTUEFI / POSTMBR code (446 B)NVRAM bootentrythe active partition'sboot blockESP: \EFI\…\*.efiboot loader(GRUB, NTLDR)GRUB or WindowsBoot Managerkernelkernel +initramfs

On power-up the hardware tests itself (POST). The BIOS runs in 16-bit real mode; UEFI in 32/64 bits, with its own drivers and file-system support.

BIOS + MBR (legacy)UEFI + GPT (today)BIOS / POSTUEFI / POSTMBR code (446 B)NVRAM bootentrythe active partition'sboot blockESP: \EFI\…\*.efiboot loader(GRUB, NTLDR)GRUB or WindowsBoot Managerkernelkernel +initramfs

The BIOS loads the disk's first sector (the MBR) into memory and runs it. UEFI instead reads the boot entries in NVRAM (efibootmgr); it looks for a file, not a sector.

BIOS + MBR (legacy)UEFI + GPT (today)BIOS / POSTUEFI / POSTMBR code (446 B)NVRAM bootentrythe active partition'sboot blockESP: \EFI\…\*.efiboot loader(GRUB, NTLDR)GRUB or WindowsBoot Managerkernelkernel +initramfs

The MBR code finds the active partition in the partition table and runs its first block (the boot block). UEFI loads the .efi program from the ESP (FAT32); the active flag isn't used.

BIOS + MBR (legacy)UEFI + GPT (today)BIOS / POSTUEFI / POSTMBR code (446 B)NVRAM bootentrythe active partition'sboot blockESP: \EFI\…\*.efiboot loader(GRUB, NTLDR)GRUB or WindowsBoot Managerkernelkernel +initramfs

In both layouts the boot loader then loads and runs the kernel; the only difference is how the boot loader is reached.

PCs have shipped with UEFI since around 2012; after 2020 the BIOS compatibility mode (CSM) was dropped from most new systems.

Disk Layout

File-System Layout

DiskMBR / GPTpartition 1partition 2partition 3bootblocksuperblocksize, block countfree spacebitmap / listi-nodes1 per fileroot directoryfiles and directoriesdata blocksext4: the partition is divided into block groups (128 MiB each, 32768 blocks with 4 KiB blocks)group 0group 1group 2…group nsuperblock*group descriptorblock bitmapi-node bitmapi-node tabledata blocks* backups of the superblock and descriptors are kept in some groups
  • Superblock: the file system's identity (magic number), block size, block and i-node counts, free counters, mount state. It is the first structure read when mounting.
  • The boot block is the first block of the partition (the disk's first block is the MBR); space is reserved for it in every partition even if it is empty.
  • ext4 places related i-nodes and their data in the same group to reduce head movement (the cylinder-group idea of BSD FFS).

Disk Layout · Step-by-step calculation

From i-node Number to Disk Address

disk blocks (4 KiB)Ssuper0iBi-bitm.1dBd-bitm.20–31332–63464–95596–1276128–1597160–1918192–2239224–25510data11data12data13data14data15…i-node table: 8 blocks × 32 = 256 i-nodes

Example layout: 4 KiB blocks, 128 B i-nodes, i.e., 32 i-nodes per block. Block 0 is the superblock, 1–2 the bitmaps, 3–10 the i-node table. Question: where on disk is i-node 77 (numbered from 0)?

disk blocks (4 KiB)Ssuper0iBi-bitm.1dBd-bitm.20–31332–63464–95596–1276128–1597160–1918192–2239224–25510data11data12data13data14data15…i-node table: 8 blocks × 32 = 256 i-nodestable block 2 (from 0): i-nodes 64–95, 128 B each77016644096 B

Byte offset from the start of the table: 77 × 128 = 9856 = 2 × 4096 + 1664. So block 2 of the table, byte 1664 within it (that block's 13th i-node).

disk blocks (4 KiB)Ssuper0iBi-bitm.1dBd-bitm.20–31332–63464–95596–1276128–1597160–1918192–2239224–25510data11data12data13data14data15…i-node table: 8 blocks × 32 = 256 i-nodestable block 2 (from 0): i-nodes 64–95, 128 B each77016644096 B

Disk block = table start + 2 = 3 + 2 = block 5. That's why the superblock's "where is the i-node table" field is needed.

disk blocks (4 KiB)Ssuper0iBi-bitm.1dBd-bitm.20–31332–63464–95596–1276128–1597160–1918192–2239224–25510data11data12data13data14data15…i-node table: 8 blocks × 32 = 256 i-nodestable block 2 (from 0): i-nodes 64–95, 128 B each77016644096 Bsector 40sector 41sector 42sector 43sector 44sector 45sector 46sector 47

The disk reads sectors: byte address 5 × 4096 + 1664 = 22,144 → 22,144 / 512 = sector 43, byte 128 within it. A sector holds 4 i-nodes.

In ext4 numbers start at 1 and the table is split across block groups: group = (n − 1) div g, index = (n − 1) mod g. With g = 8192 i-nodes/group and 256 B i-nodes, i-node 20,000 → group 2, index 3615 → block 225 of the group's table, offset 3840.

Allocation Methods · Step by step

Allocation Methods: One File, Four Ways

disk (32 blocks)01234567809110211312413141516171819202122232425262728293031directory entrynamejeepstart9length5block i = start + ione seek, sequential read

Contiguous: 5 blocks side by side (9–13). The directory entry holds only the start and length.

disk (32 blocks)0→1012345678→169→25101112131415→1161718192021222324→−125262728293031directory entrynamejeepfirst9last25each block carries the number of the next9 → 16 → 1 → 10 → 25 → −1

Linked list: the blocks are scattered over the disk (9, 16, 1, 10, 25); each points to the next.

disk (32 blocks)02123456780931011121314151161718192021222324425262728293031directory entrynamejeepfirst block9size5 blocksFAT (in memory)110916102516125EOFchain in the table

FAT: the same chain, but the pointers live in a table in memory, not in the blocks; the blocks are all data.

disk (32 blocks)02123456780931011121314151161718ind192021222324425262728293031directory entrynamejeepindex block19size5 blocksindex block 1991611025−1−1entry i = address of block i

Indexed: all addresses are in a single index block (19); block i is accessed directly.

Allocation Methods · Step by step

Contiguous Allocation

A0A1A2A3B4B5B6C7C8C9C10C11C12D13D14D15D16D17E18E19E20E21E22E23E24E25E26E27E28E29F30F31F32F33F34F35G36G37G38G397 files: A4 · B3 · C6 · D5 · E12 · F6 · G4 = 40 blocks, disk full

Each file was placed in consecutive blocks; directory entry = (start, length).

A0A1A2A3B4B5B6C7C8C9C10C11C121314151617E18E19E20E21E22E23E24E25E26E27E28E29303132333435G36G37G38G39D and F deleted: two holes of 5 and 6 blocks (11 free blocks in total)

Deleted files leave holes; free space is broken into pieces.

A0A1A2A3B4B5B6C7C8C9C10C11C121314151617E18E19E20E21E22E23E24E25E26E27E28E29303132333435G36G37G38G39H wants 8 blocks: 11 free in total, but the largest hole is 6 → it doesn't fit!external fragmentation

The new file fits no hole: the same external fragmentation as in memory management. Worse, the file's final size must be known up front.

A0A1A2A3B4B5B6C7C8C9C10C11C12E13E14E15E16E17E18E19E20E21E22E23E24G25G26G27G28H29H30H31H32H33H34H35H36373839compaction: G and E moved, one 11-block hole → H placedthe cost: rewriting 16 blocks on disk

Remedies: defragmentation/compaction (expensive), or splitting the file into several contiguous pieces, extents.

Pros

Simple; block i = start + i, so random access is easy; the whole file is read in one seek, back to back (the best HDD performance).

Cons

External fragmentation; files can't grow or must be moved. Suited to write-once media: CD-ROM/DVD (ISO 9660, UDF).

Allocation Methods · Step by step

Linked Allocation

File Ablock 0data7physical 4block 1data2physical 7block 2data10physical 2block 3data12physical 10block 4data−1physical 12File Bblock 0data3physical 6block 1data11physical 3block 2data14physical 11block 3data−1physical 14

The directory entry knows the first block (4 for A) (and the last one too in Silberschatz's layout). The first/last few bytes of each block hold the next block's number.

File Ablock 0data7physical 4block 1data2physical 7block 2data10physical 2block 3data12physical 10block 4data−1physical 12File Bblock 0data3physical 6block 1data11physical 3block 2data14physical 11block 3data−1physical 14read so far: 1 block

We want logical block 3: first physical 4 is read → next is 7.

File Ablock 0data7physical 4block 1data2physical 7block 2data10physical 2block 3data12physical 10block 4data−1physical 12File Bblock 0data3physical 6block 1data11physical 3block 2data14physical 11block 3data−1physical 14read so far: 2 blocks

Physical 7 is read → next 2.

File Ablock 0data7physical 4block 1data2physical 7block 2data10physical 2block 3data12physical 10block 4data−1physical 12File Bblock 0data3physical 6block 1data11physical 3block 2data14physical 11block 3data−1physical 14read so far: 3 blocks

Physical 2 is read → next 10.

File Ablock 0data7physical 4block 1data2physical 7block 2data10physical 2block 3data12physical 10block 4data−1physical 12File Bblock 0data3physical 6block 1data11physical 3block 2data14physical 11block 3data−1physical 14read so far: 4 blocks

Physical 10 = logical 3: 4 disk reads. Random access is O(n); on an HDD each may be a separate seek.

  • No external fragmentation, and files grow easily; internal fragmentation (in the last block) exists with every method.
  • The data per block isn't a power of two (4096 − 4 bytes); if one pointer is corrupted, the rest of the chain is lost.

Allocation Methods · Step by step

FAT: Allocation with a Table

FAT (in memory): entry i = the block after block i0rsvd1rsvd210311475free63728free9free1012111412EOF13free14EOF15freeA: start 4B: start 6

A: 4 → 7 → 2 → 10 → 12; B: 6 → 3 → 11 → 14.

FAT (in memory): entry i = the block after block i0rsvd1rsvd210311475free63728free9free1012111412EOF13free14EOF15freeA: start 4B: start 6A chain: 4

Entry 4 = 7.

FAT (in memory): entry i = the block after block i0rsvd1rsvd210311475free63728free9free1012111412EOF13free14EOF15freeA: start 4B: start 6A chain: 4 → 7

Entry 7 = 2.

FAT (in memory): entry i = the block after block i0rsvd1rsvd210311475free63728free9free1012111412EOF13free14EOF15freeA: start 4B: start 6A chain: 4 → 7 → 2

Entry 2 = 10.

FAT (in memory): entry i = the block after block i0rsvd1rsvd210311475free63728free9free1012111412EOF13free14EOF15freeA: start 4B: start 6A chain: 4 → 7 → 2 → 10

Entry 10 = 12.

FAT (in memory): entry i = the block after block i0rsvd1rsvd210311475free63728free9free1012111412EOF13free14EOF15freeA: start 4B: start 6A chain: 4 → 7 → 2 → 10 → 12

Entry 12 = EOF. The chain was followed in memory: no disk reads.

  • The pointers of the linked list are taken out of the blocks and moved into one file-allocation table (FAT); the blocks are left entirely for data.
  • Random access still follows the chain, but in memory: fast. Two copies of the table are kept on disk.
  • The problem: the table grows with the disk, and all of it must be in memory.
Example: a 200 GiB disk with 1 KiB blocks → 200 × 220 = 209,715,200 entries; at 4 B per entry → 800 MiB (600 MiB at 3 B). In FAT32 a 2 TiB volume with 32 KiB clusters → 67,108,864 entries = 256 MiB.

Allocation Methods · Step through the code

Following a FAT Chain: Code Trace

int find_block(int first, long off) {
  int b = first;
  long n = off / BLOCK;  /* block index */
  while (n > 0 && b != END) {
    b = fat[b];          /* follow chain */
    n--;
  }
  return b;          /* END: past EOF */
}
FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 4blue: file blocks · green: followed · orange: b

find_block(4, 3500): file A starts at block 4. 3500 / 1024 = 3: we want the file's block 3 (counting from 0), so the chain must advance 3 steps.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 4 → 7blue: file blocks · green: followed · orange: b

fat[4] = 7: next block 7; n = 2.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 4 → 7 → 2blue: file blocks · green: followed · orange: b

fat[7] = 2: next block 2; n = 1.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 4 → 7 → 2 → 10blue: file blocks · green: followed · orange: b

fat[2] = 10: next block 10; n = 0.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 4 → 7 → 2 → 10blue: file blocks · green: followed · orange: b

n = 0: the loop ends at disk block 10; byte 3500 is byte 3500 − 3·1024 = 428 of this block. All three steps were in memory: no disk reads.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 6blue: file blocks · green: followed · orange: b

find_block(6, 5000): file B has 4 blocks (at most 4096 bytes), but 5000 / 1024 = block 4 is requested.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 6 → 3blue: file blocks · green: followed · orange: b

fat[6] = 3: next block 3; n = 3.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 6 → 3 → 11blue: file blocks · green: followed · orange: b

fat[3] = 11: next block 11; n = 2.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 6 → 3 → 11 → 14blue: file blocks · green: followed · orange: b

fat[11] = 14: next block 14; n = 1.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 6 → 3 → 11 → 14blue: file blocks · green: followed · orange: b

fat[14] = END: the chain is over and the end of the file reached; n = 0.

FAT (bellekte) · A: 4→7→2→10→12 · B: 6→3→11→140·1·210311475·63728·9·1012111412END13·14END15·followed: 6 → 3 → 11 → 14blue: file blocks · green: followed · orange: b

At the fourth step b = END: the file has no block 4, and the offset lies past the file. read returns 0 bytes (end of file); write instead allocates a new block and links it into entry 14.

Allocation Methods

The FAT Family: FAT12/16/32 and exFAT

MS-DOS directory entry (32 bytes)Name8 BExt.3 BAttr1 BReserved10 BTime2 BDate2 BCluster2 BSize4 BAttr: attributes. FAT32 put creation/access times and the upper 16 bits of the first cluster into the "reserved" area; long names live in extra entries.
FAT12FAT16FAT32exFAT
FAT entry12 bit16 bit28 bits used32 bit
Largest volume≈ 16 MiB2 GiB (32 KiB clusters)2 TiB (512 B sectors)128 PiB
Largest filesize of the volume2 GiB4 GiB − 1 bytes16 EiB (theoretical)
Free space0 in the FAT0 in the FAT0 in the FAT + FSInfo hintseparate bitmap
Used forfloppiesold disksUSB, UEFI ESPSDXC/SDUC, large USB drives

The size field is 32 bits, so FAT32 has a 4 GiB file-size limit; exFAT removes it and may skip the FAT chain entirely for contiguous files. Windows' graphical format tool has historically limited FAT32 to 32 GB; that's a limit of the tool, not the file system.

Allocation Methods

Indexed Allocation and the i-node

  • Each file's block addresses are gathered in an index block (on UNIX, the i-node): entry i is the address of block i. Fast random access, no external fragmentation.
  • An i-node is in memory only while the file is open: K open files × i-node size. Proportional to the number of open files, not to the disk size as with FAT.
  • An i-node has a fixed size; for very large files the last entries point to further index blocks (indirect blocks).
  • The cost: even a small file uses index space; a large file needs indirect-block reads (caching reduces this).
index block 1991611025−1−1…02123456780931011121314151161718ind192021222324425262728293031the i-node holds this block's address

Allocation Methods

i-node Structure: Indirect Blocks

i-nodemode (type, permissions)UID, GIDsize, block counttimes (a/m/c)link countdirect 0…direct 11single indirectdouble indirecttriple indirectdatadatasingledatadatadoublesingledatadatatripledoublesingledata4 KiB blocks, 4 B pointers:1024 addresses per block12 direct48 KiBsingle indirect+ 4 MiBdouble indirect+ 4 GiBtriple indirect+ 4 TiBeach level ×1024

The ext2/ext3 and classic UNIX (V7: 10 direct) structure. Small files use only the direct pointers and never allocate an indirect block; as a file grows, the tree deepens.

Allocation Methods · Step-by-step calculation

Maximum File Size

Assumptions: 4 KiB blocks = 4096 B, 4 B pointers → an indirect block holds 4096 / 4 = 1024 addresses; the i-node has 12 direct, 1 single, 1 double, 1 triple indirect pointer.

Direct: 12 × 4 KiB = 48 KiB (49,152 B) · Single indirect: 1024 × 4 KiB = 4 MiB

The first 12 blocks come straight from the i-node; the next 1024 from one index block.

Double indirect: 1024² × 4 KiB = 1,048,576 blocks = 4 GiB

Each of 1024 index blocks points to 1024 data blocks.

Triple indirect: 1024³ × 4 KiB = 1,073,741,824 blocks = 4 TiB

One more level: ×1024.

Total: 12 + 1024 + 1024² + 1024³ = 1,074,791,436 blocks × 4096 = 4,402,345,721,856 B ≈ 4.004 TiB

Nearly all of it comes from the triple indirect block.

Tanenbaum's example (1 KiB blocks, 256 addresses, 10 direct): 16,843,018 blocks ≈ 16 GiB. ext2/ext3 with 4 KiB blocks actually stop at 2 TiB (32-bit counter × 512 B); ext4 extents reach 16 TiB.

Allocation Methods · Step by step

From Offset to Disk Block

offset → logical block40,000 Bblock 9 · direct1,000,000 Bblock 244 · single indirect5,000,000 Bblock 1,220 · double indirect6,000,000,000 Bblock 1,464,843 · triple indirecti-nodedirect 0–11single indirectdouble indirecttriple indirectaddress in the i-nodedataoffset in block = 40,000 mod 4096 = 3,136disk reads (i-node in memory): 1

40,000 / 4096 = 9 → below 12: direct pointer 9. One read.

offset → logical block40,000 Bblock 9 · direct1,000,000 Bblock 244 · single indirect5,000,000 Bblock 1,220 · double indirect6,000,000,000 Bblock 1,464,843 · triple indirecti-nodedirect 0–11single indirectdouble indirecttriple indirect[232]indexdataoffset in block = 1,000,000 mod 4096 = 576disk reads (i-node in memory): 2

1,000,000 / 4096 = 244; 244 − 12 = 232 < 1024 → entry 232 of the single indirect block. 2 reads.

offset → logical block40,000 Bblock 9 · direct1,000,000 Bblock 244 · single indirect5,000,000 Bblock 1,220 · double indirect6,000,000,000 Bblock 1,464,843 · triple indirecti-nodedirect 0–11single indirectdouble indirecttriple indirect[0]index[184]indexdataoffset in block = 5,000,000 mod 4096 = 2,880disk reads (i-node in memory): 3

5,000,000 / 4096 = 1220; 1220 − 12 − 1024 = 184 < 1024² → double indirect: outer index 184 div 1024 = 0, inner index 184. 3 reads.

offset → logical block40,000 Bblock 9 · direct1,000,000 Bblock 244 · single indirect5,000,000 Bblock 1,220 · double indirect6,000,000,000 Bblock 1,464,843 · triple indirecti-nodedirect 0–11single indirectdouble indirecttriple indirect[0]index[405]index[511]indexdataoffset in block = 6,000,000,000 mod 4096 = 3,072disk reads (i-node in memory): 4

6 GB / 4096 = 1,464,843; minus 12 + 1024 + 1024² leaves 415,231 → triple indirect: 0, 405, 511. 4 reads; the cache holds the first levels.

Allocation Methods

ext4: Extents

i_block inside the i-node (60 bytes)extent header · 0xF30A · entries 3 / 4 · depth 0logical 0 · length 1,000 · physical 34,000logical 1,000 · length 32,768 · physical 120,000logical 33,768 · length 200 · physical 9,100(empty entry)disk0160,0003 extents × 12 B = 36 B, no extra blockblock map: 33,968 pointers = 35 index blocks (140 KiB)extent ≤ 32,768 blocks = 128 MiB
  • An extent = (logical start, length, physical start): a run of contiguous blocks described by one record. 48-bit physical addresses → 1 EiB volumes; files up to 16 TiB.
  • Four extents fit in the i-node; if more are needed, the entries point to index nodes of an extent tree (depth > 0).
  • Delayed allocation and the multiblock allocator choose large contiguous areas when writing data to disk. XFS, Btrfs, NTFS (run lists), and APFS use extents too.

Allocation Methods

NTFS: The Master File Table (MFT)

MFT (1 KiB records)0$MFT1$MFTMirr2$LogFile3$Volume5. (root directory)6$Bitmap8$BadClus9$Secure……≥ 24user filesone file's MFT recordheader$STANDARD_INFORMATION$FILE_NAME$DATA (run list)times, attribute flagsname, parent directoryrun: (start, length)20, 464, 280, 3cluster 0cluster 100in a small file the data is inside the record: a resident $DATA
  • In NTFS everything is a file, including the MFT itself (record 0). Every file is at least one 1 KiB record whose contents are a list of attributes.
  • Large data is kept as a run list (extents); directories are B+ trees ($INDEX_ROOT, $INDEX_ALLOCATION). $LogFile is the metadata journal.
  • Extra features: hard links, reparse points, alternate data streams (ADS), compression, EFS encryption, quotas. Servers also have ReFS (CoW, checksums).

Allocation Methods

Comparing File Systems

File systemLargest fileLargest volumeCrash consistencyTypical use
FAT324 GiB − 1 B2 TiBnone (chkdsk/fsck)USB, UEFI ESP
exFAT16 EiB*128 PiBnoneSD cards, external disks
NTFS16 EiB* (Windows: 256 TiB–8 PiB)256 TiB–8 PiBmetadata journalWindows
ext416 TiB (4 KiB blocks)1 EiBjournal (jbd2)Linux, Android
XFS8 EiB8 EiBmetadata journalRHEL, large servers
Btrfs16 EiB16 EiBCoW + checksumsFedora, openSUSE, NAS
ZFS16 EiB2¹²⁸ B (theoretical)CoW + checksumsFreeBSD, TrueNAS, servers
APFS8 EiB8 EiBCoW, snapshotsmacOS, iOS

* theoretical limit; the OS and cluster size impose lower ones. Since Windows 10 (1709), NTFS reaches 8 PiB with 2 MiB clusters. HFS+ gave way to APFS in 2017; ext3 to ext4.

Implementing Directories

Directory Entries and Long Names

(a) attributes in the entry (MS-DOS)gamesattributes + disk addressmailattributes + disk addressnewsattributes + disk addressworkattributes + disk address(b) attributes in the i-node (UNIX)games14i-nodemail7i-nodenews29i-nodework51i-node(c) variable length, inlinelengthi-node · typeproject-budgetlengthi-node · typepersonnellengthi-node · typefooext4 entry: i-node, rec_len, name length, type, name(d) fixed entries + a name heapname pointerattributesname pointerattributesname pointerattributesheapproject-budgetpersonnelfooentries aligned; names at the end of the directory block

To locate a directory it suffices to know the first block's address (contiguous), the first block's number (linked list/FAT), or the i-node number (UNIX). With long names, fixed entries like (a)/(b) waste space; (c) leaves holes on deletion and a name may cross a page boundary; (d) requires managing the heap.

Implementing Directories · Step by step

Directory Lookup: List, Hash, Tree

linear list...notes.txtphoto.pnghwmakefilereport.pdfthesis.docx← 7 comparisons

Linear list: simple, but lookup is O(n). Inserting first checks that the name doesn't exist; deleting marks the entry free or moves the last entry into it.

linear list...notes.txtphoto.pnghwmakefilereport.pdfthesis.docxhash table (8 buckets)01234567h("report.pdf") mod 8 = 5report.pdfnotes.txt

Hash table: the name's hash selects the bucket; lookup is O(1) on average.

linear list...notes.txtphoto.pnghwmakefilereport.pdfthesis.docxhash table (8 buckets)01234567h("report.pdf") mod 8 = 5report.pdfthesis.docxh("thesis.docx") mod 8 = 5 → collisionat most 2 comparisons in the chainnotes.txt

Collisions are resolved by chaining. A fixed number of buckets is a problem for a growing directory; that's why modern systems use trees.

For large directories ext4 uses an htree ordered by name hash; NTFS, XFS, and Btrfs use B+ trees: sorted listing and O(log n) lookup together. Linux also keeps recently used name → i-node mappings in the dentry cache.

Free-Space Management

Choosing the Block Size

024681012140%25%50%75%100%1248163264128block size (KiB, logarithmic)MB/sdata rate (MB/s)space efficiency (%)0,4512,8
  • Model (Tanenbaum): all files are 4 KiB; average seek 5 ms, rotation 8.33 ms, a 1 MB track. Time = 5 + 4,165 + (b / 1 MB) × 8.33 ms. Computed with Python.
  • Larger blocks → better data rate, worse space utilization; smaller blocks the opposite.
  • Measurements show that most files are small while most space is taken by a few large files. Today's default is 4 KiB (the page size); large files are served by extents and cluster options.

Free-Space Management · Step by step

Free-Space Management: Four Methods

disk: green = free012345678910111213141516171819202122232425262728293031free blocks:2–5, 8–13, 17–18, 25–2715 free blocks / 32

Example disk: 32 blocks, 15 free (the Silberschatz example).

disk: green = free012345678910111213141516171819202122232425262728293031bitmap (1 = free)blocks 0–700111100blocks 8–1511111100blocks 16–2301100000blocks 24–310111000032 blocks → 32 bits = 4 bytesfirst free: the first 1 bit in the first nonzero word

Bitmap: 1 bit per block. Fixed and small; finding contiguous free blocks is easy. It must be kept in memory.

disk: green = free01→32→43→54→8567→98→109→1110→1211→1312→1713141516→1817→2518192021222324→2625→2726−12728293031head pointer → 2each free block points to the nextno extra space needed (the information is inside the free blocks)but walking the list = I/O for every block

Linked list: free blocks are linked together. Taking one free block is cheap; finding many contiguous blocks is expensive.

disk: green = free012345678910111213141516171819202122232425262728293031grouping (n = 4): the first free block holds n−1 addresses + the next groupblock 2345→ 8block 891011→ 12block 12131718→ 25block 252627→ −one read finds n−1 free blocks

Grouping: the first free block holds the addresses of the next n−1 free blocks and of the next group block (UNIX's free-list blocks).

disk: green = free012345678910111213141516171819202122232425262728293031counting: (first block, run length)(2, 4)blocks 2–5(8, 6)blocks 8–13(17, 2)blocks 17–18(25, 3)blocks 25–2715 free blocks → 4 pairsvery short if free space comes in contiguous runs

Counting: (start, length) pairs; the extent idea applied to free space. XFS keeps these in two B+ trees.

Free-Space Management · Step through the code

Allocating a Block with a Bitmap: Code Trace

int alloc_block(int hint) {
  for (int k = 0; k < N; k++) {
    int i = (hint + k) % N;
    if (bitmap[i] == 0) {  /* free? */
      bitmap[i] = 1;
      return i;
    }
  }
  return -1;               /* disk full */
}
bitmap: 1 = used, 0 = free10110213141506071819110011112013114115the file's blocks: —green: allocated to this file

The file ends at block 3 and needs 3 more blocks. For locality the search starts right after the last block: alloc_block(4).

bitmap: 1 = used, 0 = free10110213141506071819110011112013114115ithe file's blocks: —green: allocated to this file

k = 0: i = (4 + 0) mod 16 = 4; bit 1, block in use. Move on.

bitmap: 1 = used, 0 = free10110213141506071819110011112013114115ithe file's blocks: —green: allocated to this file

k = 1: i = (4 + 1) mod 16 = 5; bit 1, block in use. Move on.

bitmap: 1 = used, 0 = free10110213141516071819110011112013114115ithe file's blocks: 6green: allocated to this file

bitmap[6] = 0: free. The bit is set and block 6 is returned.

bitmap: 1 = used, 0 = free10110213141516171819110011112013114115ithe file's blocks: 6, 7green: allocated to this file

alloc_block(7): i = 7, bitmap[7] = 0: free. The bit is set and block 7 is returned. Consecutive blocks reduce head movement on sequential reads.

bitmap: 1 = used, 0 = free10110213141516171819110011112013114115ithe file's blocks: 6, 7green: allocated to this file

alloc_block(8): k = 0, i = 8; bit 1, in use.

bitmap: 1 = used, 0 = free10110213141516171819110011112013114115ithe file's blocks: 6, 7green: allocated to this file

k = 1: i = (8 + 1) mod 16 = 9; bit 1, block in use. Move on.

bitmap: 1 = used, 0 = free10110213141516171819110011112013114115ithe file's blocks: 6, 7green: allocated to this file

k = 2: i = (8 + 2) mod 16 = 10; bit 1, block in use. Move on.

bitmap: 1 = used, 0 = free10110213141516171819110111112013114115ithe file's blocks: 6, 7, 11green: allocated to this file

bitmap[11] = 0: free. The bit is set and block 11 is returned. The file got blocks 6, 7, 11: not contiguous, but close together.

Free-Space Management

Free Space: A Numerical Comparison

Tanenbaum's exampleA modern example
Disk / block500 GB, 1 KiB blocks1 TiB, 4 KiB blocks
Number of blocks500 × 10⁹ / 1024 ≈ 488.3 million2⁴⁰ / 2¹² = 268,435,456
Free-list block (32-bit numbers)1 KiB / 4 B = 256 → 255 addresses + 1 link4096 / 4 = 1024 → 1023 addresses + 1 link
List, with the disk entirely free488.3 M / 255 ≈ 1.91 million blocks268.4 M / 1023 ≈ 262,401 blocks (≈ 1 GiB)
Bitmap488.3 M bits / 8192 ≈ 59,605 blocks (≈ 60,000)2²⁸ bits = 32 MiB = 8192 blocks
  • The bitmap has a fixed size; the list is smaller only when the disk is nearly full (since it lists only free blocks).
  • Keeping one pointer block in memory is enough: write it to disk when full, read the next when empty. Tanenbaum's refinement: keep a half-full block in memory so temporary files don't cause constant disk I/O.
  • Real systems: ext4 has a bitmap per block group; XFS two B+ trees by position and by size (counting); ZFS an allocate/free log (space maps); Btrfs an extent tree.
Tanenbaum (500 GB, 1 KiB)bitmap59,605free list1,910,000Modern (1 TiB, 4 KiB)bitmap8,192free list262,401number of blocks, disk entirely free

Free-Space Management

Disk Quotas

open-file tableattributes, addressesattributes, addressesattributes, addressesattributes, addressesentry: user = aliquota pointerquota table (one record per user, ali)soft block limit9 GiBhard block limit10 GiBblocks used9.4 GiBwarnings left / time6 dayssoft file limit90,000hard file limit100,000files used41,250soft limit exceeded: warning + timer started
  • Every open-file entry points to its owner's quota record; on every new block or i-node allocation the counters go up and are compared with the limits.
  • The soft limit may be exceeded, but a warning is given and a grace period (e.g., 7 days) starts; when it expires, or the hard limit is reached, writes fail with EDQUOT.
  • Today: Linux quota tools (ext4, XFS project quotas), NTFS volume quotas, ZFS dataset and Btrfs subvolume (qgroup) quotas.

Performance · Step by step

How Many I/Os Does Creating a File Take?

open("/course/note.txt", O_RDONLY): file existsreadswrites1readrooti-node2readrootdata3readcoursei-node4readcoursedata5readnotei-node5 I/Os = 5 reads + 0 writes

The cache is empty; the root's i-node location is known from the superblock. The path is resolved component by component: for each directory, its i-node + data block, and finally the file's i-node. 5 reads; more for longer paths.

read(fd, buf, 4096): first blockreadswrites1readnotei-node2readnotedata[0]3writenotei-node3 I/Os = 2 reads + 1 writes

Each read finds the block address from the i-node, reads the block, and writes the access time. The noatime/relatime mount options usually eliminate that write.

creat("/course/note.txt"): no such filereadswrites1readrooti-node2readrootdata3readcoursei-node4readcoursedata5readi-bitm.search6writei-bitm.mark7writecoursedata8readnotei-node9writenotei-node10writecoursei-node10 I/Os = 6 reads + 4 writes

Creating is costlier: a free i-node is chosen from the bitmap, the new i-node initialized, an entry added to the directory, and the directory's i-node (size, times) updated. 10 I/Os.

write(fd, buf, 4096): needs a new blockreadswrites1readnotei-node2readd-bitm.search3writed-bitm.mark4writenotedata5writenotei-node5 I/Os = 2 reads + 3 writes

Every write that needs a new block: allocate a block from the data bitmap (read + write), write the data, write the address and size into the i-node. 5 I/Os, only 1 of them actual data.

creat + 3 × write(4096): 10 + 3 × 5 = 25 I/Osreadswrites3 thick-bordered boxes are user data; the other 22 I/Os are metadata (bitmaps, i-nodes, directory)cache: reads come from memory · delayed writes: each block is written once→ 8 writes remain: i-bitmap, d-bitmap, course data, course i-node, note i-node, 3 data blocks

Creating a file and writing 12 KiB without a cache costs 25 I/Os. That's why the buffer cache and delayed writes are indispensable; the order of writes also determines crash consistency.

Assumption: a simple file system with i-nodes and bitmaps, single-block directories, no cache. In real systems the dentry and i-node caches remove most reads; journaling and delayed writes also change the number and order of writes.

Performance

The Buffer Cache

hash table(device, block no.)01234block 12block 87block 6block 43block 8block 203block 19LRU list (oldest → newest)6 → 43 → 12 → 19 → 8 → 87 → 203lookup: O(1) via hashreplacement: the head of the LRU queuecritical blocks (i-nodes, directories)written at once if dirty
AccessLatency (2026, typical)
64 bytes from DRAM≈ 80–120 ns
NVMe SSD 4 KiB random read≈ 50–100 µs
HDD seek + rotation≈ 5–10 ms (100–280 MB/s sequential)
  • Write policy: write-through sends every write to disk at once (old MS-DOS, removable media); write-back accumulates dirty blocks. Linux writes back dirty pages older than 30 s in the background; applications force it with fsync.

Performance · Step through the code

The Buffer Cache: Code Trace

Buffer *bread(int blk) {
  Buffer *t = hash_find(blk);
  if (t != NULL) {         /* hit */
    lru_to_tail(t);
    return t;
  }
  t = lru_head();          /* oldest */
  if (t->dirty)
    disk_write(t);       /* write back */
  t->blk = blk; t->dirty = 0;
  disk_read(t);            /* miss: I/O */
  lru_to_tail(t);
  return t;
}
LRU list: oldest on the left, newest on the rightblock 5cleanblock 9dirtyblock 2cleandisk I/O: 0 reads, 0 writes

The cache holds 3 buffers: blocks 5, 9, 2 (9 is dirty: modified but not written to disk). Requests in order: 9, 7, 2, 4.

LRU list: oldest on the left, newest on the rightblock 5cleanblock 9dirtyblock 2cleandisk I/O: 0 reads, 0 writes

bread(9): found in the hash table, a hit. No disk I/O.

LRU list: oldest on the left, newest on the rightblock 5cleanblock 2cleanblock 9dirtydisk I/O: 0 reads, 0 writes

Block 9 moves to the tail of the list (newest) and is returned. Now block 5 is the oldest.

LRU list: oldest on the left, newest on the rightblock 5cleanblock 2cleanblock 9dirtydisk I/O: 0 reads, 0 writes

bread(7): not in the cache, a miss. To make room, the buffer at the head of the LRU list (block 5) is chosen.

LRU list: oldest on the left, newest on the rightblock 5cleanblock 2cleanblock 9dirtydisk I/O: 0 reads, 0 writes

Block 5 is clean: its copy on disk is current, so no write-back is needed.

LRU list: oldest on the left, newest on the rightblock 2cleanblock 9dirtyblock 7cleandisk I/O: 1 reads, 0 writes

The buffer is reused for block 7; the contents are read from disk (1 read) and it is added to the tail of the list.

LRU list: oldest on the left, newest on the rightblock 9dirtyblock 7cleanblock 2cleandisk I/O: 1 reads, 0 writes

bread(2): a hit; block 2 moves to newest. Dirty block 9 is now at the head of the list.

LRU list: oldest on the left, newest on the rightblock 9dirtyblock 7cleanblock 2cleandisk I/O: 1 reads, 0 writes

bread(4): a miss. The oldest buffer holds block 9.

LRU list: oldest on the left, newest on the rightblock 9dirtyblock 7cleanblock 2cleanblock 9 being written to disk · writes: 1

Block 9 is dirty: so the change isn't lost, the buffer is written to disk before reuse (1 write). This doubles the cost of the miss.

LRU list: oldest on the left, newest on the rightblock 7cleanblock 2cleanblock 4cleandisk I/O: 2 reads, 1 writes

Block 4 is read. 2 hits out of 4 requests; 2 reads and 1 write in total. Without a cache it would take 4 reads plus a separate write for block 9.

Performance · Step by step

The Page Cache and the Unified Buffer Cache

memory-mapped I/O (mmap)read() / write()page cachebuffer cachefile systemthe same data twice!

Not unified: mmap data is in the page cache, read/write data in the buffer cache, and it's copied between them (double caching): wasted memory and consistency problems.

memory-mapped I/O (mmap)read() / write()unified buffer (page) cachefile systemone copy, one replacement policy

Unified buffer cache: file pages live in one place, and both paths see the same pages. The page cache since Linux 2.4, the Windows cache manager, and Solaris all work this way.

The page cache shares the page-replacement algorithms of virtual memory. The exception: ZFS keeps its own ARC cache. O_DIRECT lets databases bypass the cache.

Performance

Read-Ahead, Placement, and Defragmentation

(a) i-nodes at the start of the diski-nodedata blocksevery file access: a long arm movement(b) cylinder / block groupsigroup 0igroup 1igroup 2an i-node and its data in the same groupBSD FFS, ext2/3/4
  • Read-ahead: when block k is requested, block k+1 is read too (if not cached). It helps only with sequential access; a "sequential?" hint is tracked per file. Linux adapts the window (default 128 KiB); applications give hints with posix_fadvise.
  • Reducing arm motion: place blocks that will be read consecutively close together; a bitmap makes it easy to find contiguous blocks. Allocate blocks two at a time (an old suggestion), or, as today, use extents, delayed allocation, and a multiblock allocator.
  • Defragmentation: over time files get fragmented. Windows has "Defragment and Optimize Drives"; Linux has it too: e4defrag, xfs_fsr, btrfs filesystem defragment. ext4's allocation policy reduces the need.
  • An SSD has no seek time: defragmentation just means unnecessary writes (wear); Windows sends TRIM instead of defragmenting an SSD.

Performance

SSD Caveats: FTL and TRIM

  • A flash page can't be updated in place; the SSD's FTL redirects every write to a new page, so internally an SSD is log-structured.
  • When the file system deletes a file it only marks the blocks free in its own bitmap. Unless it tells the SSD with TRIM (NVMe: Deallocate), the SSD thinks those pages are valid and copies them needlessly during garbage collection.
  • On Linux the notification is sent either in batches by the weekly fstrim.timer or on every delete with the discard mount option.

Consistency and Recovery

Backup Strategies

Physical dumpLogical dump
What is copied?all (used) blocks from block 0 to the endselected directories and files, with their paths
Speed, simplicityvery fast, simpleslower, complex
Skipping directories, incremental backupscan'tcan (changed files)
Restoring a single filehardeasy; on another machine too
Bad blocksthe controller must remap them, or the OS must know the listno problem
day 1day 2day 3day 4day 5day 6day 7full dump+ incrementalsrestore: full from day 1,then 2 … 7 in order
  • Two goals: recovery from disaster (disk failure, fire, ransomware) and recovery from carelessness (an accidentally deleted file).
  • Full + incremental: e.g., a full dump monthly, and daily only what changed since the last dump. Restore: the latest full dump onto an empty file system (directories first, then files), then the incrementals in order.
  • Temporary and special files (/dev, /proc, /tmp) aren't backed up. For a consistent backup while the system is in use, take a snapshot (LVM, ZFS/Btrfs, Windows VSS).
  • The 3-2-1 rule: 3 copies, on 2 different media, with 1 copy elsewhere (offline or immutable). Tape is still used for archiving: an LTO-10 cartridge holds up to 40 TB uncompressed.

Consistency and Recovery · Step by step

The Logical Dump Algorithm

12345678910bitmap010203040506070809010

Squares are directories, circles files; the numbers are i-node numbers. Orange: modified since the last dump (3, 6, 10).

12345678910bitmap111213140516171809110phase 1

Phase 1: walk the tree from the root, marking every modified file and all directories.

12345678910bitmap111213140516071809110phase 2

Phase 2: unmark directories with nothing modified beneath them: 7 (it contains only the unmodified 9).

12345678910bitmap111213140516071809110phase 3

Phase 3: dump the marked directories in i-node order (1, 2, 4, 8), even if unmodified, so paths can be rebuilt.

12345678910bitmap111213140516071809110phase 4

Phase 4: dump the marked files (3, 6, 10). On restore, directories are written first, then files.

More details: the free list isn't dumped but rebuilt on restore; a file with several hard links must be dumped only once; holes in sparse files and FIFO/device files need special handling.

Consistency and Recovery · Step by step

Consistency Checking: fsck

block number0123456789101112131415in use1101011110011100in free list0010100001100011Each block is 1 in one counter and 0 in the other: consistent.

(a) Consistent: fsck reads all i-nodes and the free list/bitmap, keeping two counters per block.

block number0123456789101112131415in use1101011110011100in free list0000100001100011Block 2 appears nowhere: a missing block → added to the free list.

(b) Missing block: harmless but wastes space; it goes on the free list.

block number0123456789101112131415in use1101011110011100in free list0010200001100011Block 4 is twice in the free list → the free list is rebuilt.

(c) Duplicate block in the free list: the same block could be handed out twice; the list is rebuilt. (This can't happen with a bitmap.)

block number0123456789101112131415in use1101021110011100in free list0010100001100011Block 5 in two files → allocate a new block, copy, notify the user.

(d) Duplicate data block: the worst case; the block is copied so each file gets its own, and the user is informed.

  • File consistency: walk all directories from the root, counting how often each i-node is seen, and compare with the i-node's link count. A link count that's too high just wastes space; one that's too low is dangerous (the file would be deleted early) → it is fixed. Orphaned i-nodes go into lost+found.
  • The Windows equivalent is chkdsk (called scandisk in Windows 9x). A full scan of a terabyte disk takes hours; that's why journaling and CoW were born.

Consistency and Recovery

Crashes and Atomicity: Why a Journal?

1. remove the directory entrycrash2. release the i-node to the free poolcrash3. release the blocks to the free poolcrash after 1: the i-node and blocks are orphaned (a leak; fsck finds it)crash after 2: the blocks leakin the reverse order: the directory could point to an i-node reused by another file → data gets mixed up!
  • One file operation changes several blocks; disks make only single-sector writes atomic. A crash in between leaves the structure inconsistent.
  • The family of solutions: (1) repair afterward: fsck; (2) write the intent first: journaling; (3) never overwrite: CoW and LFS; (4) order the writes: BSD soft updates.
  • Idempotence: the operations in the journal must give the same result however many times they are replayed. "Mark block n free" is idempotent; "append n to the free list" is not (it could be added twice).

Consistency and Recovery · Step by step

Journaling: Write to the Log First

journalfree journal spacefinal locationsi-node bitmapdata bitmap (v1)i-node table (v1)data block (new)

1. Data: the block Db added to the file is written to its final location first (ordered mode, ext4's default).

journalTxBI[v2]B[v2]free journal spacefinal locationsi-node bitmapdata bitmap (v1)i-node table (v1)data block (new)

2. Journal write: the transaction begin TxB and the metadata blocks to change (i-node I, bitmap B) are written to the journal sequentially.

journalTxBI[v2]B[v2]TxEfree journal spacefinal locationsi-node bitmapdata bitmap (v1)i-node table (v1)data block (new)

3. Commit: once the earlier writes reach the disk (barrier/FLUSH), TxE is written. Once TxE is on disk, the transaction is committed.

journalTxBI[v2]B[v2]TxEfree journal spacefinal locationsi-node bitmapdata bitmap (v2)i-node table (v2)data block (new)checkpoint

4. Checkpoint: I and B are written to their final locations; if these writes are interrupted, they can be repeated from the journal.

journalfree journal spacetransaction freedfinal locationsi-node bitmapdata bitmap (v2)i-node table (v2)data block (new)

5. Free: the transaction is marked complete in the journal superblock; the journal is reused like a circular buffer.

journalfree journal spacefinal locationsi-node bitmapdata bitmap (v2)i-node table (v2)data block (new)crash before TxE → the transaction is ignored; after → the journal is replayed

Recovery: at boot the journal is scanned; transactions with a TxE are replayed from the start (redo, idempotent), the others discarded. No fsck needed; it takes seconds.

Modes: journal (data journaled too: safest, slowest), ordered, writeback (no data ordering). Journaling file systems: ext3/ext4 (jbd2), XFS, NTFS ($LogFile), JFS. At Apple, HFS+ gave way to the CoW-based APFS.

Consistency and Recovery · Step through the code

Recovering from the Journal: Code Trace

void recover(Rec g[], int n) {
  int i = 0;
  while (i < n && g[i].type == TXB) {
    int j = i + 1;
    while (j < n && g[j].type == BLOCK)
      j++;
    if (j == n || g[j].type != TXE)
      break;              /* uncommitted: drop */
    for (int k = i + 1; k < j; k++)
      write(g[k].dest, g[k].data);
    i = j + 1;          /* next */
  }
}
Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39iFinal locations:I7v0Bv0I9v0D40v0

The system crashed while transaction 3 was being written. The journal holds three transaction headers; the blocks in their final locations are still old (v0). At boot, recover runs.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39iFinal locations:I7v0Bv0I9v0D40v0

g[0] is a TxB: transaction 1 begins.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39ijFinal locations:I7v0Bv0I9v0D40v0

At j = 3, TxE 1 is found: the transaction is committed and its blocks can be replayed.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39ijFinal locations:I7v1Bv0I9v0D40v0

I7's v1 copy is written to its final location. Even if it was written before the crash, that's fine: rewriting is idempotent.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39ijFinal locations:I7v1Bv1I9v0D40v0

B's v1 copy is written to its final location.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39iFinal locations:I7v1Bv1I9v0D40v0

g[4] is a TxB: transaction 2 begins.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39ijFinal locations:I7v1Bv1I9v0D40v0

At j = 7, TxE 2 is found: the transaction is committed and its blocks can be replayed.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39ijFinal locations:I7v1Bv1I9v2D40v0

I9's v2 copy is written to its final location.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39ijFinal locations:I7v1Bv1I9v2D40v2

D40's v2 copy is written to its final location.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39iFinal locations:I7v1Bv1I9v2D40v2

g[8] is a TxB: transaction 3 begins.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39iFinal locations:I7v1Bv1I9v2D40v2

Skipping the block records, j = 10 = n: the journal ended but there's no TxE. Transaction 3 wasn't committed: it is discarded, and I7's v3 is never written.

Journal (on disk after the crash):TxB10I7v11Bv12TxE13TxB24I9v25D40v26TxE27TxB38I7v39Final locations:I7v1Bv1I9v2D40v2

Result: transactions 1 and 2 were applied in full, transaction 3 as if it never happened. The file system is consistent; no whole-disk fsck is needed.

Consistency and Recovery · Step by step

Copy-on-Write (CoW): ZFS, Btrfs, APFS

superblockrootI1I2L1L2L3L4

The file system is a tree of blocks: superblock (the uberblock in ZFS) → root → internal nodes → data.

superblockrootI1I2L1L2L3L4L3'

L3 is to change: it is not written in place; a new copy, L3', goes to a free location. The old L3 is still intact.

superblockrootI1I2L1L2L3L4L3'I2'R'

The parents pointing to L3' are copied too: I2', root R'. Unchanged subtrees (I1, L4) are shared.

superblockrootI1I2L1L2L3L4L3'I2'R'gray: the old version = a snapshot (or free)

One atomic write: the superblock points to R'. After a crash either the old or the new tree is visible, never an intermediate state; no journal is needed.

superblockrootI1I2L1L2L3L4L3'I2'R'gray: the old version = a snapshot (or free)every pointer carries the child's checksum

If the old root is kept, a snapshot is free. ZFS/Btrfs keep a checksum in every pointer: silent corruption is caught on read and repaired from a mirror (scrub).

ZFS (Sun 2005, now OpenZFS), Btrfs (Linux), APFS (Apple, 2017), ReFS (Windows Server). The cost: fragmentation and write amplification on random overwrites; CoW can be turned off for database files.

Consistency and Recovery · Step by step

The Log-Structured File System (LFS)

memory bufferD(A,0)D(B,0)I(A)I(B)imapdisk (log)CRcheckpointregion

Writes are buffered in memory: data blocks, modified i-nodes, and pieces of the i-node map (imap).

memory bufferdisk (log)CRcheckpointregionsegment 1D(A,0)D(B,0)I(A)I(B)imap

When the buffer fills, everything is written sequentially to the end of the log as one segment (e.g., 1–2 MiB). i-nodes have no fixed location; the imap finds them, and the checkpoint region (CR) points to the latest imap.

memory bufferD(A,0)'I(A)'imap'disk (log)CRcheckpointregionsegment 1D(A,0)D(B,0)I(A)I(B)imapsegment 2D(A,0)'I(A)'imap'

When A's block changes, the new version goes to a new segment; the old D(A,0), I(A), and imap in segment 1 become dead.

memory bufferdisk (log)CRcheckpointregionsegment 1 (cleaned, empty)segment 3D(B,0)I(B)D(A,0)'I(A)'imap''

The cleaner: it reads segments with few live blocks, compacts the live ones into a new segment, and frees the old. Cleaning cost under load is LFS's weak spot.

Rosenblum and Ousterhout, Sprite LFS (1991). Designed on the assumption that reads are served from a large cache; it brings write bandwidth close to the disk's sequential speed. The idea lives on in F2FS (Android), NILFS2, NetApp WAFL, ZFS/Btrfs CoW, and SSD FTLs.

VFS and Networking · Step by step

The Virtual File System (VFS)

process: read(fd, buf, n)VFSsuperblock · inode (v-node) · dentry · file objectsext4XFSNFS clientprocfspage cache → block layer → NVMe driverRPC → TCP/IP → networkregister_filesystem()

Registration: a file system registers with the VFS at boot or when its module loads, handing over a table of the functions the VFS will call.

process: read(fd, buf, n)VFSsuperblock · inode (v-node) · dentry · file objectsext4XFSNFS clientprocfspage cache → block layer → NVMe driverRPC → TCP/IP → networkmount → s_op table

Mount: the VFS asks that file system to build a superblock object, which carries the operations table (s_op).

process: read(fd, buf, n)VFSsuperblock · inode (v-node) · dentry · file objectsext4XFSNFS clientprocfspage cache → block layer → NVMe driverRPC → TCP/IP → networkopen → i_op, f_op

open: the path is resolved through dentries; the concrete system fills a VFS i-node (v-node) with its i-node data; a file object is created and added to the fd table.

process: read(fd, buf, n)VFSsuperblock · inode (v-node) · dentry · file objectsext4XFSNFS clientprocfspage cache → block layer → NVMe driverRPC → TCP/IP → networkf_op->read_iter()

read: the VFS just calls the file->f_op->read_iter function pointer; for an ext4 file ext4's function runs, for an NFS file an RPC goes over the network. The application sees no difference.

The VFS's purpose is to offer one interface to different file systems at the same time (several systems on one disk are already possible via partitioning). Its overhead is one indirect call; the performance loss is small. The Windows counterpart is the I/O manager and file-system drivers.

VFS and Networking

The Network File System (NFS)

clientserverprocessVFSNFS clientRPC / XDRext4 / XFS / ZFS → diskVFSnfsdRPC / XDRTCP 2049READ(file handle, offset, n)mount server:/export /mnt/project
  • A remote directory is mounted into the local tree, and applications use it like local files. Sun designed it for SunOS in 1984, and it became an open standard.
  • Machine and architecture independence come from RPC and XDR. NFSv3 is stateless (UDP or TCP); NFSv4 is stateful, TCP only, on a single port (2049), with Kerberos security, ACLs, locking, and delegation; v4.1 adds pNFS.
  • The Windows-world counterpart is SMB 3 (encryption, multichannel). NFS originally ran over UDP/IP; today TCP is preferred; the only requirement is an IP network.

Summary

Summary

Files and directories

File: a named, persistent byte sequence + attributes. Directory: a name → i-node mapping; a tree + links, one tree via mounting.

Open files

fd → system-wide entry (position) → in-memory i-node. Path resolution proceeds component by component.

Allocation

Contiguous (fast, external fragmentation), linked/FAT (flexible, poor random access), i-node/extents (today's solution).

Free space

Bitmap, list, grouping, counting; block size is a trade-off between speed and space.

Performance

A unified page cache, read-ahead, nearby placement; TRIM on SSDs.

Consistency

fsck → journaling → CoW/LFS. Backups are still a must: 3-2-1.

Summary

Review Questions

  1. In a loop reading a 6000-byte file with a 4096-byte buffer, what does each read return?
  2. After ln a b and ln -s a c, what happens to b and c if you run rm a?
  3. How many disk reads does resolving /usr/ast/mbox take without caching (root i-node in memory)?
  4. 200 GiB disk, 1 KiB blocks, 4 B entries: how much memory does the FAT need?
  5. 4 KiB blocks, 4 B pointers, 12 direct + single/double/triple indirect: the largest file?
  6. In the same structure, at what level is byte 5,000,000, and how many reads does it take?
  7. Free blocks 2–5, 8–13, 17–18, 25–27: show them with the counting method.
  8. In journaling, what is the difference between a crash before and after TxE is written?
  9. Why is TRIM needed when a file is deleted on an SSD?
  10. The i-node table starts at block 3, blocks are 4 KiB, i-nodes 256 B (numbered from 0): which block and sector hold i-node 40?
Answers
  1. 4096, 1904, 0 (EOF).
  2. b works (link count 1); c dangles (ENOENT).
  3. 5: the root directory block, i-node 6, block 132, i-node 26, block 406.
  4. 209,715,200 entries × 4 B = 800 MiB.
  5. (12 + 1024 + 1024² + 1024³) × 4 KiB ≈ 4 TiB (4,402,345,721,856 B).
  6. Block 1220 → double indirect (0, 184); 3 reads.
  7. (2, 4) (8, 6) (17, 2) (25, 3).
  8. Before: the transaction is ignored, leaving the old consistent state. After: the journal is replayed.
  9. The SSD doesn't know which pages were freed; it copies them needlessly during garbage collection, increasing write amplification.
  10. 40 × 256 = 10,240 = 2 × 4096 + 2048 → block 3 + 2 = 5; (5 × 4096 + 2048) / 512 = sector 44.

Summary

References

  • A. Silberschatz, P. B. Galvin, G. Gagne, Operating System Concepts, 10th ed., Wiley, 2018 (Chapters 13 File-System Interface, 14 File-System Implementation, 15 File-System Internals).
  • A. S. Tanenbaum, H. Bos, Modern Operating Systems, 4th ed. (2014) / 5th ed. (2023), Pearson (Chapter 4: File Systems).
  • R. H. Arpaci-Dusseau, A. C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces (OSTEP), ostep.org: "Files and Directories", "File System Implementation", "Fast File System", "Crash Consistency: FSCK and Journaling", "Log-structured File Systems", "Flash-based SSDs", "Data Integrity", "Sun's NFS".
  • M. Rosenblum, J. K. Ousterhout, "The Design and Implementation of a Log-Structured File System", ACM TOCS, 1992; M. K. McKusick et al., "A Fast File System for UNIX", ACM TOCS, 1984.
  • Linux kernel documentation: "Overview of the Linux Virtual File System", "ext4 Data Structures and Algorithms"; Microsoft Learn: NTFS and the exFAT specification; OpenZFS and Btrfs documentation; the UEFI specification (GPT).

The End

Chapter 10: File Systems
Previous: Chapter 9: Virtual Memory · Next: Chapter 11: Input/Output
1 / 1