Chapter 13: Security

Operating Systems
Security goals and threatsAttackers and defense in depthEncryption, hashes, signatures, PKI, TLSProtection domains and the access matrixACLs and capabilities, least privilegeBell-LaPadula, Biba, covert channelsAuthentication: passwords, MFA, passkeysSoftware exploits and defensesInsider attacksMalwareSandboxes, containers, SELinuxSecure boot and side channels
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Learning Outcomes

Goals and threats

Distinguish confidentiality, integrity and availability, attack types and attacker profiles.

Cryptography

Explain symmetric and asymmetric encryption, hash functions, digital signatures, certificates and the TLS handshake.

Protection

Read the access matrix as ACLs and as capabilities; apply least privilege, setuid and Linux capabilities.

Formal models

Apply the Bell-LaPadula and Biba rules; explain why covert channels cannot be closed.

Authentication

Compare salted slow password hashes, MFA, TOTP and FIDO2 passkeys.

Attacks and defenses

Recognize memory-safety bugs, TOCTOU and malware; canaries, NX, ASLR, CFI, sandboxes and secure boot.

Introduction

Security Goals: The CIA Triad

ConfidentialitysecrecyIntegrityno tamperingAvailabilityuptimeAssetdata · services · hardwareno unauthorized readsservice stays upno unauthorized changesViolation:disclosure (espionage, leaks)Violation:denial of service (DoS)Violation: tampering, forged records, ransomware encryption

Confidentiality keeps data secret, integrity keeps it unaltered, availability keeps it usable. Often added: authenticity and accountability (nonrepudiation, audit logs).

Introduction

Threats and Breach Categories

CategoryDefinitionExample
Breach of confidentialityUnauthorized reading of dataA credit-card database is leaked
Breach of integrityUnauthorized modification of dataGrades tampered with in a records system
Breach of availabilityDestruction of data or serviceRansomware encrypts the disk
Theft of serviceUnauthorized use of resourcesCrypto mining on a hijacked server
Denial of service (DoS/DDoS)Preventing legitimate useSYN flood from a botnet
MasqueradingPretending to be someone elseA password stolen by phishing
Privilege escalationGaining more rights than grantedBecoming root through a bug in a setuid program
Threatpossible harmVulnerabilityweaknessAttackattempt to use itExploitcode / methodBreachloss of CIApatches fix vulnerabilities, authentication stops attacks, canaries/NX stop exploits, backups limit the damage
A threat is potential harm, a vulnerability is a weakness that makes it possible, an attack is an attempt to use the vulnerability, and an exploit is the code or method that does it.

Introduction · Step by step

Network Attacks: Eavesdropping, Replay, Man in the Middle

AliceclientBobserverEve / Malloryattackerpassword=… · “pay 1000 TL”

Normal traffic between Alice and Bob. If the channel is unencrypted and unauthenticated, every attack below is possible.

AliceclientBobserverEve / Malloryattackerpassword=… · “pay 1000 TL”eavesdropping (passive): content is read

Eavesdropping: a passive attacker copies the traffic → breach of confidentiality. Fix: encryption (TLS).

AliceclientBobserverEve / Malloryattacker“pay 1000 TL” (signed, valid)the same message issent again (replay)

Replay attack: even an encrypted, signed message can be resent. Fix: a nonce, a timestamp, a sequence number.

AliceclientBobserverEve / MalloryattackerAlice thinks she is talkingto “Bob”Bob thinks he is talkingto “Alice”man in the middle: reads and alters

Man in the middle (MITM): the attacker poses as the other party to both sides. Fix: verify the peer's identity with a certificate.

AliceclientBobserverEve / MalloryattackerCookie: session=7f3a…with the stolen cookiejoins the session

Session hijacking: the session token issued after authentication is stolen. Fix: HTTPS only, Secure/HttpOnly cookies, short lifetimes.

Introduction

Attackers and Accidental Data Loss

Curious insider

Little technical skill; “peeks” at others' email or files. Unethical, often illegal.

Malicious employee

Abuses their authority: data theft, sabotage, logic bombs. The hardest group to detect.

Cybercriminal

Profit-driven: ransomware, identity and card theft, botnets for hire. The most common threat today.

State-sponsored groups (APT)

Military and commercial espionage, sabotage of critical infrastructure; zero-day exploits and supply-chain attacks.

Accidental data lossExampleCountermeasure
Acts of naturefire, flood, earthquake (and rats gnawing tapes)geographically separate backup
Hardware/software errordisk failure, CPU fault, software bugRAID, checksums, versioning
Human errorwrong command, formatting the wrong disk, lost device3-2-1 backups, encrypted disks

Accidents lose more data than attacks do. 3-2-1: 3 copies, 2 different media, 1 copy offsite and preferably immutable — which also helps against ransomware.

Introduction

Defense in Depth and Good Practice

Policy, training, physical securityNetwork: firewall, segmentationEndpoint: patching, EDR, disk encryptionOS: access control, sandboxingApplication: input validation, memory-safe languageDataFirewallInternetPC 1PC 2PC 3only permitted connections pass
  • Patching: apply security updates quickly (most attacks use known vulnerabilities).
  • Strong authentication and least-privilege access control; logging and monitoring.
  • Regular penetration testing; training users against phishing; zero trust: the inside of the network is not trusted either.

Introduction

The Trusted Computing Base (TCB)

User spaceProcess AProcess BProcess CAttackerTrusted computing base (TCB)Reference monitorchecks every accessKernel, driversAuthenticationHardware (MMU, TPM)denied

A reference monitor must meet three conditions: it is invoked on every access (cannot be bypassed), it is tamperproof, and it is small enough to verify. The bigger the TCB, the likelier a bug — hence the interest in microkernels and formally verified kernels such as seL4.

Cryptography · Step by step

Encryption Basics and Secret-Key Cryptography

Plaintext PEncrypt ECiphertext CDecrypt DPkey Kkey K (same)attacker sees CC = E(P, K) · P = D(C, K)

Symmetric (secret-key) encryption: the same K encrypts and decrypts. Kerckhoffs's principle: the algorithm may be public; security must rest only on keeping the key secret.

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMERHABA

Monoalphabetic substitution: each letter maps to a fixed letter; the key is this 26-letter permutation (26! ≈ 4×1026 possible keys).

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMDERHABA

M → D: the letter under M in the table is D.

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMDETRHABA

E → T

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMDETRKHABA

R → K

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMDETRKHIABA

H → I

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMDETRKHIAQBWAQ

Result: MERHABA (“hello”) → DTKIQWQ. Both A's became Q: letter frequencies survive.

AQBWCEDRETFYGUHIIOJPKALSMDNFOGPHQJRKSLTZUXVCWVXBYNZMPlaintextCiphertextMDETRKHIAQBWAQ

The key space is huge, yet frequency analysis (in English E, T and A dominate) breaks the cipher with a few lines of text. Today: AES-256-GCM, ChaCha20-Poly1305 (encryption + integrity together).

Cryptography

Public-Key Cryptography

AlicesenderBobreceiverpublicPub_BprivatePriv_Bthe public key can be given to anyoneE(P, Pub_B)D(C, Priv_B)CWhat Pub_B encrypts, only Priv_B decrypts.
SymmetricAsymmetric (public key)
KeyOne, sharedPublic + private pair
SpeedVery fast (AES-NI)100–1000× slower
UseBulk dataKey exchange, signatures
ExampleAES, ChaCha20RSA, ECDH/X25519, Ed25519, ML-KEM

Asymmetric encryption rests on one-way trapdoor functions: multiplying two large primes is easy, factoring the product (RSA) or taking a discrete logarithm (ECC) is hard. Quantum-resistant ML-KEM was standardized in 2024 (FIPS 203).

Cryptography · Step by step

Hash Functions and Digital Signatures

SenderReceiverDocument D

The sender wants to deliver document D unaltered and provably from them (secrecy is not required).

SenderReceiverDocument DSHA-256H(D)

A hash function (SHA-256, SHA-3) turns input of any size into a fixed 256-bit digest: one-way and collision-resistant; flipping one bit changes half the digest. MD5 and SHA-1 are broken.

SenderReceiverDocument DSHA-256H(D)Sign (Priv_A)signature S

The digest is signed with the sender's private key (RSA-PSS, ECDSA, Ed25519). Only the holder of Priv_A can do this.

SenderReceiverDocument DSHA-256H(D)Sign (Priv_A)signature SDS+ certificate(Pub_A)

The document, the signature and a certificate carrying the sender's public key are sent together.

SenderReceiverDocument DSHA-256H(D)Sign (Priv_A)signature SDS+ certificate(Pub_A)SHA-256H(D) againVerify (Pub_A)match

The receiver computes the digest itself and verifies the signature with Pub_A: if they match, the document is intact and came from Alice (nonrepudiation).

SenderReceiverDocument DSHA-256H(D)Sign (Priv_A)signature SD'S+ certificate(Pub_A)SHA-256H(D) againVerify (Pub_A)NO MATCH

If a single byte changes in transit (D′), the digest no longer matches and verification fails. The attacker cannot make a new valid signature without Priv_A.

A hash is not compression; an unsigned hash proves nothing on its own (the attacker can change the hash along with the document).

Cryptography

Certificates and Public-Key Infrastructure (PKI)

Root CAself-signed; in the OS/browsertrust storeIntermediate CAsigned by the rootwww.example.edu.trsigned by the intermediate CApublic key + domain namesignssignsBrowser verification1. Server cert + intermediate CA arrive2. Signatures verified along the chain3. Is the root in the trust store?4. Domain name matches? Expiry?5. Revocation (OCSP/CRL) and CT log6. All good → connection trustedForged certificate (MITM)not signed by a trusted CA, sono chain can be built → warning

A certificate (X.509) = public key + identity (domain name) + validity period + CA signature. It answers the question of whether a public key really belongs to its claimed owner. Thanks to Let's Encrypt (ACME), certificates are free and last 90 days; lifetimes are set to shrink gradually to 47 days.

Cryptography · Step by step

The TLS 1.3 Handshake (Simplified)

ClientServerClientHello: cipher suites, key_share (X25519 public value), random{ } = sent encrypted (green)

In its first message the client sends the algorithms it supports and an ephemeral Diffie-Hellman public value.

ClientServerClientHello: cipher suites, key_share (X25519 public value), randomServerHello: chosen suite, key_share{ } = sent encrypted (green)

The server picks one and sends its own DH public value. Everything so far is in the clear.

ClientServerClientHello: cipher suites, key_share (X25519 public value), randomServerHello: chosen suite, key_shareBoth sides compute the same secret with ECDHE → session keys{ } = sent encrypted (green)

Each side combines the peer's public value with its own secret to compute the same shared secret; the secret never crosses the network.

ClientServerClientHello: cipher suites, key_share (X25519 public value), randomServerHello: chosen suite, key_shareBoth sides compute the same secret with ECDHE → session keys{Certificate, CertificateVerify, Finished}{ } = sent encrypted (green)

The server sends its certificate and signs the handshake transcript with its private key (CertificateVerify): a MITM cannot produce this signature.

ClientServerClientHello: cipher suites, key_share (X25519 public value), randomServerHello: chosen suite, key_shareBoth sides compute the same secret with ECDHE → session keys{Certificate, CertificateVerify, Finished}{Finished} — client verified chain and signature{ } = sent encrypted (green)

The client verifies the chain and signature, and its Finished confirms the handshake was not tampered with. Just one round trip (1-RTT).

ClientServerClientHello: cipher suites, key_share (X25519 public value), randomServerHello: chosen suite, key_shareBoth sides compute the same secret with ECDHE → session keys{Certificate, CertificateVerify, Finished}{Finished} — client verified chain and signature{Application data} ⇄ AES-GCM / ChaCha20-Poly1305{ } = sent encrypted (green)

Application data flows under symmetric, authenticated encryption. Ephemeral keys give forward secrecy: even if the private key is stolen later, old traffic cannot be decrypted.

Cryptography · Step through the code

Diffie-Hellman Key Exchange: Code Trace

long modpow(long g, long e, long p) {
    long r = 1;
    while (e > 0) {
        if (e & 1) r = r * g % p;
        g = g * g % p;
        e >>= 1;
    }
    return r;
}
int main(void) {
    /* public: p = 23, g = 5 */
    long A  = modpow(5, 6, 23);
    long B  = modpow(5, 15, 23);
    long sA = modpow(B, 6, 23);
    long sB = modpow(A, 15, 23);
}
Alicea = 6 (secret)s = ?Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = ?B = ?modpow(5, 6, 23)e:110r = 1 g = 5e = 6 = 110₂; bits processed right to left

Alice has picked secret a = 6; her public value A = 5⁶ mod 23 is being computed. Square-and-multiply scans the exponent's bits right to left; r = 1.

Alicea = 6 (secret)s = ?Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = ?B = ?modpow(5, 6, 23)e:110r = 1 g = 2bit 0 = 0 → r unchanged

e = 6 is even: r stays. The base is squared: g = 5·5 mod 23 = 2; e = 3.

Alicea = 6 (secret)s = ?Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = ?B = ?modpow(5, 6, 23)e:110r = 2 g = 4bit 1 = 1 → r = r·g mod p

e = 3 is odd: r = 1·2 = 2. g = 2·2 = 4; e = 1.

Alicea = 6 (secret)s = ?Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = ?B = ?modpow(5, 6, 23)e:110r = 8 g = 16bit 2 = 1 → r = r·g mod p

e = 1 is odd: r = 2·4 = 8. g = 16; e = 0, the loop ends. 3 rounds instead of 6 multiplications: an n-bit exponent takes only n rounds.

Alicea = 6 (secret)s = ?Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = 8B = ?A = modpow(5, 6, 23)A = 8Alice sends A to Bob over the network

The function returns 8: A = 8. This value is public; a = 6 never leaves Alice.

Alicea = 6 (secret)s = ?Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = 8B = 19B = modpow(5, 15, 23)e = 15 = 1111₂: 4 rounds → B = 19Bob sends B to Alice over the network

Bob runs the same function with his secret b = 15: 5¹⁵ mod 23 = 19. B = 19 travels to Alice.

Alicea = 6 (secret)s = 2Bobb = 15 (secret)s = ?network (everyone sees)p = 23, g = 5A = 8B = 19sA = modpow(19, 6, 23)Alice: B^a mod p = 19⁶ mod 23 = 2the shared secret never crossed the network

Alice raises Bob's public value to her secret exponent: 19⁶ mod 23 = 2.

Alicea = 6 (secret)s = 2Bobb = 15 (secret)s = 2network (everyone sees)p = 23, g = 5A = 8B = 19sB = modpow(8, 15, 23)Bob: A^b mod p = 8¹⁵ mod 23 = 2both sides found 5^(6·15) mod 23 = 2

Bob also gets 8¹⁵ mod 23 = 2: both computed 5^(a·b) mod p. An eavesdropper sees only p, g, A and B; finding a is the discrete logarithm problem (in practice a 2048-bit p or X25519).

Protection

Protection Domains

Domain 1File1[R]File2[RW]Domain 2File3[R]File4[RWX]File5[RW]Domain 3File6[RWX]Plotter2[W]Printer1[W]Printer1 is in both D2 and D3: objects can be shared between domains
  • A domain is a set of (object, rights) pairs. At any moment a process runs in one domain.
  • UNIX/Linux: domain = (UID, set of GIDs); kernel mode and user mode are also two separate domains.
  • Domain switching: running a setuid program, entering the kernel through a system call.

Protection

The Protection Matrix

File1File2File3File4File5Printer1File6Plotter2D1D2D3
D1RRWEnter
D2RRWXRWW
D3WRWXW
  • Row = domain, column = object, cell = rights. Domains can be objects too: Enter in cell D1 → D2 says a process in D1 may switch to D2.
  • The matrix is very sparse: thousands of users × millions of files. It is never stored whole; either the columns (ACLs) or the rows (capabilities) are stored.
  • Authorized and unauthorized states: every state of the matrix is checked against the security policy. If policy says Eve must not read a file, yet an R appears in that cell after some sequence of commands, the system has entered an unauthorized state.
  • Harrison-Ruzzo-Ullman (HRU, 1976): in general, “can a right leak?” is undecidable, which is why restricted models (BLP, Biba) are used.

Protection · Step by step

One Matrix, Two Forms: ACLs and Capabilities

File1File2File3File4File5Printer1File6Plotter2D1RRWD2RRWXRWWD3WRWXW15 of 24 cells are empty: the matrix is sparse

The protection matrix of three domains and eight objects.

File1File2File3File4File5Printer1File6Plotter2D1RRWD2RRWXRWWD3WRWXWACL = column (stored with the object)Printer1D2:W → D3:W

Access control list (ACL): a column of the matrix, stored with the object. Printer1's ACL: D2 may write, D3 may write.

File1File2File3File4File5Printer1File6Plotter2D1RRWD2RRWXRWWD3WRWXWACL = column (stored with the object)File1D1:RFile2D1:RWFile4D2:RWXPrinter1D2:W → D3:WFile6D3:RWX

Each object's nonempty entries become its ACL. On access, the kernel looks up the process's identity in the object's ACL.

File1File2File3File4File5Printer1File6Plotter2D1RRWD2RRWXRWWD3WRWXWCapability list = row (kept with the process / in the kernel)Process (D2)File3 : RFile4 : RWXFile5 : RWPrinter1 : W

Capability list (C-list): a row of the matrix, belonging to the domain (process). D2's capabilities: File3:R, File4:RWX, File5:RW, Printer1:W.

File1File2File3File4File5Printer1File6Plotter2D1RRWD2RRWXRWWD3WRWXWCapability list = row (kept with the process / in the kernel)Process (D2)File3 : RFile4 : RWXFile5 : RWPrinter1 : WTo access an object the process presents the capability; no name lookup or ACL scan is needed

Comparison: ACLs make “who can access this object?” and revocation easy; capabilities make delegation and fast checks easy, but revoking every copy is hard.

Protection

Access Control Lists in Practice

$ ls -l report.txt
-rw-r----- 1 ali accounting 4096 report.txt
 rw-  owner   r--  group  ---  other

$ setfacl -m u:ayse:rw report.txt
$ getfacl report.txt
user::rw-
user:ayse:rw-
group::r--
mask::rw-
other::---
GroupABCD
Group 1RRWX——
Group 2W———
Group 3——XR
Group 4————
AdminRWXRWXRWXRWX
  • The 9 rwx bits are fast but coarse; an ACL adds named users/groups, and mask caps their maximum rights (+ in ls).
  • Pro: easy to understand, easy per-object revocation.
  • Con: hard to manage at scale; a list scan on every access (eased by caching).
  • NTFS DACL: ordered allow/deny entries (SID + access mask), explicit denies first, inheritance from folders (icacls). POSIX, NFSv4 and NTFS ACLs do not map exactly onto each other.

Protection · Step through the code

Access Control List: Code Trace

/* ACL of file A: R=4 W=2 X=1 */
struct entry { int group, rights; } acl[] =
    {{G1, 4}, {G2, 2}, {ADM, 7}};

int access(int group, int req) {
    for (int i = 0; i < 3; i++)
        if (acl[i].group == group) {
            int h = acl[i].rights;
            return (h & req) == req;
        }
    return 0;   /* not listed: denied */
}
request: —acl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

File A's ACL is column A of the protection matrix: G1 reads, G2 writes, the admin can do anything. G3 and G4 are not listed; empty cells are not stored.

request: G2 · W (2)acl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

access(G2, W): the entry at i = 0 belongs to G1, so the group does not match; the scan continues.

request: G2 · W (2)allowacl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

G2 found at i = 1: h = 2. Since (2 & 2) == 2, access is allowed.

request: G1 · W (2)denyacl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

access(G1, W): the first entry matches, h = R. 4 & 2 = 0 ≠ 2: denied. The decision is made at the first matching entry; the list is not scanned further.

request: G4 · R (4)acl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

access(G4, R): G4 is in no entry, so the kernel scans the whole list. That is the cost of every access with long ACLs.

request: G4 · R (4)denyacl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

The loop ends with no match: it returns 0. Fail-safe default: anything not explicitly allowed is forbidden.

request: ADM · R|X (5)allowacl[] (stored with file A)G1 : R--i = 0G2 : -W-i = 1YON : RWXi = 2

access(ADM, R|X): matches at i = 2, h = 7. 7 & 5 = 5: allowed. Several rights are checked in one request; access is granted only if all are present.

Protection

Capability Lists

Processuser modesees only an index:read(cap 2, …)C-list in the kernel0File3 R1File4 RWX2File5 RW3Printer1 WCryptographically protected capabilityServerObjectRightsf(Object, Rights, Secret)Cannot be forged even if users hold it:f is one-way (e.g. HMAC); only the serverknows the secret (Amoeba).
Copy capability

A new capability for the same object (delegation).

Copy object

A new object and a new capability for it.

Remove capability

The entry is deleted from the list; the object stays.

Destroy object

Object and capability are deleted for good.

Capabilities must be unforgeable: kept in the kernel (accessed by index), protected by tagged hardware (CHERI), or cryptographically signed. Everyday examples: UNIX file descriptors (no permission recheck after open), FreeBSD Capsicum, seL4, Fuchsia.

Protection · Step through the code

Least Privilege and setuid

/* /usr/bin/passwd  -rwsr-xr-x root */
int main(void) {
    uid_t me = getuid();
    int fd = open("/etc/shadow", O_RDWR);
    update_own_entry(fd, me);
    close(fd);
    setresuid(me, me, me);
    /* ... rest of the work ... */
    fd = open("/etc/shadow", O_RDWR);
    if (fd < 0) perror("open");
    else close(fd);
    return 0;
}
passwd processruid = 1000 (ali)euid = 1000/etc/shadow-rw-r-----root shadowshell: ali, unprivileged

Ali (uid 1000) runs passwd. The file is owned by root and its setuid bit (s) is set.

passwd processruid = 1000 (ali)euid = 0/etc/shadow-rw-r-----root shadowexecve: euid ← file owner

During execve the effective ID becomes the file owner: euid = 0. The real ID (ruid) is still Ali.

passwd processruid = 1000 (ali)euid = 0/etc/shadow-rw-r-----root shadowgetuid() → 1000

getuid() returns the real ID: the program knows on whose behalf it runs and changes only that user's line.

passwd processruid = 1000 (ali)euid = 0/etc/shadow-rw-r-----root shadowallowed (euid 0)

The kernel checks access against the effective ID: euid 0, so the file opens.

passwd processruid = 1000 (ali)euid = 1000/etc/shadow-rw-r-----root shadowprivilege dropped for good

As soon as the job is done, setresuid sets the real, effective and saved IDs all to 1000: least privilege, with no way back.

passwd processruid = 1000 (ali)euid = 1000/etc/shadow-rw-r-----root shadowEACCES: permission denied

Even if a bug in the rest of the program is exploited, the attacker is no longer root. Setuid-root programs are the classic target of privilege-escalation attacks.

Protection

Linux Capabilities and Design Principles

root (uid 0)allowed everythingCAP_NET_BIND_SERVICEports below 1024CAP_NET_ADMINnetwork configurationCAP_SYS_TIMEset the clockCAP_DAC_OVERRIDEbypass file permissionsCAP_KILLsignal othersCAP_SYS_ADMIN“the new root”: very broad≈ 41 capabilities; each thread has permitted, effective, inheritable and bounding sets
$ sudo setcap cap_net_bind_service=+ep /usr/local/bin/websrv
$ getcap /usr/local/bin/websrv
/usr/local/bin/websrv cap_net_bind_service=ep     # may bind port 80, no other root rights
  • Least privilege (Saltzer & Schroeder, 1975): every component should run with the smallest set of rights that gets its job done, so a compromise does limited damage.
  • Other principles: fail-safe defaults (anything not explicitly allowed is forbidden), complete mediation (every access is checked), open design, separation of privilege, psychological acceptability.
  • In practice: privilege separation (OpenSSH: a small root process + an unprivileged child), systemd CapabilityBoundingSet=, NoNewPrivileges=.

Formal Models · Step by step

Bell-LaPadula: No Read Up, No Write Down

Top Secret (4)objectSecret (3)objectProcessConfidential (2)objectUnclassified (1)objectThe process runs with “Secret” clearance

Military classification levels; every process and object has a level. The goal: information must not leak downward (confidentiality).

Top Secret (4)objectSecret (3)objectProcessConfidential (2)objectUnclassified (1)objectread ✓

Simple security property: a process at level k may read only objects at levels ≤ k. Reading down is allowed.

Top Secret (4)objectSecret (3)objectProcessConfidential (2)objectUnclassified (1)objectread ✗

No read up: a process cleared for Secret cannot read a Top Secret document.

Top Secret (4)objectSecret (3)objectProcessConfidential (2)objectUnclassified (1)objectwrite ✓

* property: a process at level k may write only at levels ≥ k. Writing up does not break secrecy (usually a “blind” append).

Top Secret (4)objectSecret (3)objectProcessConfidential (2)objectUnclassified (1)objectwrite ✗

No write down: otherwise Secret information could leak by being copied into a Confidential file.

Top Secret (4)objectSecret (3)objectProcessConfidential (2)objectUnclassified (1)objectwrite ✗even a Trojan horse running inside a Secret process cannot write down

The payoff: even a Trojan horse the user runs cannot move information down, because the rule depends on the system, not on the user's good intentions (mandatory access control, MAC).

Formal Models · Step by step

Biba: No Read Down, No Write Up

High (system) (3)objectMedium (user) (2)objectProcessLow (Internet) (1)objectThe process runs at medium integrity

Biba (1977) mirrors BLP for integrity: levels express trustworthiness. The goal: untrusted data must not contaminate upward.

High (system) (3)objectMedium (user) (2)objectProcessLow (Internet) (1)objectwrite ✓

* integrity property: a process at level k may write only at levels ≤ k.

High (system) (3)objectMedium (user) (2)objectProcessLow (Internet) (1)objectwrite ✗

No write up: a user process cannot modify system files.

High (system) (3)objectMedium (user) (2)objectProcessLow (Internet) (1)objectread ✓

Simple integrity property: a process may read only from levels ≥ k; reading trusted data is fine.

High (system) (3)objectMedium (user) (2)objectProcessLow (Internet) (1)objectread ✗

No read down: a process that read untrusted data from the Internet and acted on it would be “contaminated”.

High (system) (3)objectMedium (user) (2)objectLow (Internet) (1)objectProcesswrite ✗a low-integrity browser process cannot write to user files

A real example: Windows Mandatory Integrity Control (MIC). Browser and protected-mode processes run at Low and cannot write to Medium-level documents (no write up).

Formal Models

Comparing Bell-LaPadula and Biba

Bell-LaPadula (1973)Biba (1977)
ProtectsConfidentialityIntegrity
ReadDown allowed, no upUp allowed, no down
WriteUp allowed, no downDown allowed, no up
PreventsInformation leaking downTainted data spreading up
TodaySELinux MLS, military systemsWindows MIC, FreeBSD/Linux integrity policies
Bell-LaPadula (confidentiality)Top SecretprocessConfidentialreadscannot readwritescannot writeBiba (integrity)HighprocessLowreadscannot readwritescannot write
  • Used together, the two models leave a process talking almost only to its own level: one protects confidentiality, the other integrity, and they conflict.
  • Naming: in BLP the simple security property forbids reading up and the * property forbids writing down; in Biba (1977) the simple integrity property forbids reading down and the * integrity property forbids writing up.
  • Neither model stops covert channels: the rules govern only explicit reads and writes.

Formal Models

Covert Channels and Steganography

ClientconfinedServerprocesses secret dataCollaboratorsecret dataovert channel ✗covert channel ✓Steganography181…1114…073…1209…190…037…1low bit = hidden message
Storage channel

A change in some shared state: a file exists or not, locked or unlocked, disk fullness.

Timing channel

The timing of events: heavy CPU use = 1, idling = 0; cache latency.

Resource channel

Use of a shared resource: page-fault rate, network traffic volume.

Covert channel: an information path the designer never intended for communication, which bypasses the security policy. Lampson's confinement problem (1973): closing them all is practically impossible, though their bandwidth can be reduced. Steganography: text (say, Shakespeare's plays) can be hidden in the low bits of the pixels of a photo of three zebras; the image looks unchanged.

Formal Models · Step by step

A Covert Channel Using File Locks

ServerFile FCollaboratort1t2t3t4t5t6t7t8Bits received: —

The server and collaborator agree in advance: in each clock interval, F locked means 1, unlocked means 0. No read or write takes place; F's contents never change.

ServerFile FCollaboratort1locklocked1t2t3t4t5t6t7t8Bits received: 1

The server locks F at t1; the collaborator tries to lock it, fails → 1.

ServerFile FCollaboratort1locklocked1t2unlockfree0t3locklocked1t4locklocked1t5t6t7t8Bits received: 1011

The first four bits the same way: 1 0 1 1.

ServerFile FCollaboratort1locklocked1t2unlockfree0t3locklocked1t4locklocked1t5unlockfree0t6unlockfree0t7locklocked1t8unlockfree0Bits received: 10110010

One byte in eight intervals: 10110010 (0xB2). At 100 intervals per second that is ≈ 100 bit/s; a password or key leaks in seconds.

ServerFile FCollaboratort1locklocked1t2unlockfree0t3locklocked1t4locklocked1t5unlockfree0t6unlockfree0t7locklocked1t8unlockfree0Bits received: 10110010

Countermeasures: hide lock state from others, add noise and delay, partition resources by level. The bandwidth drops but never reaches zero.

Authentication

Authentication Factors

Something you know

Password, PIN, security question. Cheap, but can be stolen, guessed and reused.

Something you have

Phone, smart card, FIDO2 key, TOTP generator. Can be stolen, but not copied remotely.

Something you are

Fingerprint, face, iris. Easy, but unchangeable; once leaked, burned for good.

Where you are / how you act

Location, IP, typing rhythm, device fingerprint. Usually an extra risk signal.

Multifactor authentication (MFA): at least two factors from different categories. “Password + security question” is not MFA (both are knowledge). MFA is the single most effective defense against stolen passwords.

Challenge-response authentication asks the user preregistered questions; but answers like “mother's maiden name” or “first pet” can be found on social media, so it is considered weak today.

Authentication · Step through the code

Password Storage: Salt and Slow Hashes

# Registration / password change
salt = os.urandom(16)
h = argon2(password, salt)
store(user, salt, h)

# Login attempt
salt, h = lookup(user)
g = argon2(attempt, salt)
if constant_eq(g, h): ok()
PasswordSaltSlow hash Hali7Gx!veli7Gx!cankQ4z

Passwords are never stored in the clear. Two of the three users picked the same password (people often do).

PasswordSaltSlow hash Hali7Gx!veli7Gx!cankQ4zThe same password → thesame hash: the attackersees repeats and cracks themwith precomputed tables

A plain (unsalted) hash is not enough: the same password gives the same hash; the attacker spots repeats and cracks them all at once with a rainbow table.

PasswordSaltSlow hash Hali7Gx!a3f1+veli7Gx!9b2e+cankQ4z5d7c+A random salt per user:even the same passwordbecomes a different input

Salt: a random, nonsecret value per user is added to the password. The same password is now a different input; precomputation is useless.

PasswordSaltSlow hash Hali7Gx!a3f1+b91c4e…veli7Gx!9b2e+2f87aa…cankQ4z5d7c+d013f5…Stored: salt + H.Argon2 is deliberately slowand memory-hard: brute forceon GPUs/ASICs is expensive

A slow, memory-hard hash: Argon2id (or scrypt, bcrypt). One check takes milliseconds; billions of guesses take years. A constant-time comparison during verification prevents timing leaks.

Plain SHA-256/MD5 is unfit for passwords: they are fast, exactly what you do not want for passwords. 1970s UNIX used a 2-character salt + slowed-down DES (crypt); the idea is the same, the parameters far stronger today. After a breach, passwords must still be reset.

Authentication

MFA and Time-Based One-Time Passwords (TOTP)

Servershared secret KApplicationsame secret KTOTP = HMAC(K, time / 30s)KclockHMAC739 412renewed every 30 sThe code never carries K over the network and cannot be reused (valid only for those 30 s)
SMS codes

Common but weak: they can be hijacked by SIM swapping and SS7 attacks. Better than nothing.

TOTP (authenticator app)

RFC 6238; works offline. Phishable: the user may type the code into a fake site.

Push approval

Easy, but open to “MFA fatigue” attacks. Number matching helps.

Authentication · Step through the code

Generating a TOTP Code: Code Trace

uint32_t totp(uint8_t *K, uint64_t t) {
    uint64_t T = t / 30;    /* 30 s */
    uint8_t h[20];
    hmac_sha1(K, T, h);
    int o = h[19] & 0x0F;   /* offset */
    uint32_t v = (h[o] & 0x7F) << 24
               | h[o+1] << 16
               | h[o+2] << 8
               | h[o+3];
    return v % 1000000;     /* 6 digits */
}
h[0..19]: not computed yetK: 20-byte secret shared by QR code at enrollmentt = 59: Unix time (seconds)

The phone and the server know the same secret K and the same time. RFC 6238's test key (“12345678901234567890”) at t = 59 s.

h[0..19]: not computed yetT = 59 / 30 = 10–29 s → 0, 30–59 s → 1, 60–89 s → 2

Time is cut into 30-second steps: T = 1. Every moment within a step gives the same code.

750a418a2193d44cb5e160076484e98a10c11139127e13ea1474157a162d173318ab19h = HMAC-SHA1(K, T)20 bytes; no one without K can produce it

T, as an 8-byte number, goes through HMAC-SHA1 with K: a 20-byte digest. Incrementing T by one gives a completely different digest.

750a418a2193d44cb5e160076484e98a10c11139127e13ea1474157a162d173318ab19h[19] = 0xab → low 4 bits = 11offset o = 11

Dynamic truncation: the low nibble of the last byte (0xab & 0x0F) picks the offset: o = 11.

750a418a2193d44cb5e160076484e98a10c11139127e13ea1474157a162d173318ab19h[11..14] = c1 39 7e eatop bit cleared → v = 1094287082

The four bytes h[11]…h[14] form a 32-bit number; clearing the top bit (& 0x7F) avoids sign issues: v = 1094287082.

750a418a2193d44cb5e160076484e98a10c11139127e13ea1474157a162d173318ab191094287082 mod 1000000code = 287082

The last 6 digits: 287082. The phone shows this; the server does the same computation and compares.

0b0ac1b72fa30842f5ef6307788229111093118b12c113c514e71504161617ff184419t = 75 → T = 2, o = 4code = 359152

30 seconds later T = 2: the digest and offset change and the code becomes 359152. A stolen old code goes stale within seconds; servers usually accept T − 1 … T + 1 to allow for clock drift.

Authentication

FIDO2, WebAuthn and Passkeys

BrowserSecurity keythe private key stays hereServerrandom challengeorigin-bound signaturesigned response (with private key)verifies the signaturewith the public keyThe signature includes the site's origin: a response given to a fake site is invalid on the real one → unphishable
  • FIDO2/WebAuthn and passkeys: a public/private key pair per site. The private key never leaves the device (TPM, Secure Enclave, security key).
  • The server stores only the public key: useless even if leaked. No shared secret, no password to reuse.
  • The signature is bound to the page's origin → phishing-resistant. Passkeys can sync across platforms; they were designed to replace passwords.

Authentication

Biometric Authentication

MetricDescription
False acceptance (FAR)Rate of accepting a stranger — a security risk
False rejection (FRR)Rate of rejecting the real user — a usability problem
Liveness testAgainst photos, masks, fake fingerprints (liveness)
IrrevocabilityA leaked fingerprint cannot be changed; a password can
  • Biometrics are good as an identity, bad as a secret: faces and fingerprints are not secret.
  • Proper use: locally on the device, as the factor that unlocks secure hardware (Face ID → Secure Enclave). The raw template must never leave the device.
  • Not suitable alone for remote authentication; usually combined with “something you have”.

Early devices that measured finger length are historical; today's systems use fingerprint ridge details (minutiae), 3D faces and iris patterns, and hardware-backed liveness testing is a must.

Authentication

How Do Attackers Get In?

RouteDescriptionCountermeasure
PhishingHarvesting passwords with a fake login pageFIDO2, user training
Credential stuffingTrying leaked passwords on other sitesUnique passwords, MFA
Brute force / dictionaryGuessing weak passwordsSlow hashes, rate limits, lockout
Unpatched vulnerabilityUsing a known flawFast patch management
Social engineeringFooling the help deskProcess and verification
Default/weak credentialsadmin/admin, open portsHardening
1 · Initial accessphishing, exploit2 · Privilege escalationlocal exploit → root3 · Lateral movementother machines4 · Persistencebackdoor, rootkitMFA and FIDO2 · patching and least privilege · network segmentation · EDR and secure boot: a defense at every link
An attack is usually a chain: initial access by phishing → privilege escalation through a local exploit → lateral movement → persistence (a backdoor). Defenses must be layered too; one broken link should not hand over the system.

Software Exploits · Step through the code

Stack Buffer Overflow: The Frame Step by Step

/* small, deliberately buggy example */
void A(char *in) {
    char buffer[8];
    strcpy(buffer, in);   // NO bounds check
    return;
}
A(argv[1]);               // in: user input
HIGH addressLOW address (stack grows down)argv[1] pointerreturn addressreturns ✓frame pointerbuffer[4..7]buffer[0..3]write direction

Each function call opens a frame on the stack: local variables, the saved frame pointer and the return address (where the CPU goes when the function ends).

HIGH addressLOW address (stack grows down)argv[1] pointerreturn addresscorrect ✓frame pointerbuffer[4..7]buffer[0..3]write direction

As long as the input fits, all is well: strcpy writes at most eight bytes, terminating \0 included, into buffer.

HIGH addressLOW address (stack grows down)argv[1] pointerreturn addressBOZULDU ✗frame pointerbuffer[4..7]buffer[0..3]write direction

Overflow: strcpy does not check the length, so extra bytes spill into neighboring cells — most dangerously, over the return address.

HIGH addressLOW address (stack grows down)argv[1] pointerreturn addressattacker ✗frame pointerbuffer[4..7]buffer[0..3]write direction

When the function returns, the CPU jumps to the corrupted address. If the attacker can choose it, they hijack the flow of execution. That is why untrusted input is never copied without a bounds check.

Software Exploits

Defenses Against Buffer Overflows

Stack canary

A random value before the return address; if it is corrupted before the return, the program is stopped (SSP).

DEP / NX

Data pages are marked nonexecutable; code injected onto the stack cannot run.

ASLR

Stack, heap and libraries at random addresses, so the attacker cannot know the target address. Used with PIE.

CFI

Control-flow integrity: indirect jumps may reach only valid targets.

Shadow stack / CET

A hardware-protected copy of the return address: Intel CET, ARM GCS. (ARM PAC/BTI are separate protections.)

Memory-safe language

Rust, Go, Java, C#: bounds and lifetime checks in the compiler/runtime. The most fundamental fix.

ROP and return-to-libc: once NX blocks injected code, attackers run existing code pieces (system() in libc, or short “gadget” sequences) by chaining return addresses. CFI, shadow stacks and ASLR are combined to make these attacks hard — no single defense suffices.

Software Exploits · Step through the code

Stack Canary: Code Trace

/* the compiler adds the canary lines */
void copy_in(const char *input) {
    long canary = __stack_chk_guard;
    char buf[8];
    strcpy(buf, input);
    if (canary != __stack_chk_guard)
        __stack_chk_fail();
}
/* input: "AAAAAAAABBBBBBBB" */
return address9611400000000000saved rbpf0c4203afd7f0000canary················buf[8]················low address →strcpy writes upward ↑call: return address and rbp pushed

copy_in was called: the return address (0x401196) and the caller's frame pointer are on the stack. The input is 16 characters; buf is 8 bytes.

return address9611400000000000saved rbpf0c4203afd7f0000canary005fb8d4e2713c9abuf[8]················low address →strcpy writes upward ↑canary = random; first byte 00

Compiler-inserted prologue code places a value chosen at random at process start between buf and the saved rbp. Its first byte is 0x00, so string functions can neither read and leak it nor write it verbatim.

return address9611400000000000saved rbpf0c4203afd7f0000canary005fb8d4e2713c9abuf[8]AAAAAAAAlow address →strcpy writes upward ↑strcpy: 8 bytes fit in buf

strcpy knows no length; it copies up to the \0 in the source. The first 8 bytes (A) fill buf.

return address9611400000000000saved rbpf0c4203afd7f0000canaryBBBBBBBBbuf[8]AAAAAAAAlow address →strcpy writes upward ↑overflow: canary overwritten

The next 8 bytes (B) land outside buf, on top of the canary: the canary is now 0x4242424242424242.

return address9611400000000000saved rbp00c4203afd7f0000canaryBBBBBBBBbuf[8]AAAAAAAAlow address →strcpy writes upward ↑terminating \0 spilled into saved rbp

The terminating \0 corrupts the first byte of the saved rbp. A slightly longer input would reach the return address next — but the only way there is through the canary.

return address9611400000000000saved rbp00c4203afd7f0000canaryBBBBBBBBbuf[8]AAAAAAAAlow address →strcpy writes upward ↑0x4242424242424242 ≠ 0x9a3c71e2d4b85f00

Before returning, the epilogue compares the canary on the stack with the value from process start: they differ.

return address9611400000000000saved rbp00c4203afd7f0000canaryBBBBBBBBbuf[8]AAAAAAAAlow address →strcpy writes upward ↑return never runs: the process is stopped

__stack_chk_fail prints the message and kills the process with SIGABRT. The corrupted frame is never returned to; the attack becomes a crash (lost availability, but no code execution).

Software Exploits

Other Memory and Injection Bugs

VulnerabilityWhy it is dangerousCountermeasure
Format stringprintf(user) — reads/writes memory via %n/%xprintf("%s", user)
Integer overflowlen = a + b wraps around, causing a small allocationChecked arithmetic, size types
Use after freeAccessing freed memory (UAF)Ownership (Rust), smart pointers, quarantine
Double freeThe same block freed twice; the heap is corruptedNULL after free; a hardened allocator
Out-of-bounds readMemory leaks like HeartbleedBounds checks, fuzzing
Command/SQL injectionInput gets mixed into code/queriesParameterized queries, escaping, allowlists

The common root cause: data mixed with code or control, and untrusted input used unchecked. Defense: validate input, keep data apart from code; memory-safe languages and fuzzing catch bugs early.

Software Exploits · Step through the code

The TOCTOU Race Condition

/* TOCTOU: a setuid-root helper */
if (access(path, W_OK) == 0) {   // CHECK
    fd = open(path, O_WRONLY);     // USE
    write(fd, data, n);
}
Victim (root)Attackeraccess("/tmp/f") → allowed

The program first checks with access(): “may ali write this file?” /tmp/f is an ordinary, permitted file. The answer is yes.

Victim (root)Attackeraccess("/tmp/f") → allowed(CPU yielded)ln -sf /etc/passwd /tmp/fwindow

There is a small time window between check and use. If the program loses the CPU right here...

Victim (root)Attackeraccess("/tmp/f") → allowed(CPU yielded)ln -sf /etc/passwd /tmp/fopen("/tmp/f")→ now /etc/passwd!window

The attacker deletes /tmp/f and puts a symbolic link to /etc/passwd in its place. Same file name, different target.

Victim (root)Attackeraccess("/tmp/f") → allowed(CPU yielded)ln -sf /etc/passwd /tmp/fopen("/tmp/f")→ now /etc/passwd!write(): passwd corruptedwindow

open() now opens the new target, not the name: /etc/passwd, with root rights. Time of check ≠ time of use (TOCTOU).

Victim (root)Attackeraccess("/tmp/f") → allowed(CPU yielded)ln -sf /etc/passwd /tmp/fopen("/tmp/f")→ now /etc/passwd!write(): passwd corruptedwindow

The fix: never resolve the name twice. Open the file once (open + O_NOFOLLOW), then check the same descriptor with fstat/faccessat; or use openat and drop privileges.

Insider Attacks

Insider Attacks: Logic Bombs, Back Doors

(a) Normal codeif (user_ok(name, pw)) grant_access();else deny();(b) Code with a back doorif (user_ok(name, pw) || eq(name, "zzz")) grant_access();else deny();“zzz” always gets in
Logic bomb

Malicious code triggered by a condition (a date, being fired). Often planted by a disgruntled employee.

Back door

A hidden entry that bypasses authentication. Can also be slipped into the supply chain.

Login spoofing

A fake login screen collects the user name and password, then hands off to the real one.

Against login spoofing, a secure attention key: in Windows, Ctrl+Alt+Del always summons the real kernel login screen, and no user program can imitate it. Against back doors: code review, two-person approval, reproducible builds.

Malware

Types of Malware

TypeSpreadGoal / behavior
VirusAttaches itself to another program; infects when the user runs itCorrupts files, spreads
WormSelf-propagating over the network; no user neededSpreads fast, consumes resources
Trojan horseHidden inside a useful-looking programOpens back doors, steals data
RansomwarePhishing, exploits, RDPEncrypts files, demands a ransom
RootkitInstalled after gaining privilegesHides itself and other malware
SpywareTrojans, bundled software, fake extensionsCollects data, sends it out
Bot / botnetWorm + back doorDDoS, spam, mining on remote command

The lines are blurry: modern malware usually combines several types (e.g. a bot that spreads like a worm, hides with a rootkit and carries ransomware). Spyware may change the browser home page or search engine, add toolbars and produce an unstoppable stream of ads.

Malware · Step by step

Parasitic Viruses

(a) clean programProgramThe entry point is redirected to the virus; the virus runs, then jumps to the real program

A clean executable: the real program code from start to end.

(b) prependedVirusProgramThe entry point is redirected to the virus; the virus runs, then jumps to the real program

Prepending virus: the virus code goes at the start and the entry point is redirected to it.

(c) appendedProgramVirusThe entry point is redirected to the virus; the virus runs, then jumps to the real program

Appending virus: the virus goes at the end, with a jump to it placed at the start (most common).

(d) spread over cavitiesPrgVPrgVPrgVPrgThe entry point is redirected to the virus; the virus runs, then jumps to the real program

Cavity virus: scatters itself over unused gaps in the program; the file size does not change, which makes it harder to detect.

Other kinds: companion viruses (a .com file that runs before the .exe of the same name), memory-resident, boot sector, device driver, macro (in documents) and source code viruses. Most are historical; modern malware uses scripts, in-memory (fileless) and supply-chain routes.

Malware · Step by step

Ransomware

Phishing attachment / exploitRuns, spreadsDeletes backupsEncrypts filesRansom note

Initial access usually comes from a phishing attachment, an exposed service or stolen RDP credentials.

Phishing attachment / exploitRuns, spreadsDeletes backupsEncrypts filesRansom note

The malware runs and moves laterally to reach as many machines as it can.

Phishing attachment / exploitRuns, spreadsDeletes backupsEncrypts filesRansom note

To block recovery it deletes shadow copies and backups (hence the need for offline/immutable backups).

Phishing attachment / exploitRuns, spreadsDeletes backupsEncrypts filesRansom noteFiles encrypted with AES

Files are locked with strong symmetric encryption; the key goes to the attacker's server.

Phishing attachment / exploitRuns, spreadsDeletes backupsEncrypts filesRansom noteFiles encrypted with AESPaying ≠ a guaranteed key; backups + patches are the only reliable defense

The ransom note appears. Paying does not guarantee the key and encourages new attacks. Defense: 3-2-1 backups, patching, MFA, segmentation, EDR.

Malware

Rootkits and Stealth

Applicationalters ls, ps outputLibraryhooks libc functionsKernelmodifies the syscall tableHypervisormoves the OS into a VMFirmwareUEFI/peripherals; survives reinstalldeeper = stealthier, harder to remove

A rootkit hides its presence: it conceals files, processes and network connections from the OS. The deeper it sits, the stronger it is, but the harder to install. Defenses at the kernel and below: signed drivers, secure boot, measured boot (TPM) and scanning from outside (offline).

Malware · Step by step

Antivirus and Evasion Techniques

(a) cleanProgramSignature: search for a known byte sequence

The clean program is the reference.

(b) infectedVirusProgramThe signature catches the virus

Signature scanning: look for the byte sequences (signatures) of known viruses. Fast, but catches only what is known.

(c) compressedVCompressed programCompression hides the signature; the scanner must unpack and look

If the virus compresses itself the signature is invisible; the scanner has to unpack the file and scan inside.

(d) encrypted virusDecryptorEncrypted virusProgramEncrypted body: only the decryptor is constant, so its signature is sought

Encrypted virus: the body is encrypted and only a small decryptor is in the clear; the scanner looks for the decryptor's signature.

(e) polymorphicMutating decryptorEncrypted virusProgramPolymorphic: the decryptor differs in every copy → signatures fail; behavior analysis is needed

Polymorphic virus: even the decryptor changes in every copy. Signatures are not enough → heuristic/behavioral analysis, running it in a sandbox and watching, is needed.

Signature scanning

Known patterns; fast, few false alarms, misses the new.

Integrity checking

Stores file hashes and notices changes.

Behavioral / heuristic

Watches for suspicious actions (encryption, injection).

Sandbox + cloud

Runs the suspect in an isolated environment and observes.

Defense and Isolation · Step by step

Code Signing

PublisherOperating system

The publisher computes the program's hash (the same as a digital signature, this time applied to code).

PublisherOperating systemProgramhash

The hash is signed with the publisher's private key; the signature ships with the program.

PublisherOperating systemProgramhashwith private keysignsignature

The user receives the program and signature; the publisher's certificate carries its public key.

PublisherOperating systemProgramhashwith private keysignsignatureProgramsignature

The OS recomputes the hash and verifies the signature with the public key; if they match, the program is unmodified and from that publisher.

PublisherOperating systemProgramhashwith private keysignsignatureProgramsignatureverifies the signatureverifytrusted ✓

App stores, driver signing, apt/dnf package signatures and platform “signed code only” policies all rest on this.

Defense and Isolation

Sandboxing and Confinement

JailPrisoner programJailerEvery system callis checked by the jailer;unauthorized onesare blockedModern sandbox• seccomp-bpf: allowed system calls• namespaces: separate PID/net/file views• cgroups: CPU/memory limits• capabilities dropped• SELinux/AppArmor profile

Sandboxing: run untrusted code in a restricted environment to limit the damage it can do. Browser tabs, mobile apps and server daemons all run sandboxed today. The Linux building blocks: seccomp, namespaces, cgroups and mandatory access control.

Defense and Isolation

Containers and Isolation

HardwareOne shared kernel (namespaces + cgroups + seccomp)Container 1ApplicationLibrariesContainer 2ApplicationLibrariesContainer 3ApplicationLibrariesContainer 4ApplicationLibraries
ContainerVirtual machine
IsolationKernel featuresHardware + hypervisor
OverheadLow, starts in secondsHigh, separate kernel
Attack surfaceKernel bug = escape riskStronger boundary

Defense and Isolation

Mandatory Access Control: SELinux and AppArmor

DAC (discretionary)

Permissions granted by the owner (rwx, ACLs). Users can change the permissions of their own files — and so can malware.

MAC (mandatory)

The system/administrator sets the policy; a process cannot loosen its own restrictions. BLP/Biba, SELinux, AppArmor.

SELinux

A label on every subject and object; fine-grained rules through type enforcement. Powerful but complex.

AppArmor

Path-based profiles; a per-application allowlist. Easier to write and understand.

httpd processopen(/etc/shadow)DAC: rwx, ACLowner can changeMAC: SELinux/AppArmorpolicy; process cannot change itallowEACCESif DAC denies, MAC is never askedboth must say “yes”

Mandatory access control (MAC) confines even a compromised process to what its profile allows: if a web server is exploited, the attacker reaches only the files that profile names. Android runs entirely in SELinux enforcing mode.

Hardware and Summary

Secure Boot and the TPM

Hardware root of trustimmutableUEFI firmwaresignature ✓Bootloadersignature ✓Kernelsignature ✓Operating systemsignature ✓Each stage runs the next only after verifying its signature (chain of trust)Unsigned/modifiedis not runTPMstores measurements, seals

Secure boot: each layer verifies the next one's signature, anchored in a hardware root of trust. The TPM (Trusted Platform Module) stores hashes of boot components (PCRs); with measured boot the disk-encryption key is released only if the system is in the expected state (remote attestation). Rootkits and bootkits try to break this chain.

Hardware and Summary

Spectre, Meltdown and Side Channels

Attacker processVictim / kernelsecret dataShared cacheCPU hardwarespeculative accessTiming measurementleaks bit by bitThe architectural result is rolled back, but the cache footprint stays → secret data is extracted

A side channel leaks data not directly but through an indirect physical effect (time, power, cache, sound). Meltdown and Spectre (2018) abuse CPUs' speculative execution: the footprint a canceled operation leaves in the cache is read by timing, breaking isolation. Mitigations: kernel page-table isolation (KPTI), microcode, compiler barriers — most at a performance cost.

Summary

Summary

TopicKey ideaToday
GoalsConfidentiality, integrity, availabilityZero trust, + authenticity/accountability
CryptographySymmetric + asymmetric + hash + signatureAES-GCM, X25519, Ed25519, TLS 1.3, ML-KEM
Access controlMatrix → ACLs or capabilitiesLeast privilege, Linux capabilities, MAC
Formal modelsBLP confidentiality, Biba integritySELinux MLS, Windows MIC
AuthenticationSalted slow hashes + MFAArgon2, FIDO2/passkeys
Memory safetyAn overflow corrupts the return addressCanary+NX+ASLR+CFI, Rust
IsolationSandboxes limit damageseccomp, namespaces, containers
HardwareChain of trust and measurementSecure boot, TPM; side channels

Summary

Review Questions

  1. Explain the three parts of the CIA triad, with an example breach of each.
  2. Compare the ACL and capability forms of the same access matrix: in which is revocation easy, and in which delegation? Why?
  3. Write the Bell-LaPadula and Biba rules (read/write directions). Which protects confidentiality, which integrity?
  4. Why store passwords with salted Argon2/bcrypt rather than plain SHA-256? What are the separate roles of the salt and the slowness?
  5. Why are FIDO2/passkeys phishing-resistant compared with password + TOTP?
  6. How is the return address hijacked in a stack buffer overflow? At which stage do ASLR, NX and the stack canary make it harder?
  7. Describe a TOCTOU race with the access()/open() example and give a fix.
  8. Distinguish a covert storage channel from a timing channel with examples. Why can they never be fully closed?
  9. Explain the secure boot chain of trust and the TPM's role. Where does it stop a rootkit?
  10. What is the fundamental difference between container and VM isolation? In which is a kernel bug more dangerous?
Answers
  1. Confidentiality: unauthorized reading (a data leak); integrity: unauthorized change (tampered records); availability: blocked service (DoS, ransomware).
  2. An ACL is a column kept with the object: “who can access?” and revocation are easy. A capability is a row kept with the subject: delegation and fast checks are easy, but revoking all copies is hard.
  3. BLP: no read up, no write down → confidentiality. Biba: no read down, no write up → integrity.
  4. Plain SHA-256 is fast; an attacker tries billions per second. The salt defeats precomputation and repeated hashes; slowness (and memory hardness) makes every guess expensive.
  5. The signature is bound to the site's origin and the private key never leaves the device; a response given to a fake site is invalid on the real one, and there is no shared secret.
  6. The overflowing input overwrites the return address; on return the CPU jumps there. The canary catches the corruption at return; NX refuses to run injected code; ASLR hides the target address.
  7. Between the check with access() and the use with open(), the attacker swaps the file for a symbolic link. Fix: open once and check the same descriptor (O_NOFOLLOW, fstat, openat).
  8. Storage: a file lock present or not. Timing: how long the CPU is kept busy. Every shared resource can be a channel, so closing them all is impossible; only the bandwidth can be reduced.
  9. Each layer verifies the next one's signature; the TPM stores boot measurements and seals keys to them. An unsigned or modified bootloader or kernel (a bootkit) is never run.
  10. A container shares the kernel (light, weak boundary); a VM uses a separate kernel and a hypervisor (heavy, strong boundary). A kernel bug can mean escape from a container, but usually not from a VM.

Summary

References

  • A. S. Tanenbaum, H. Bos, Modern Operating Systems, 4th ed. (2014) / 5th ed. (2023), Pearson — Chapter 9: Security.
  • A. Silberschatz, P. B. Galvin, G. Gagne, Operating System Concepts, 10th ed., Wiley, 2018 — Chapter 16: Security; Chapter 17: Protection.
  • R. H. Arpaci-Dusseau, A. C. Arpaci-Dusseau, Operating Systems: Three Easy Pieces (OSTEP), ostep.org — security chapters.
  • R. Anderson, Security Engineering, 3rd ed., Wiley, 2020.
  • J. H. Saltzer, M. D. Schroeder, “The Protection of Information in Computer Systems”, 1975 (least privilege and design principles).
  • D. E. Bell, L. J. LaPadula (1973); K. J. Biba (1977) — formal security models. B. W. Lampson, “A Note on the Confinement Problem”, 1973.
  • RFC 8446 (TLS 1.3), RFC 6238 (TOTP), RFC 9106 (Argon2); FIDO2/WebAuthn (W3C); NIST FIPS 203 (ML-KEM).
  • Aleph One, “Smashing the Stack for Fun and Profit”, Phrack 49, 1996; Kocher et al., “Spectre”; Lipp et al., “Meltdown”, 2018.

The End

Chapter 13: Security
Previous: Chapter 12: Multiprocessor Systems · Next: Chapter 14: Operating-System Design
1 / 1