Undefined Behavior and Secure Coding

1Three categories, often confused

The first two are survivable: you can write portable code by not depending on unspecified choices, and you can look up implementation-defined ones. The third is different in kind. The standard imposes no requirement on a program that exhibits it — not on the operation, not on the surrounding code, not on anything that happened before.

The practical consequence is the part people miss. Undefined behavior does not mean "you get a wrong value here". It means the compiler may assume it does not occur, and optimize the whole function on that assumption.

2How an optimizer exploits it

int check(int *p)
{
    int value = *p;              /* dereference: if p were null, UB */
    if (p == NULL) {             /* so the compiler concludes p != NULL */
        return -1;               /* and deletes this branch entirely */
    }
    return value;
}

The reasoning is valid. Dereferencing a null pointer is undefined, so a correct program never reaches this code with p == NULL; therefore the test is redundant; therefore it can be removed. Your null check is gone, and at -O0 it was still there.

The same mechanism eliminates overflow checks:

/* A check that cannot work */
if (a + b < a) {                 /* assumes signed overflow wraps */
    return ERROR_OVERFLOW;       /* deleted: signed overflow is UB,   */
}                                /* so a + b < a is assumed false     */

/* A check that works: test before the operation */
if (b > 0 && a > INT_MAX - b) {
    return ERROR_OVERFLOW;
}
if (b < 0 && a < INT_MIN - b) {
    return ERROR_OVERFLOW;
}
int sum = a + b;                 /* now provably safe */

This is not hypothetical. A 2009 Linux kernel vulnerability arose from exactly this pattern: the compiler removed a null check that followed a dereference, leaving an exploitable hole in code whose source looked correct.

3The common sources

There are over two hundred in the standard. These ten cover almost everything encountered in practice.

4Strict aliasing

The strict aliasing rule says an object may only be accessed through an lvalue of a compatible type — with char as the universal exception. The compiler relies on it to know that a write through an int * cannot disturb a float, which allows it to keep values in registers across the store.

/* Violation: reading a float's bytes through an int pointer */
float f = 1.0f;
int bits = *(int *)&f;              /* undefined behavior */

It usually appears to work, which is why it survives in old code. It breaks when the optimizer reorders the load and the store because it has concluded they cannot refer to the same object.

Two correct ways to reinterpret bytes:

/* 1. memcpy — the standard idiom; optimizes to nothing */
int bits;
memcpy(&bits, &f, sizeof bits);

/* 2. a union — explicitly permitted in C (unlike C++) */
union { float f; int bits; } u = { .f = 1.0f };
int bits = u.bits;

The memcpy form looks wasteful and is not: every mainstream compiler recognizes a small fixed-size copy and emits a single register move. Confirm it with week 34's assembly technique.

restrict

restrict is a promise in the other direction: within this scope, the object reached through this pointer is reached through no other.

void add_arrays(int *restrict dst, const int *restrict a,
                const int *restrict b, size_t n)
{
    for (size_t i = 0; i < n; i++) {
        dst[i] = a[i] + b[i];
    }
}

Without it the compiler must assume dst might overlap a, so each store could change a later load — and it must reload on every iteration and cannot vectorize. With it, the loop can be vectorized.

The promise is unchecked. Pass overlapping pointers and the behavior is undefined, with no diagnostic. This is exactly the distinction between memcpy (whose parameters are restrict) and memmove (whose are not), which week 14 introduced from the other side.

5From undefined behavior to vulnerability

The integer-overflow allocation bug

/* count comes from the network */
size_t bytes = count * sizeof(Record);      /* can wrap */
Record *r = malloc(bytes);                  /* far too small */
for (size_t i = 0; i < count; i++) {
    r[i] = read_record();                   /* writes way past the end */
}

With sizeof(Record) of 48 and a count near SIZE_MAX / 48, the multiplication wraps to a small number. The allocation succeeds, the loop writes gigabytes, and the attacker chose the data. Two fixes: use calloc(count, sizeof(Record)), which is required to detect the overflow, or check count > SIZE_MAX / sizeof(Record) first.

The format-string bug

printf(user_input);              /* catastrophic */
printf("%s", user_input);        /* correct */

If the input contains %s, printf reads an argument that was never passed. If it contains %n, printf writes to an address taken from the stack. This has been a remote-code-execution vector since the 1990s and -Wformat-security catches it.

6Defences that work

And the practices: validate every input at the boundary, bound every copy, use calloc for untrusted counts, prefer snprintf, set pointers to NULL after freeing, and treat every compiler warning as a defect.

7Worked example: correct at -O0, broken at -O2

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <stdint.h>
#include <stdbool.h>

/* ---------- 1. the overflow check that disappears ---------- */

static bool add_checked_broken(int a, int b, int *out)
{
    int sum = a + b;               /* UB on overflow */
    if (sum < a) {                 /* assumes wrapping; may be deleted */
        return false;
    }
    *out = sum;
    return true;
}

static bool add_checked(int a, int b, int *out)
{
    if (b > 0 && a > INT_MAX - b) return false;   /* checked BEFORE */
    if (b < 0 && a < INT_MIN - b) return false;
    *out = a + b;
    return true;
}

/* ---------- 2. the null check that disappears ---------- */

static int deref_then_check(int *p)
{
    int value = *p;                /* if p were NULL this is UB ... */
    if (p == NULL) {               /* ... so the compiler may delete this */
        return -1;
    }
    return value;
}

static int check_then_deref(int *p)
{
    if (p == NULL) return -1;      /* the only order that works */
    return *p;
}

/* ---------- 3. strict aliasing ---------- */

static uint32_t bits_violating(float f)
{
    return *(uint32_t *)&f;        /* undefined behavior */
}

static uint32_t bits_memcpy(float f)
{
    uint32_t bits;
    memcpy(&bits, &f, sizeof bits);   /* correct, and free at -O2 */
    return bits;
}

static uint32_t bits_union(float f)
{
    union { float f; uint32_t u; } conv = { .f = f };
    return conv.u;                    /* correct in C */
}

/* ---------- 4. the allocation-size overflow ---------- */

typedef struct { int id; char pad[44]; } Record;   /* 48 bytes */

static Record *allocate_broken(size_t count)
{
    return malloc(count * sizeof(Record));         /* can wrap */
}

static Record *allocate_checked(size_t count)
{
    if (count > SIZE_MAX / sizeof(Record)) {
        return NULL;                               /* refuse */
    }
    return calloc(count, sizeof(Record));          /* checks as well */
}

/* ---------- 5. format strings ---------- */

static void print_broken(const char *user)
{
    printf(user);                  /* -Wformat-security warns */
    putchar('\n');
}

static void print_safe(const char *user)
{
    printf("%s\n", user);
}

int main(void)
{
    printf("built at optimization level: ");
#ifdef __OPTIMIZE__
    puts("optimized");
#else
    puts("-O0");
#endif

    puts("\n== 1. signed overflow ==");
    int result = 0;
    printf("  broken check, INT_MAX + 1 : %s\n",
           add_checked_broken(INT_MAX, 1, &result) ? "ACCEPTED (wrong)"
                                                   : "rejected");
    printf("  correct check, INT_MAX + 1: %s\n",
           add_checked(INT_MAX, 1, &result) ? "ACCEPTED (wrong)" : "rejected");
    printf("  correct check, 100 + 200  : %s (%d)\n",
           add_checked(100, 200, &result) ? "accepted" : "rejected", result);
    puts("  compile at -O2 and compare: the broken check may vanish");

    puts("\n== 2. null checks ==");
    int value = 42;
    printf("  check_then_deref(&value) = %d\n", check_then_deref(&value));
    printf("  check_then_deref(NULL)   = %d\n", check_then_deref(NULL));
    puts("  deref_then_check(NULL) is NOT called here: at -O2 the test");
    puts("  may be gone and it would simply crash");

    puts("\n== 3. strict aliasing ==");
    float f = 1.0f;
    printf("  memcpy : 0x%08X\n", bits_memcpy(f));
    printf("  union  : 0x%08X\n", bits_union(f));
    printf("  cast   : 0x%08X  <-- undefined behavior, works by luck\n",
           bits_violating(f));
    puts("  all three agree today; only two are guaranteed to");

    puts("\n== 4. allocation size overflow ==");
    size_t huge = SIZE_MAX / 48 + 2;
    printf("  requesting %zu records of %zu bytes\n", huge, sizeof(Record));
    printf("  count * sizeof wraps to %zu bytes\n", huge * sizeof(Record));
    Record *bad = allocate_broken(huge);
    printf("  malloc(wrapped) : %s  <-- a tiny buffer for a huge count\n",
           bad ? "SUCCEEDED" : "failed");
    free(bad);
    Record *good = allocate_checked(huge);
    printf("  checked version : %s\n", good ? "succeeded" : "refused");
    free(good);

    puts("\n== 5. format strings ==");
    print_safe("a normal message");
    print_safe("%s %s %s %n");     /* harmless: it is just data */
    puts("  the same string passed to printf() directly would read");
    puts("  arguments that were never pushed, and %n would WRITE");

    puts("\n== 6. shifting too far ==");
    unsigned shift_by = 32;
    uint32_t one = 1;
    printf("  1u << 32 on a 32-bit type is undefined\n");
    if (shift_by < 32) {           /* the guard that makes it defined */
        printf("  guarded: %u\n", one << shift_by);
    } else {
        puts("  guarded: refused, as it must be");
    }

    return EXIT_SUCCESS;
}

Run it at both levels and compare

gcc -std=c17 -Wall -Wextra -Wformat-security -O0 -o ub0 ub.c
gcc -std=c17 -Wall -Wextra -Wformat-security -O2 -o ub2 ub.c
./ub0 > out0.txt
./ub2 > out2.txt
diff out0.txt out2.txt

On many GCC versions the "broken check" line differs between the two builds: at -O0 the addition wraps and the test happens to catch it; at -O2 the compiler has concluded the test can never be true and removed it. Same source, same inputs, different behavior — which is the definition of a program you cannot reason about.

Let the sanitizer name it

gcc -std=c17 -Wall -Wextra -g -fsanitize=undefined -o ubsan ub.c
./ubsan

ub.c:14:15: runtime error: signed integer overflow: 2147483647 + 1
            cannot be represented in type 'int'

File, line, operation, and values. UBSan is the only practical way to find this class of bug, because nothing else reports it — the program does not crash, it just becomes wrong.

See memcpy cost nothing

gcc -O2 -S ub.c -o ub.s
sed -n '/bits_memcpy:/,/ret/p' ub.s
sed -n '/bits_violating:/,/ret/p' ub.s

Both functions typically compile to the same one or two instructions. The memcpy version is correct and free; the cast version is undefined and no faster. There is no trade-off to make.

Watch the format-string warning

ub.c:118:5: warning: format not a string literal and no format
            arguments [-Wformat-security]

-Wformat-security is not in -Wall or -Wextra. Add it to the warning profile from week 29 — it catches a vulnerability class for one flag.

Confirm the allocation wrap

The output prints the requested count, the size of a record, and the wrapped product — a number far smaller than either. malloc succeeds because the request is tiny; the loop that follows would write past the end of a handful of bytes. calloc is required by the standard to detect this and return NULL, which is why it is the right allocator for any count that came from outside your program.

8Common mistakes

10Where this leads

Week 37 covers the neighbouring category: behavior that is defined but differs between platforms — type widths, byte order, alignment, character encoding. That is what stands between code that runs here and code that runs everywhere, and it is the prerequisite for the portable binary format the embedded weeks depend on.