Concurrency and the C Memory Model

1Threads

C11 added <threads.h>; POSIX has had pthreads since 1995. The C11 interface is cleaner but less widely implemented — notably absent from glibc until recently and from macOS — so pthreads remains what production code uses.

/* POSIX */
#include <pthread.h>

static void *worker(void *arg)
{
    int id = *(int *)arg;
    printf("thread %d\n", id);
    return NULL;
}

pthread_t t;
int id = 1;
pthread_create(&t, NULL, worker, &id);
pthread_join(t, NULL);                 /* wait for it to finish */

All threads share the heap, globals, and file descriptors. Each has its own stack, so locals are private. That asymmetry is the whole subject: anything reachable from more than one thread needs protection.

2Data races

int counter = 0;                       /* shared */

void *increment(void *arg)
{
    for (int i = 0; i < 1000000; i++) {
        counter++;                     /* a data race */
    }
    return NULL;
}

Run this on four threads and the result is not 4 000 000. counter++ is three operations — load, add, store — and two threads interleaving them lose updates.

The C standard's position is stronger than "you get a wrong number". Two threads accessing the same object without synchronization, where at least one writes, is a data race and therefore undefined behavior. The compiler may keep the variable in a register, reorder the accesses, or assume the race cannot happen. Reasoning about which interleaving you got is already the wrong question.

And, as week 38 said: volatile does not fix this. It forces the accesses to happen; it makes nothing atomic.

3Mutexes

pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;

pthread_mutex_lock(&lock);
counter++;                             /* the critical section */
pthread_mutex_unlock(&lock);

A mutex guarantees that only one thread is inside the critical section at a time, and — equally important — establishes the ordering that makes one thread's writes visible to the next.

Deadlock

/* thread 1 */            /* thread 2 */
lock(A);                  lock(B);
lock(B);   /* waits */    lock(A);   /* waits */

Each holds what the other needs. Nothing times out; the program simply stops. The standard prevention is a lock ordering: assign every mutex a rank and always acquire in increasing rank. If every thread obeys it, a cycle is impossible.

4Condition variables

A mutex protects data. A condition variable lets a thread wait for the data to reach a state — without spinning.

pthread_mutex_lock(&lock);
while (queue_is_empty(&q)) {           /* while, not if */
    pthread_cond_wait(&not_empty, &lock);
}
item = queue_pop(&q);
pthread_mutex_unlock(&lock);

pthread_cond_wait atomically releases the mutex and sleeps; on waking it reacquires the mutex before returning. That atomicity is the point — without it, a signal arriving between the test and the sleep would be missed.

The loop is mandatory. A waiting thread can wake without the condition being true: a spurious wakeup, permitted by the standard, or another thread taking the item first. if instead of while produces a bug that appears once a month under load.

5Atomics and the memory model

#include <stdatomic.h>

atomic_int counter = 0;
atomic_fetch_add(&counter, 1);         /* indivisible */
counter++;                              /* also atomic for an atomic type */

For a single counter, an atomic is faster than a mutex — one instruction rather than a system call in the contended case. For anything involving two related values, a mutex is required: two atomic operations are not one atomic operation.

Memory ordering

Modern processors and compilers reorder memory operations. The C memory model defines what one thread is guaranteed to observe of another.

Use the default. Relaxed ordering is correct only in narrow cases — a statistics counter nobody reads until the end — and reasoning about it is genuinely difficult. A wrong choice produces a bug that appears on one processor architecture and not another.

_Thread_local

_Thread_local int per_thread_errors = 0;   /* one instance per thread */

Each thread gets its own copy, so no synchronization is needed. This is how errno is implemented on a threaded system.

6Reentrancy

Several standard library functions keep static state and cannot be called from two threads at once. Week 14's strtok was the first example; there are others.

The _r suffix means reentrant: the caller supplies the buffer, so there is no shared state. errno itself is thread-local on every modern system, which is why week 27's advice still holds here.

7Worked example: a producer-consumer pipeline

#define _POSIX_C_SOURCE 200809L
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
#include <stdatomic.h>
#include <pthread.h>
#include <unistd.h>
#include <time.h>

#define QUEUE_CAPACITY 8
#define PRODUCERS      3
#define CONSUMERS      2
#define PER_PRODUCER   2000

/* ---------- 1. the race, demonstrated ---------- */

static long   racy_counter   = 0;
static atomic_long atomic_counter = 0;
static long   guarded_counter = 0;
static pthread_mutex_t guard = PTHREAD_MUTEX_INITIALIZER;

#define BUMPS 200000

static void *bump_all(void *arg)
{
    (void)arg;
    for (int i = 0; i < BUMPS; i++) {
        racy_counter++;                               /* DATA RACE */
        atomic_fetch_add(&atomic_counter, 1);         /* atomic     */
        pthread_mutex_lock(&guard);
        guarded_counter++;                            /* protected  */
        pthread_mutex_unlock(&guard);
    }
    return NULL;
}

/* ---------- 2. a bounded blocking queue ---------- */

typedef struct {
    int             items[QUEUE_CAPACITY];
    size_t          head, tail, count;
    bool            closed;
    pthread_mutex_t lock;
    pthread_cond_t  not_empty;
    pthread_cond_t  not_full;
} Queue;

static void queue_init(Queue *q)
{
    q->head = q->tail = q->count = 0;
    q->closed = false;
    pthread_mutex_init(&q->lock, NULL);
    pthread_cond_init(&q->not_empty, NULL);
    pthread_cond_init(&q->not_full, NULL);
}

static void queue_destroy(Queue *q)
{
    pthread_mutex_destroy(&q->lock);
    pthread_cond_destroy(&q->not_empty);
    pthread_cond_destroy(&q->not_full);
}

static void queue_push(Queue *q, int value)
{
    pthread_mutex_lock(&q->lock);
    while (q->count == QUEUE_CAPACITY && !q->closed) {
        pthread_cond_wait(&q->not_full, &q->lock);    /* while, not if */
    }
    if (!q->closed) {
        q->items[q->tail] = value;
        q->tail = (q->tail + 1) % QUEUE_CAPACITY;
        q->count++;
        pthread_cond_signal(&q->not_empty);
    }
    pthread_mutex_unlock(&q->lock);
}

static bool queue_pop(Queue *q, int *out)
{
    pthread_mutex_lock(&q->lock);
    while (q->count == 0 && !q->closed) {
        pthread_cond_wait(&q->not_empty, &q->lock);
    }
    if (q->count == 0) {                              /* closed and drained */
        pthread_mutex_unlock(&q->lock);
        return false;
    }
    *out = q->items[q->head];
    q->head = (q->head + 1) % QUEUE_CAPACITY;
    q->count--;
    pthread_cond_signal(&q->not_full);
    pthread_mutex_unlock(&q->lock);
    return true;
}

static void queue_close(Queue *q)
{
    pthread_mutex_lock(&q->lock);
    q->closed = true;
    pthread_cond_broadcast(&q->not_empty);            /* wake everyone */
    pthread_cond_broadcast(&q->not_full);
    pthread_mutex_unlock(&q->lock);
}

/* ---------- 3. the workers ---------- */

typedef struct { Queue *q; int id; } Worker;

static atomic_long produced = 0;
static atomic_long consumed = 0;
static atomic_long checksum = 0;

_Thread_local long my_items = 0;                      /* private per thread */

static void *producer(void *arg)
{
    Worker *w = arg;                                  /* its own object */
    for (int i = 0; i < PER_PRODUCER; i++) {
        int value = w->id * 1000000 + i;
        queue_push(w->q, value);
        atomic_fetch_add(&produced, 1);
        my_items++;
    }
    printf("  producer %d finished, %ld items (thread-local count)\n",
           w->id, my_items);
    return NULL;
}

static void *consumer(void *arg)
{
    Worker *w = arg;
    int value;
    while (queue_pop(w->q, &value)) {
        atomic_fetch_add(&consumed, 1);
        atomic_fetch_add(&checksum, value % 97);
        my_items++;
    }
    printf("  consumer %d finished, %ld items\n", w->id, my_items);
    return NULL;
}

int main(void)
{
    puts("== 1. three counters, one loop, four threads ==");
    pthread_t bumpers[4];
    for (int i = 0; i < 4; i++) pthread_create(&bumpers[i], NULL, bump_all, NULL);
    for (int i = 0; i < 4; i++) pthread_join(bumpers[i], NULL);

    long expected = 4L * BUMPS;
    printf("  expected          : %ld\n", expected);
    printf("  racy (unprotected): %ld  %s\n", racy_counter,
           racy_counter == expected ? "(lucky this run)" : "<-- updates lost");
    printf("  atomic            : %ld\n", atomic_load(&atomic_counter));
    printf("  mutex-protected   : %ld\n", guarded_counter);
    puts("  the racy result varies between runs; that is what UB looks like");

    puts("\n== 2. producer-consumer with a bounded queue ==");
    Queue q;
    queue_init(&q);

    pthread_t pt[PRODUCERS], ct[CONSUMERS];
    Worker    pw[PRODUCERS], cw[CONSUMERS];

    struct timespec t0;
    clock_gettime(CLOCK_MONOTONIC, &t0);

    for (int i = 0; i < CONSUMERS; i++) {
        cw[i] = (Worker){ .q = &q, .id = i };         /* its own object */
        pthread_create(&ct[i], NULL, consumer, &cw[i]);
    }
    for (int i = 0; i < PRODUCERS; i++) {
        pw[i] = (Worker){ .q = &q, .id = i };
        pthread_create(&pt[i], NULL, producer, &pw[i]);
    }

    for (int i = 0; i < PRODUCERS; i++) pthread_join(pt[i], NULL);
    queue_close(&q);                                   /* now the consumers end */
    for (int i = 0; i < CONSUMERS; i++) pthread_join(ct[i], NULL);

    struct timespec t1;
    clock_gettime(CLOCK_MONOTONIC, &t1);
    double secs = (double)(t1.tv_sec - t0.tv_sec)
                + (double)(t1.tv_nsec - t0.tv_nsec) / 1e9;

    printf("  produced %ld, consumed %ld, checksum %ld, %.3f s\n",
           atomic_load(&produced), atomic_load(&consumed),
           atomic_load(&checksum), secs);
    printf("  queue capacity was %d, so producers blocked whenever\n",
           QUEUE_CAPACITY);
    puts("  consumers fell behind — that is back-pressure, for free");

    queue_destroy(&q);

    puts("\n== 3. reentrancy ==");
    time_t now = time(NULL);
    struct tm safe;
    localtime_r(&now, &safe);                          /* caller's buffer */
    char stamp[64];
    strftime(stamp, sizeof stamp, "%H:%M:%S", &safe);
    printf("  localtime_r : %s\n", stamp);
    puts("  localtime() returns a pointer to a shared static buffer;");
    puts("  two threads calling it would overwrite each other's result");

    char text[] = "alpha,beta,gamma";
    char *save = NULL;
    printf("  strtok_r    :");
    for (char *tok = strtok_r(text, ",", &save); tok != NULL;
         tok = strtok_r(NULL, ",", &save)) {
        printf(" [%s]", tok);
    }
    puts("\n  the save pointer replaces strtok's hidden static state");

    return EXIT_SUCCESS;
}

gcc -std=c17 -Wall -Wextra -g -pthread -o concurrent concurrent.c
./concurrent

-pthread rather than -lpthread: it sets the required preprocessor definitions as well as linking the library.

Let the sanitizer name the race

gcc -std=c17 -Wall -Wextra -g -pthread -fsanitize=thread -o concurrent_tsan concurrent.c
./concurrent_tsan

WARNING: ThreadSanitizer: data race (pid=12345)
  Write of size 8 at 0x55a... by thread T2:
    #0 bump_all concurrent.c:28

  Previous write of size 8 at 0x55a... by thread T1:
    #0 bump_all concurrent.c:28

  Location is global 'racy_counter' of size 8

It names the variable, both threads, and the line. ThreadSanitizer reports races that did not manifest on this run — it reasons about the synchronization, not about what happened to interleave. That is what makes it worth the five- to fifteen-fold slowdown. Note that it cannot be combined with AddressSanitizer; run them in separate builds.

Run the racy counter ten times

for i in $(seq 10); do ./concurrent | grep 'racy'; done

Ten different numbers, all below 800 000. Every lost update is one thread's store overwriting another's. Then raise BUMPS and watch the loss grow — and note that at low contention the racy counter is sometimes exactly right, which is precisely why these bugs reach production.

Cause a deadlock deliberately

static pthread_mutex_t a = PTHREAD_MUTEX_INITIALIZER;
static pthread_mutex_t b = PTHREAD_MUTEX_INITIALIZER;

static void *ab(void *_) { (void)_;
    pthread_mutex_lock(&a); sleep(1); pthread_mutex_lock(&b);
    pthread_mutex_unlock(&b); pthread_mutex_unlock(&a); return NULL; }

static void *ba(void *_) { (void)_;
    pthread_mutex_lock(&b); sleep(1); pthread_mutex_lock(&a);
    pthread_mutex_unlock(&a); pthread_mutex_unlock(&b); return NULL; }

The program hangs with no message. Attach a debugger to see why:

gdb -p $(pgrep concurrent)
(gdb) thread apply all bt

Both threads sit in pthread_mutex_lock, each holding what the other wants. Now make both functions lock a then b — the same order — and the deadlock becomes impossible. That single rule is the whole prevention.

Break the condition-variable loop

Change while (q->count == 0 && !q->closed) to if. The program usually still works, and occasionally a consumer wakes to find the queue empty, pops garbage, and the counts disagree. Run it a hundred times to see it. This is the shape of a concurrency bug: correct almost always, wrong rarely, and impossible to reproduce on demand — which is why the rule is followed unconditionally rather than tested.

8Common mistakes

10Where this leads

That completes the Advanced level, and with it the language, the standard library, the operating system interface, and concurrency. The Professional level starts from a different premise: not what C can express, but what working in C with other people actually requires. Week 42 begins by reading excellent C written by someone else.