Embedded Debugging and Testing

1What is missing

The strategy that follows: test as much as possible where the tools are. Only what genuinely depends on hardware needs to run on hardware, and with the right structure that is a small fraction of the code.

2On-chip debugging

A debug probe connects to the processor's built-in debug unit over JTAG or the two-wire SWD, and presents a gdb server. QEMU offers the same interface with -s, which is why everything in this block has been debuggable without hardware.

# the simulator, halted at reset
qemu-system-arm -M lm3s6965evb -nographic -kernel firmware.elf -S -s

# real hardware, with an ST-Link or J-Link probe
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

gdb-multiarch firmware.elf
(gdb) target remote :1234        # or :3333 for OpenOCD
(gdb) load                       # program the flash (hardware only)
(gdb) monitor reset halt
(gdb) break main
(gdb) continue

From there it is week 35's debugger: breakpoints, stepping, backtrace, info locals. Two additions matter here.

Hardware breakpoints are a scarce resource. Code in flash cannot be patched with a software breakpoint, so the processor's comparators are used — typically six on Cortex-M, and four watchpoints. GDB will refuse the seventh.

Halting changes the system. Stopping the core does not stop a motor, a timer, or a serial peer. Resume after a breakpoint and you may find a UART overrun, a watchdog reset, or a mechanism that has run away. For timing-sensitive code, tracing beats breakpoints.

(gdb) print/x *(UART_Type *)0x4000C000     # a peripheral, as a struct
(gdb) x/16xw 0x20000000                    # raw memory
(gdb) info registers
(gdb) p/x $sp                              # where is the stack now
(gdb) monitor reset halt                   # OpenOCD: restart cleanly

3Tracing

Every tracing mechanism perturbs what it measures. The question is by how much.

The trace buffer is the workhorse: write timestamped events into a circular array in RAM and read it afterwards with the debugger. It is fast enough to leave enabled in production, and it survives a crash if you place it in a region the startup code does not clear — week 52's retained RAM.

typedef struct { uint32_t tick; uint16_t id; uint16_t value; } TraceEvent;

static TraceEvent trace[256];
static volatile uint8_t trace_index;

static inline void trace_log(uint16_t id, uint16_t value)
{
    uint8_t i = trace_index++;              /* wraps at 256 naturally */
    trace[i].tick  = SYST_CVR;              /* cycle-resolution timer */
    trace[i].id    = id;
    trace[i].value = value;
}

Then in GDB: print trace, or dump it over the UART on demand. A toggled GPIO pin is even cheaper and, with an oscilloscope or logic analyzer, measures interrupt latency directly — set the pin on ISR entry, clear it on exit, and read the width.

4A hardware abstraction layer

This is the structural idea that makes the rest possible. Firmware written as one layer — logic and register access interleaved — can only run on the device. Separate them and most of the code becomes ordinary C.

/* hal.h — the boundary */
typedef struct {
    bool     (*uart_read)(uint8_t *out);
    void     (*uart_write)(uint8_t byte);
    uint32_t (*ticks)(void);
    void     (*led_set)(bool on);
} Hal;

/* logic.h — pure logic, no registers anywhere */
void protocol_init(Protocol *p, const Hal *hal);
void protocol_poll(Protocol *p);

On the device, the Hal is filled with the drivers from week 50. In a host test it is filled with functions that read from an array and record what was written — and the protocol logic, which is where the bugs are, runs under AddressSanitizer with a real test framework.

The goal is to push as much as possible into the first row. Firmware where 80% of the logic is host-testable is a different engineering experience from firmware where none of it is.

5Worked example: the week 50 driver, made testable

Split the layer

/* hal.h */
#ifndef HAL_H
#define HAL_H
#include <stdint.h>
#include <stdbool.h>

typedef struct {
    bool     (*uart_read)(void *ctx, uint8_t *out);
    void     (*uart_write)(void *ctx, uint8_t byte);
    uint32_t (*ticks)(void *ctx);
    void     *ctx;
} Hal;
#endif

/* protocol.h — the logic under test */
#ifndef PROTOCOL_H
#define PROTOCOL_H
#include "hal.h"

#define PROTO_MAX_LINE 64

typedef enum { PROTO_OK, PROTO_OVERFLOW, PROTO_TIMEOUT } ProtoStatus;

typedef struct {
    const Hal  *hal;
    char        line[PROTO_MAX_LINE];
    uint8_t     used;
    uint32_t    last_byte_tick;
    uint32_t    lines_handled;
    ProtoStatus last_status;
} Protocol;

void        protocol_init(Protocol *p, const Hal *hal);
ProtoStatus protocol_poll(Protocol *p);
#endif

/* protocol.c — no register touches anywhere */
#include "protocol.h"
#include <string.h>
#include <stdio.h>

#define TIMEOUT_TICKS 1000

void protocol_init(Protocol *p, const Hal *hal)
{
    memset(p, 0, sizeof *p);
    p->hal = hal;
    p->last_status = PROTO_OK;
}

static void reply(Protocol *p, const char *s)
{
    for (; *s; s++) p->hal->uart_write(p->hal->ctx, (uint8_t)*s);
}

ProtoStatus protocol_poll(Protocol *p)
{
    uint8_t byte;
    uint32_t now = p->hal->ticks(p->hal->ctx);

    while (p->hal->uart_read(p->hal->ctx, &byte)) {
        p->last_byte_tick = now;

        if (byte == '\n') {
            p->line[p->used] = '\0';
            char out[PROTO_MAX_LINE + 16];
            int n = snprintf(out, sizeof out, "%u:%s\n",
                             (unsigned)++p->lines_handled, p->line);
            if (n > 0) reply(p, out);
            p->used = 0;
            continue;
        }

        if (p->used >= PROTO_MAX_LINE - 1) {     /* the bound that matters */
            reply(p, "overflow\n");
            p->used = 0;
            p->last_status = PROTO_OVERFLOW;
            return PROTO_OVERFLOW;
        }
        p->line[p->used++] = (char)byte;
    }

    if (p->used > 0 && now - p->last_byte_tick > TIMEOUT_TICKS) {
        reply(p, "timeout\n");
        p->used = 0;
        p->last_status = PROTO_TIMEOUT;
        return PROTO_TIMEOUT;
    }
    return PROTO_OK;
}

The device implementation

/* hal_device.c — only this file knows about registers */
#include "hal.h"

#define UART0_DR (*(volatile uint32_t *)0x4000C000u)
#define UART0_FR (*(volatile uint32_t *)0x4000C018u)
#define SYST_CVR (*(volatile uint32_t *)0xE000E018u)

static bool dev_read(void *ctx, uint8_t *out)
{
    (void)ctx;
    if (UART0_FR & (1u << 4)) return false;      /* RXFE: empty */
    *out = (uint8_t)(UART0_DR & 0xFFu);
    return true;
}

static void dev_write(void *ctx, uint8_t byte)
{
    (void)ctx;
    while (UART0_FR & (1u << 5)) { }            /* TXFF: full */
    UART0_DR = byte;
}

static uint32_t dev_ticks(void *ctx) { (void)ctx; return SYST_CVR; }

const Hal device_hal = {
    .uart_read  = dev_read,
    .uart_write = dev_write,
    .ticks      = dev_ticks,
    .ctx        = NULL
};

The fake, and the tests

/* test_protocol.c — builds and runs on the host */
#include "protocol.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static int checks = 0, failures = 0;

#define CHECK(cond) do {                                          \
    checks++;                                                     \
    if (!(cond)) { failures++;                                    \
        printf("  FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); }\
    else printf("  ok   %s\n", #cond);                            \
} while (0)

/* A scripted hardware layer: input from a string, output captured. */
typedef struct {
    const char *input;
    size_t      input_pos;
    char        output[1024];
    size_t      output_len;
    uint32_t    now;                 /* time we control exactly */
} Fake;

static bool fake_read(void *ctx, uint8_t *out)
{
    Fake *f = ctx;
    if (f->input[f->input_pos] == '\0') return false;
    *out = (uint8_t)f->input[f->input_pos++];
    return true;
}

static void fake_write(void *ctx, uint8_t byte)
{
    Fake *f = ctx;
    if (f->output_len < sizeof f->output - 1) {
        f->output[f->output_len++] = (char)byte;
        f->output[f->output_len] = '\0';
    }
}

static uint32_t fake_ticks(void *ctx) { return ((Fake *)ctx)->now; }

static void fake_init(Fake *f, const char *input, Hal *hal)
{
    memset(f, 0, sizeof *f);
    f->input = input;
    hal->uart_read  = fake_read;
    hal->uart_write = fake_write;
    hal->ticks      = fake_ticks;
    hal->ctx        = f;
}

static void test_single_line(void)
{
    puts("\n-- one line --");
    Fake f; Hal hal; Protocol p;
    fake_init(&f, "hello\n", &hal);
    protocol_init(&p, &hal);

    CHECK(protocol_poll(&p) == PROTO_OK);
    CHECK(strcmp(f.output, "1:hello\n") == 0);
    CHECK(p.lines_handled == 1);
}

static void test_split_across_polls(void)
{
    puts("\n-- a line arriving in pieces --");
    Fake f; Hal hal; Protocol p;
    fake_init(&f, "hel", &hal);
    protocol_init(&p, &hal);

    protocol_poll(&p);
    CHECK(f.output_len == 0);            /* nothing yet: no newline */

    f.input = "lo\n"; f.input_pos = 0;   /* the rest arrives */
    protocol_poll(&p);
    CHECK(strcmp(f.output, "1:hello\n") == 0);
}

static void test_overflow(void)
{
    puts("\n-- a line longer than the buffer --");
    char big[PROTO_MAX_LINE + 20];
    memset(big, 'x', sizeof big - 2);
    big[sizeof big - 2] = '\n';
    big[sizeof big - 1] = '\0';

    Fake f; Hal hal; Protocol p;
    fake_init(&f, big, &hal);
    protocol_init(&p, &hal);

    CHECK(protocol_poll(&p) == PROTO_OVERFLOW);
    CHECK(strstr(f.output, "overflow") != NULL);
    CHECK(p.used == 0);                  /* reset, not left half-full */
}

static void test_timeout(void)
{
    puts("\n-- a partial line that goes quiet --");
    Fake f; Hal hal; Protocol p;
    fake_init(&f, "partial", &hal);
    protocol_init(&p, &hal);

    f.now = 100;
    CHECK(protocol_poll(&p) == PROTO_OK);    /* not yet */

    f.now = 100 + 2000;                      /* time advances instantly */
    CHECK(protocol_poll(&p) == PROTO_TIMEOUT);
    CHECK(strstr(f.output, "timeout") != NULL);
}

static void test_several_lines_one_poll(void)
{
    puts("\n-- three lines in one burst --");
    Fake f; Hal hal; Protocol p;
    fake_init(&f, "a\nb\nc\n", &hal);
    protocol_init(&p, &hal);

    protocol_poll(&p);
    CHECK(strcmp(f.output, "1:a\n2:b\n3:c\n") == 0);
    CHECK(p.lines_handled == 3);
}

int main(void)
{
    puts("host tests for the protocol layer");
    test_single_line();
    test_split_across_polls();
    test_overflow();
    test_timeout();
    test_several_lines_one_poll();

    printf("\n%d checks, %d failed\n", checks, failures);
    return failures == 0 ? EXIT_SUCCESS : EXIT_FAILURE;
}

gcc -std=c17 -Wall -Wextra -g -fsanitize=address,undefined \
    -o test_protocol test_protocol.c protocol.c
./test_protocol ; echo "status $?"

What the host build buys

Sanitizers on firmware logic. Change the bound to p->used > PROTO_MAX_LINE - 1 — an off-by-one — and rerun. On the device this would silently corrupt whatever follows line in the structure. On the host:

ERROR: AddressSanitizer: stack-buffer-overflow
WRITE of size 1 at 0x7ffd… in protocol_poll protocol.c:41

File, line, and the variable. That diagnosis is unavailable on the target at any price.

Time under your control. The timeout test sets f.now to a value two thousand ticks later and polls. Testing that path on hardware means waiting; here it takes nanoseconds and is deterministic.

Input under your control. The split-line test reproduces a condition that on hardware depends on precise UART timing. As a fake it is two assignments.

Fuzzing. Week 48 applies directly, because the logic is now a pure function of its input:

/* fuzz_protocol.c */
#include "protocol.h"
#include <string.h>

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
    if (size > 4096) return 0;
    char *input = malloc(size + 1);
    memcpy(input, data, size);
    input[size] = '\0';

    Fake f; Hal hal; Protocol p;
    fake_init(&f, input, &hal);
    protocol_init(&p, &hal);
    for (int i = 0; i < 4; i++) protocol_poll(&p);

    free(input);
    return 0;
}

clang -g -O1 -fsanitize=fuzzer,address,undefined \
      -o fuzz_protocol fuzz_protocol.c protocol.c
./fuzz_protocol corpus/

A fuzzer on firmware. That is the payoff of the abstraction layer, and it is not available to code that reads registers inline.

Simulator in the loop

Host tests cannot cover the driver. Run the real image under QEMU, automatically:

#!/bin/sh
# run_target_test.sh — exit 0 if the firmware behaves
set -e
arm-none-eabi-gcc -mcpu=cortex-m3 -mthumb -ffreestanding -nostdlib -O2 -g \
    -T firmware.ld -o firmware.elf startup.c hal_device.c protocol.c main.c

OUT=$(printf 'hello\nworld\nquit\n' | timeout 10 \
      qemu-system-arm -M lm3s6965evb -nographic -kernel firmware.elf)

echo "$OUT" | grep -q '1:hello' || { echo "FAIL: first line";  exit 1; }
echo "$OUT" | grep -q '2:world' || { echo "FAIL: second line"; exit 1; }
echo "target test passed"

# .github/workflows/firmware.yml
- name: host tests
  run: make test-host          # sanitizers, coverage, fuzz corpus

- name: target tests
  run: ./run_target_test.sh    # the real image, under the simulator

- name: size check
  run: |
    arm-none-eabi-size firmware.elf
    test $(arm-none-eabi-size -A firmware.elf | awk '/\.text/{print $2}') -lt 32768

Three gates: the logic under sanitizers, the real image under emulation, and a flash budget that fails the build before the part does. Week 35's CI, adapted to a target that is not the build machine.

Debug the real image

qemu-system-arm -M lm3s6965evb -nographic -kernel firmware.elf -S -s &
gdb-multiarch firmware.elf
(gdb) target remote :1234
(gdb) break protocol_poll
(gdb) continue
(gdb) print *p
(gdb) print p->line
(gdb) watch p->used             # who changes it, and when
(gdb) print/x *(UART_Type *)0x4000C000

Identical to week 35's workflow, on a machine that does not exist. Once -s is replaced by OpenOCD's port, the same commands drive real silicon — which is what makes the simulator a rehearsal rather than a substitute.

6Common mistakes

8Where this leads

Week 56 closes the course with the longest timescale of all: code that outlives its authors. Bringing legacy C under test before changing it, refactoring without altering behavior, migrating across standards and compilers, and deciding what to do about technical debt.