Undefined Behavior & Memory Bugs
Overflows, use-after-free, why they happen
- › Define undefined behavior (UB) and why the C standard leaves it undefined
- › List the UB you'll actually hit: overflow, out-of-bounds, use-after-free, null deref, uninitialized reads
- › Explain why the compiler is allowed to assume UB never happens — and the surprises that follow
- › Use sanitizers to catch UB, and connect these bugs to the exploits ahead
The most misunderstood idea in C
Undefined behavior is any operation the C standard declares to have *no defined meaning at all*. Not "implementation-defined," not "unspecified" — genuinely undefined: the standard makes zero promises about what happens. And here's the part that surprises everyone: the compiler is allowed to assume UB never occurs, and optimizes on that assumption. So UB doesn't just "do something weird" — it can make the compiler delete your checks, and it's the root of nearly every serious security vulnerability in C and C++. Understanding UB is understanding why C is powerful, fast, and dangerous.
The UB you'll actually meet
You've already brushed against all of these; here they are named together:
| UB | What it is | Where you met it |
|---|---|---|
| Out-of-bounds | reading/writing outside an array | day 17 (a[i] has no bounds check) |
| Stack overflow | writing past a stack buffer | day 14 / day 23 (strcpy) |
| Use-after-free | using memory after free() | day 22 |
| Double-free | freeing the same block twice | day 22 |
| Null deref | dereferencing a NULL pointer | day 16 |
| Signed overflow | signed int arithmetic that overflows | day 2 / day 8 |
| Uninitialized read | reading a variable before setting it | reads whatever bytes were there |
Why the compiler assumes UB can't happen
This is the crux, and it feels almost unfair until you see the logic. The standard says signed
overflow is UB. So the compiler is entitled to assume it never happens — which lets it optimize
x + 1 > x (for signed x) to simply true, because "if x+1 overflowed it'd be UB, and UB can't
happen, so x+1 is always > x." Great for speed. But if your code *did* rely on overflow wrapping, the
compiler just deleted your logic. The same reasoning lets it delete a null check *after* you've
already dereferenced the pointer ("you dereferenced it, so it can't be null").
1// The compiler may assume 'p' is non-null because you dereferenced it above:2int v = *p; // if p were NULL this is UB...3if (p == NULL) { // ...so the optimizer may DELETE this check entirely4 return -1; // "dead code": p was already assumed non-null5}6 7// Signed overflow is UB, so this loop can be 'optimized' in surprising ways:8for (int i = 0; i <= n; i++) // if n == INT_MAX, i <= n is always true -> UB / infinite loop9 ;You are not defenseless: sanitizers
The good news: the tools that catch UB are excellent, and you should compile with them habitually while learning and testing:
- AddressSanitizer (
-fsanitize=address) catches out-of-bounds, use-after-free, double-free — it - prints the exact line and the memory involved.
- UndefinedBehaviorSanitizer (
-fsanitize=undefined) catches signed overflow, bad shifts, - misaligned access, and more.
- Valgrind (day 22) catches leaks and invalid accesses without recompiling.
These turn silent, optimizer-warping UB into a loud, located error. Running your labs under
-fsanitize=address,undefined is how a professional develops C — the sanitizer is your day-27 gdb,
automated.
int is_valid(int *p) { int x = *p; return p != NULL; } compiled at -O2. What might the optimizer
do to the return, and why is that terrifying?-Wall -Wextra flags many mistakes at compile time, and
-fsanitize=... catches the rest at run time. But no tool catches *all* UB, which is why C demands
that you understand the rules, not just lean on tools. The deepest reason this whole spine exists —
knowing exactly what memory is and how the machine treats it — is so that UB is something you can
*reason about and avoid*, not a mystery that bites you.Provoke UB, then catch it
In ub.c, write small functions that each trigger one UB from this lesson: an out-of-bounds array write, a use-after-free, a signed-overflow comparison, and an uninitialized read. For each: predict what 'should' happen naively, then compile at -O0 AND -O2 and observe any difference, then compile with -fsanitize=address,undefined and read exactly what the sanitizer reports (line, kind, memory). Write a sentence per bug on why the compiler is allowed to do surprising things. Do NOT rely on any particular 'result' — the point is that there isn't one. Uses only concepts from days 2, 14, 17, 22.
$ gcc -O0 -Wall -Wextra ub.c -o ub_o0 && ./ub_o0$ gcc -O2 ub.c -o ub_o2 && ./ub_o2 # compare behaviour to -O0$ gcc -O1 -g -fsanitize=address,undefined ub.c -o ub_san && ./ub_san- › You triggered at least four distinct kinds of UB
- › You observed a behaviour difference (or a deleted check) between -O0 and -O2 for at least one
- › The sanitizer pinpointed each bug with a location
- › You can explain why 'it worked at -O0' is not a correctness argument