Core Concepts Interviewers Probe
Low-level interviews for systems, embedded, security, or compiler roles typically test whether a candidate understands what's actually happening beneath high-level code: how a function call manipulates the stack, why signed and unsigned comparisons can diverge on the same bit pattern, and how the CPU's condition flags (zero, carry, sign, overflow) drive conditional branches. A common opening question is to trace through cmp eax, ebx followed by jg label and explain precisely which flags are checked (SF equals OF, and ZF is clear) — testing whether the candidate actually understands flag semantics rather than having memorized instruction names.
Cricket analogy: Explaining flag semantics under interview pressure is like a commentator explaining exactly why an umpire's decision-review verdict came back 'umpire's call' — you need to articulate the precise underlying rule, not just state the outcome.
cmp eax, ebx ; computes eax - ebx, sets flags, discards result
jg greater_label ; jump if SF == OF and ZF == 0 (signed 'greater than')
; Contrast with unsigned comparison:
cmp eax, ebx
ja greater_label ; jump if CF == 0 and ZF == 0 (unsigned 'above')Stack Mechanics and Buffer Overflow Reasoning
A classic interview scenario asks the candidate to explain what happens when a local char buffer[8] is overrun by a strcpy of a longer string: on x86-64, the stack grows downward, so writing past the buffer's end overwrites adjacent stack memory in the direction of higher addresses — potentially corrupting saved registers, the stack canary (if present), and ultimately the saved return address, which is exactly the mechanism classic stack-smashing exploits rely on. A strong answer also mentions mitigations: stack canaries (checked before leave; ret), non-executable stack (NX/DEP), and ASLR, and explains that these layer defense rather than eliminate the underlying bug class.
Cricket analogy: Explaining stack overflow direction is like explaining exactly how a chasing team's required run-rate creeps upward over unused overs — understanding the direction of the pressure matters as much as noticing it exists.
When asked 'what does the stack canary protect against', the precise answer is: it detects that a buffer overflow has already overwritten the saved return address region, causing the program to abort via __stack_chk_fail before ret executes — it does not prevent the overflow itself.
Calling Convention and Register Preservation Questions
Interviewers frequently ask which registers a called function must preserve versus which it's free to clobber. Under System V AMD64, rbx, rbp, r12-r15 are callee-saved (a called function must restore them before returning if it modifies them), while rax, rcx, rdx, rsi, rdi, r8-r11 are caller-saved (the caller must not assume they survive a call). A common follow-up asks why rax specifically is never callee-saved: it's the designated return-value register, so preserving it across a call would be meaningless since the call is expected to overwrite it with its result.
Cricket analogy: Callee-saved registers are like fielding positions a substitute fielder must return to their original spot before leaving the field, while caller-saved registers are like positions the incoming fielder is free to leave wherever, since the captain will reset them anyway.
A frequent interview trap: candidates assume the stack pointer (rsp) is 'just another register.' In System V AMD64, rsp must be 16-byte aligned at the point of a call instruction (i.e. before the call pushes the return address) — misaligning it before calling into functions that use SSE/AVX instructions expecting aligned data can cause a crash.
- Interviewers test understanding of condition flags (ZF, SF, OF, CF) and how signed vs unsigned comparisons use different flag combinations.
- Buffer overflows on x86-64 write toward higher addresses since the stack grows downward, potentially overwriting the saved return address.
- Stack canaries detect overflow damage before ret executes; they don't prevent the overflow itself.
- Under System V AMD64, rbx/rbp/r12-r15 are callee-saved; rax/rcx/rdx/rsi/rdi/r8-r11 are caller-saved.
- rax is never callee-saved because it's the designated return-value register, always expected to change across a call.
- rsp must be 16-byte aligned at the point of a call instruction under System V AMD64 for SSE/AVX compatibility.
- Strong interview answers explain the precise mechanism (which flags, which registers, why) rather than just naming the concept.
Practice what you learned
1. Which condition flags does the signed conditional jump `jg` (jump if greater) check?
2. In System V AMD64, which of the following registers is caller-saved rather than callee-saved?
3. What does a stack canary actually protect against?
4. Why does an x86-64 buffer overflow in a local array typically threaten the saved return address?
5. What alignment requirement applies to rsp at the point of a `call` instruction under System V AMD64?
Was this page helpful?
You May Also Like
Reverse Engineering with Assembly
How to read disassembled and decompiled code to recover program logic, recognize compiler-generated patterns, and identify common function idioms.
x86 vs ARM Assembly
A comparison of x86 and ARM assembly language philosophies, instruction formats, register models, and calling conventions.
Assembly Language Quick Reference
A condensed reference covering common x86-64 instructions, registers, flags, and addressing syntax for quick lookup.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics