Fault Tolerance: Let It Crash

Analogy categoryAnalogies

The 'Let It Crash' Philosophy

Rather than wrapping every operation in defensive guard clauses to anticipate every conceivable failure mode, idiomatic Erlang code is often written to handle only the expected, 'happy path' cases explicitly and simply let anything else crash. The crashing process terminates, its supervisor (linked via spawn_link and trap_exit) detects the failure, and the supervisor's restart strategy brings a fresh instance of that process back into a known-good state — moving the burden of recovery out of scattered application code and into a small, centralized, well-tested supervision layer.

🏏

Cricket analogy: A fast bowler who doesn't over-think every possible batting stroke before each delivery but instead bowls their natural line and trusts the captain to make a bowling change if it goes wrong mirrors how Erlang code doesn't defensively guard every edge case, trusting the supervisor to react.

Distinguishing Expected Errors from Unexpected Crashes

The key discipline in 'let it crash' is distinguishing two very different categories of failure. Anticipated, expected conditions that are a normal part of the business logic — an invalid withdrawal amount, a not-found lookup, a validation failure on user input — should be represented explicitly, typically as {error, Reason} tuples that the caller pattern-matches on and handles. Genuinely unexpected conditions — malformed data that violates an internal invariant, a programmer error, a dependency behaving in a way that was never anticipated — are left to crash via a failed pattern match or an unhandled exception, rather than being defensively caught and silently papered over.

🏏

Cricket analogy: A batsman explicitly leaving a delivery clearly outside off-stump is a handled, expected outcome like an {error, Reason} tuple, while an unplayable unpredictable jaffa that clean bowls them through no fault of their own is an unexpected crash the system can't defensively code around.

erlang

%% Happy path uses pattern matching; anything else crashes intentionally.
process_order(#{status := paid, amount := Amount}) when Amount > 0 ->
    ship_order(Amount);  %% badmatch/function_clause crash on malformed input

%% Expected business error is returned explicitly, not a crash.
validate_amount(Amount) when Amount =< 0 ->
    {error, invalid_amount};
validate_amount(Amount) ->
    {ok, Amount}.

How Crashes Become Recovery

When a process does crash, the failure is first written to the system's log via the error_logger (or the newer logger module), capturing the crash reason and a stack trace for later debugging. The process's linked supervisor, which has trap_exit enabled, receives the resulting 'EXIT' signal, consults its restart strategy and the crashed child's spec, and restarts it by invoking its init/1 again — discarding whatever corrupted or inconsistent state the old process held and replacing it with a fresh, known-good starting state, all without any operator having to intervene.

🏏

Cricket analogy: When a fielder cramps up and has to leave the field, the twelfth man is sent on as a fresh, uncramped replacement immediately by team management, mirroring how a supervisor restarts a crashed child process with clean, fresh state via init/1 rather than trying to nurse the corrupted one back.

This self-healing pattern — crash, get detected by a link, get restarted into clean state — is part of why systems like the Ericsson AXD301 ATM switch, built on Erlang/OTP, became famous for reportedly achieving 'nine nines' (99.9999999%) availability despite individual processes crashing routinely under load.

The Limits of Let It Crash

'Let it crash' is a deliberate design discipline, not a blanket excuse to skip validation everywhere. Data crossing a genuine system boundary — a malformed HTTP request body, untrusted external input, a malformed message from another node — should still be validated explicitly and rejected with a clear error, rather than allowed to crash a process on every malformed request, which would be noisy and wasteful. Just as important, any operation that performs an irreversible side effect must be designed to be idempotent, because a crash partway through it followed by a supervisor restart can cause the operation to be attempted again from a fresh state that has no memory of the earlier partial attempt.

🏏

Cricket analogy: A team can't simply "let it crash" on ball-tampering allegations and restart the innings — some things, like verifying the ball's condition at the boundary line, must be checked upfront, just as Erlang systems still validate data at system boundaries rather than crashing on every malformed input.

Let it crash is not a license to skip input validation or idempotency. If a process crashes partway through an irreversible side effect — like charging a payment or sending an email — and its supervisor restarts it, a naive retry of that same operation can duplicate the effect. Validate data at system boundaries and design side-effecting operations to be safely repeatable.

'Let it crash' means Erlang code often avoids defensively guarding every edge case and instead lets unexpected failures terminate the process.
Expected, anticipated failures should be returned explicitly as {error, Reason} tuples and handled by the caller.
Truly unexpected conditions (bad input, programmer bugs) are allowed to crash via pattern-match failures or unhandled exceptions.
A crashed process's supervisor detects the exit via a link and restarts it into a known-good state via init/1.
This self-healing recovery is why Erlang systems can achieve extremely high uptime despite individual process failures.
Let it crash is not a license to skip all validation — data crossing system/API boundaries should still be validated explicitly.
Operations that might be re-executed after a restart must be designed to be idempotent to avoid duplicated side effects like double charges.