Bash Script Debugging Techniques

Analogy categoryAnalogies

Why Bash Scripts Are Hard to Debug

Debugging bash scripts is harder than debugging most languages because the shell silently rewrites your commands before running them: word splitting breaks unquoted variables into multiple arguments, globbing expands wildcards, and pipelines fork subshells that hide variable changes from the parent. A script can look correct on the page while executing something entirely different, so effective debugging means making the shell show you exactly what it decided to run, not what you assumed it would run.

🏏

Cricket analogy: It is like a coach reviewing ball-by-ball Hawk-Eye replay after Virat Kohli is given out lbw, to see the actual trajectory instead of trusting what everyone assumed happened at full speed.

Tracing Execution with set -x and PS4

The single most useful debugging tool in bash is set -x, which prints every command to stderr after expansion but before execution, prefixed by the PS4 variable (default +). Customizing PS4 to include ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]} turns a wall of + lines into a precise trace showing which file, line, and function produced each command, which is essential once a script spans multiple sourced files. set -v is the blunter sibling: it echoes raw input lines before any expansion, useful for spotting syntax-level typos rather than runtime values.

🏏

Cricket analogy: Setting PS4 to show file and line number is like a scorecard that logs not just each ball bowled but the bowler's name and over number, so you can trace exactly who bowled the no-ball.

bash

#!/usr/bin/env bash
set -euo pipefail

# Show file:line:function for every traced command
export PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]:-main}(): '

deploy() {
  local env="$1"
  set -x
  aws s3 sync ./dist "s3://app-${env}-bucket" --delete
  systemctl restart "app-${env}.service"
  set +x
}

deploy "staging"

Trapping Errors with trap for Structured Diagnostics

trap lets you run a handler when the shell receives a signal or reaches a pseudo-event, and the two most valuable for debugging are ERR and DEBUG. A handler on trap 'echo "Failed at line $LINENO: $BASH_COMMAND" >&2' ERR fires whenever a command exits non-zero under set -e, printing the exact failing command and its line number without you needing to instrument every call site. trap ... DEBUG fires before every single command, which is verbose but invaluable for building a custom step-through debugger, while trap ... EXIT guarantees cleanup code (removing temp files, releasing locks) runs even if the script dies partway through.

🏏

Cricket analogy: An ERR trap is like the third umpire being automatically called in whenever a run-out appeal happens, without the on-field umpire needing to remember to request a review each time.

$LINENO inside an ERR trap reports the line of the trap invocation itself unless you capture it at the failure site; combine trap 'rc=$?; echo "line $LINENO exited $rc"' ERR with set -o functrace to get accurate line numbers inside functions and subshells too.

Static Analysis and Interactive Debugging

Runtime tracing only catches bugs you can reproduce; ShellCheck catches an entire class of bugs before the script ever runs by statically analyzing quoting, word splitting, and POSIX compatibility issues, and it integrates directly into editors and CI pipelines as a linter. For scripts where tracing output is too noisy to read, bashdb provides an actual interactive debugger with breakpoints, single-stepping, and variable inspection modeled on gdb, letting you pause execution at a specific line and inspect the environment rather than scrolling through thousands of set -x lines.

🏏

Cricket analogy: ShellCheck is like a pre-match pitch inspection that flags a dangerous crack before a single ball is bowled, catching the hazard long before it causes an injury mid-match.

Never leave set -x enabled around commands that handle secrets (API keys, passwords, tokens) in CI pipelines — the expanded command line, including the secret value, gets written straight to build logs. Wrap only the specific block you need traced with set -x / set +x, and prefer trap ... DEBUG with selective filtering when tracing near credentials.

set -x traces expanded commands to stderr; customize PS4 with ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]} for precise, attributable traces.
set -v echoes raw unexpanded input lines, useful for catching syntax typos rather than runtime value bugs.
trap ... ERR fires on any non-zero exit under set -e, letting you log the failing command and line without instrumenting every call site.
trap ... EXIT guarantees cleanup (temp files, locks) runs even on abnormal termination; trap ... DEBUG fires before every command for fine-grained stepping.
ShellCheck performs static analysis that catches quoting and word-splitting bugs before the script ever runs, and belongs in every CI pipeline.
bashdb provides interactive breakpoints and single-stepping for scripts too complex to debug via trace output alone.
Never trace commands containing secrets — expanded values land directly in logs; scope set -x/set +x tightly around only the block you need.