Mistakes That Repeat Across Teams
Most Docker and Kubernetes incidents trace back to a small set of well-known gotchas: missing resource limits, forgotten health checks, latest tags, or a misunderstanding of how networking and storage actually behave. Knowing these pitfalls in advance — and why they happen — is often more valuable in an interview or on-call rotation than memorizing commands, because it demonstrates operational judgment. Each item below describes the mistake, why it causes trouble, and the fix.
Cricket analogy: A club's worst collapses usually trace back to the same handful of known causes — no fitness limits on bowling overs, skipped pre-match fitness checks, fielding an out-of-form 'automatic' pick, or misreading pitch conditions — recognizing these patterns matters more for a captain's reputation than memorizing field placements.
Using the `latest` tag in production
Tagging images as latest (or relying on it implicitly by omitting a tag) means the same reference can point to different actual image content over time, so a Pod restart or node reschedule can silently pull a newer, untested build. This breaks reproducibility and makes rollbacks unreliable, because you cannot pin latest to a known-good state. Always build and deploy with immutable, specific tags — ideally the Git commit SHA or a semantic version — so the exact same bytes are guaranteed to run wherever that tag is used.
Cricket analogy: Calling up 'the current fast bowler' without naming him means a mid-series injury replacement could quietly take the new ball without anyone re-checking his form; always name the exact player selected, like naming a commit SHA, so everyone knows precisely who's bowling.
Running containers as root without a defined user
By default, a container process runs as root inside the container's user namespace unless the image specifies otherwise, which means a container-escape vulnerability has a much larger blast radius. The fix is to add a USER instruction in the Dockerfile pointing to a non-root UID, and in Kubernetes, to set securityContext.runAsNonRoot: true (and ideally runAsUser) on the Pod or container spec so the cluster actively rejects Pods that try to run as root. This is a standard finding in security reviews and a common interview topic.
Cricket analogy: A ground with no assigned security zones defaults to letting anyone wander onto the pitch, so a single intruder can reach the entire playing area; the fix is issuing specific access badges (USER) and having security actively reject anyone without one (runAsNonRoot).
Omitting CPU/memory requests and limits
Without requests, the scheduler has no idea how much capacity a Pod actually needs, which can lead to overcommitted nodes and noisy-neighbor problems; without limits, a single misbehaving container can consume all available memory on a node and get OOM-killed unpredictably, sometimes taking other Pods down with it via node pressure. Setting sensible requests (based on observed steady-state usage) and limits (with headroom for spikes) lets the scheduler place Pods intelligently and lets the kubelet evict or throttle predictably instead of destabilizing the node.
Cricket analogy: If a captain doesn't tell the groundskeeper how many players each team roughly needs (requests), the ground gets double-booked; if there's no cap on how long one team can occupy the nets (limits), one greedy team can hog the facility and force others out unpredictably.
Confusing liveness and readiness probes
A liveness probe failing causes Kubernetes to kill and restart the container, while a readiness probe failing only removes the Pod from Service endpoints without restarting it. A very common mistake is pointing a liveness probe at a check that depends on a downstream dependency (like a database connection); if that dependency has a temporary outage, Kubernetes will restart every affected Pod in a loop even though restarting does nothing to fix the actual problem. Downstream-dependent checks belong on the readiness probe, so Kubernetes stops routing traffic without needlessly killing healthy processes.
Cricket analogy: If an umpire declares a player 'unfit to continue' (liveness fail) every time the team's physio room is busy, they'd substitute a healthy player pointlessly over and over; the physio-room check should only bench the player temporarily from fielding (readiness), not force a full substitution.
Storing secrets as plain environment variables or baked into images
Hardcoding credentials in a Dockerfile or plain env vars means they end up in image layers (recoverable even after later 'removal' since layers persist) and are visible to anyone who can run docker inspect or kubectl get pod -o yaml. Kubernetes Secrets are base64-encoded, not encrypted, by default, so they should be paired with encryption at rest and RBAC restrictions, or better, an external secret manager (e.g. Vault, cloud KMS-backed secret stores) synced in via an operator. The core rule: never bake credentials into image layers, and treat Secrets objects as sensitive even though Kubernetes stores them accessibly by default.
Cricket analogy: Writing a player's confidential medical report directly on the public team notice board and later crossing it out doesn't erase it — anyone who saw the board before still remembers it; baking passwords into a Dockerfile persists them in image history even after a later 'removal' commit.
Expecting a Deployment update to guarantee zero downtime automatically
A rolling update by itself does not guarantee zero-downtime unless the application also cooperates: the Pod needs an accurate readiness probe so traffic isn't sent to it before it's ready, and it needs to handle SIGTERM gracefully (finishing in-flight requests) during the terminationGracePeriodSeconds window before being force-killed with SIGKILL. Many teams see dropped requests during deploys because the old Pod is killed before in-flight connections drain, or the new Pod receives traffic before its initialization is actually complete. A preStop hook that sleeps briefly before shutdown can help align in-flight connection draining with load balancer deregistration.
Cricket analogy: Swapping a fielder mid-over only works cleanly if the outgoing fielder finishes catching any ball already in flight (graceful shutdown) before leaving, and the incoming fielder is actually warmed up and in position (readiness) before the next ball is bowled; rushing either side drops catches.
- Never deploy
latest— use immutable tags like a commit SHA - Set
USERin Dockerfiles andrunAsNonRootin Pod security contexts - Always set CPU/memory requests and limits on every container
- Keep downstream-dependency checks on readiness probes, not liveness probes
- Never bake secrets into image layers; use Secrets + encryption or an external secret manager
- Prefer PersistentVolumeClaims over hostPath for real persistent data
- Most production incidents trace back to a small, recurring list of misconfigurations
- Liveness vs. readiness confusion is one of the most common self-inflicted outage causes
- Immutable tags and non-root users are baseline hygiene, not advanced practices
- Rolling updates need application-level cooperation (graceful shutdown, accurate readiness) to be truly zero-downtime
- Layer cleanup must happen in the same RUN instruction that created the bloat
- Fixing symptoms (deleting Pods) rarely fixes root causes (bad ReplicaSet template)
Practice what you learned
1. What is the main risk of deploying containers tagged `latest`?
2. Why is it dangerous to put a downstream database health check on a liveness probe instead of a readiness probe?
3. What happens if you delete a file in a later Dockerfile RUN instruction that was created by an earlier RUN instruction?
4. Why are hostPath volumes generally discouraged for persistent application data in production?
5. What is the correct way to recover from a bad rollout where the new image version is crashing?
6. Why should Kubernetes Secrets not be treated as fully secure by default?
Was this page helpful?
You May Also Like
Health Checks and Restart Policies
How Docker uses HEALTHCHECK instructions and restart policies to detect unhealthy containers and automatically recover from failures.
Docker Security Basics
Defensive practices for hardening container images and runtime, including non-root users, minimal base images, vulnerability scanning, and dropped capabilities.
Rolling Updates and Rollbacks
Learn how Kubernetes Deployments update Pods gradually with zero downtime and how to roll back safely when a release breaks.
Persistent Volumes and Claims
Learn how Kubernetes decouples storage from Pods using PersistentVolumes, PersistentVolumeClaims, and StorageClasses for durable, dynamically provisioned data.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics