What Is Distributed Tracing?
Distributed tracing follows a single logical request as it flows through every service, queue, and database it touches, recording the timing and outcome of each hop as a 'span.' A trace is the tree of all spans belonging to one request, letting you see not just that a checkout was slow, but exactly which of the eight downstream calls it made was the slow one.
Cricket analogy: Following one specific delivery's full journey, from the bowler's run-up, through release, seam position, bat contact, and fielder's throw to the keeper, is a trace; each stage is a span with its own duration.
Spans, Context Propagation, and Trace IDs
Each span carries a trace ID (shared by every span in the trace), a unique span ID, an optional parent span ID (forming the call tree), a start time, duration, and key-value attributes. For a trace to stay connected across service boundaries, the trace context must be propagated on every outgoing call, typically via the W3C Trace Context standard's traceparent HTTP header, which every OpenTelemetry-instrumented library reads and forwards automatically.
Cricket analogy: A DRS ball-tracking session records ball ID, delivery number, and which prior review it relates to, exactly like a span's trace ID, span ID, and parent span ID form a linked chain.
from opentelemetry import trace
from opentelemetry.propagate import inject
import requests
tracer = trace.get_tracer("order-service")
def call_inventory_service(sku):
with tracer.start_as_current_span("check_inventory") as span:
span.set_attribute("sku", sku)
headers = {}
inject(headers) # adds traceparent header automatically
resp = requests.get(f"http://inventory-service/stock/{sku}", headers=headers)
span.set_attribute("http.status_code", resp.status_code)
return resp.json()Sampling Strategies
Tracing every single request in a high-throughput system is often too expensive to store, so most systems apply sampling: head-based sampling decides at the start of a trace (e.g., keep 1% of requests, or 100% of requests to a canary), while tail-based sampling buffers all spans for a trace and decides after seeing the full result, letting you always keep traces that contain an error or exceed a latency threshold even if the overall sample rate is low.
Cricket analogy: A broadcaster reviewing every single delivery in full slow-motion would be prohibitively expensive, so they sample: routine deliveries get one camera angle, but any delivery resulting in a wicket or boundary (the equivalent of an error) gets the full multi-angle replay treatment, like tail-based sampling.
OpenTelemetry (OTel) has become the vendor-neutral standard for generating traces, metrics, and logs; it exports data to backends like Jaeger, Zipkin, Grafana Tempo, or commercial APM tools, so you can instrument once and switch backends without re-instrumenting your code.
If even one service in the call chain fails to propagate the traceparent header (common with older HTTP clients, message queue producers, or manually written integrations), the trace breaks into disconnected fragments and you lose the ability to see the full request path.
- A trace is the tree of all spans belonging to one logical request; a span is one timed unit of work.
- Trace context (trace ID, span ID, parent span ID) must be propagated on every outbound call.
- The W3C Trace Context traceparent header is the standard propagation mechanism.
- OpenTelemetry is the vendor-neutral instrumentation standard, exporting to Jaeger, Tempo, Zipkin, or APM tools.
- Head-based sampling decides at trace start; tail-based sampling decides after seeing the full trace.
- Tail-based sampling can guarantee you keep every erroring or slow trace even at a low overall sample rate.
- A single service failing to propagate trace context breaks the trace into disconnected fragments.
Practice what you learned
1. What uniquely identifies all spans belonging to the same logical request?
2. Which HTTP header does the W3C Trace Context standard define for propagating trace data?
3. What is the main advantage of tail-based sampling over head-based sampling?
4. What happens if one service in a call chain fails to propagate trace context?
5. What is OpenTelemetry primarily used for?
Was this page helpful?
You May Also Like
Observability in Microservices
How to understand the internal state of a distributed system from the logs, metrics, and traces it produces, so you can debug failures you never anticipated.
Testing Microservices
How the test pyramid, contract testing, and chaos experiments combine to give confidence in a system made of many independently deployed services.
Microservices Quick Reference
A condensed cheat sheet of the core microservices patterns, communication styles, and resilience mechanisms for fast review before a design discussion or interview.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 40 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics
Software EngineeringSystem Design Study Notes
Architecture · 40 topics