OpenTelemetry End to End: Overview and Architecture

In this lab you click one button on a web page and then find that click again in Grafana: as a trace that crosses four services, as log lines that carry the same trace ID, and as a bump in a request counter. By the end of this part you’ll know what you are about to build, what each piece is for, and which way every signal travels.

What You Will Build

A small three-tier web application, running on a Kubernetes cluster on your laptop, with OpenTelemetry in every tier:

• a static page served by nginx, with the OpenTelemetry JavaScript SDK loaded in the browser;
• a Traefik ingress that routes / to nginx and /api to the API;
• an API written in Python with FastAPI, which calls
• a backend, also FastAPI, that reads a record with a cache-aside pattern: Redis first, SQLite on a miss;
• an OpenTelemetry Collector that receives, pulls and forwards every signal, and
• the homelab’s mon cluster (Grafana, Loki, Tempo and Mimir) to look at it all.

Everything is driven by a Makefile. One command builds the images and deploys the whole stack:

cd otel-static-site-k3d
make up

Then the page is at http://localhost:8081, and the lab’s dashboard is in the homelab’s Grafana at https://grafana.mon.home.cloudutsuk.com/d/lab-otel-signals (folder Labs).

Since 2026-10-14 the lab runs on the homelab platform (lab contract v2): its k3d cluster joins the homelab network and sends every signal to the mon cluster instead of a Grafana LGTM container of its own. The screenshots in this series were taken with that earlier, self-contained version, lab commit 8fbd1b1, which needs nothing but Docker and k3d.

TODO: The lab code lives in otel-static-site-k3d (remote github.com/cloudutsukpilot/otel-static-site-k3d), which isn’t public yet. Should the repo be published so readers can clone it? If so, add repo.url to _data/labs/opentelemetry-lab.yml and a clone command here.

Prerequisites

You need the homelab platform running: the homelab Docker network and the core, platform and mon clusters, with infra-local checked out at ~/gitrepos/iac/infra-local and kube contexts k3d-platform and k3d-mon in your kubeconfig. The lab itself needs about 3 GB of memory: its own pods plus the platform’s monitoring pack, which Argo CD installs when the lab registers. The first make up pulls the Collector, Redis, nginx, Python and Node base images through the homelab’s Zot mirror.

Tool	Version this lab was run with	Check
Docker	29.2.1 (Engine)	docker version
k3d	v5.9.0 (bundles k3s v1.35.5)	k3d version
kubectl	v1.36.2	kubectl version --client
make	GNU Make 4.3	make --version
curl, python3, jq	any recent	used by make up and the experiment targets

Host ports 8081 (the site, through Traefik) and 6451 (the cluster’s API), both on 127.0.0.1, must be free when the cluster is created. k3d fixes port mappings at creation time; changing them later means make rebuild.

No cloud account and no Helm install in the lab: its three images are built locally and imported into the cluster with k3d image import.

Architecture

Step through the animation: first the request path of one click on Call /api/demo, then how each signal reaches Grafana. Tick the checkbox to replay it as a cache hit, which skips SQLite.

1. A click starts a trace: otel.js wraps the fetch to /api/demo in a span and adds a W3C traceparent header carrying the trace ID.
2. Traefik continues it: The ingress reads traceparent and records a gateway span as a child, in the same trace.
3. Pages come from nginx: Traefik routes / to nginx, which serves the static page. nginx traces and logs its own requests, separately.
4. /api goes to FastAPI: opentelemetry-instrument wraps the API, so the request becomes a server span with no tracing code in main.py.
5. The API calls the backend: The HTTPX client span injects traceparent again, so the backend's spans join the same trace.
6. Check Redis first: The backend's Redis GET span is renamed GET (cache miss) or GET (cache hit) by a response hook.
7. SQLite on a miss (miss): On a miss the backend runs a manually created sqlite SELECT records span, then SETEX writes Redis for 60 s.
8. Spans are pushed: Every service exports spans over OTLP to the Collector. The browser's go through nginx at /v1/traces.
9. Metrics and logs are pulled: The Collector scrapes the nginx exporter every 15 s and tails JSON log lines from the pod log files.
10. One pipeline per signal: Traces go to Tempo, logs to Loki, metrics to Mimir, all in the homelab's mon cluster, tagged with the lab's cluster, lab-otel.
11. One place to look: Grafana queries all three. The shared trace ID is what lets you jump from a log line to its trace.

Component	Role	Runs as	Service name in telemetry
Browser + otel.js	Creates spans for page load, clicks and fetch, adds traceparent	JavaScript in the page	state-site-browser
Traefik v3.6.13	Ingress: / → nginx, /api → API; gateway spans and OTLP metrics	k3s-bundled Deployment in kube-system	state-site-traefik
nginx 1.30 + otel module	Serves the page; server spans; JSON access logs; proxies /v1/traces	Deployment static-site (with an exporter sidecar)	state-site-nginx
API (FastAPI)	/api/demo: calls the backend, logs a JSON completion line	Deployment api	state-site-api
Backend (FastAPI)	/data: Redis GET, SQLite SELECT on a miss, SETEX	Deployment backend	state-site-backend
Redis 7.4	Cache, 60-second TTL	Deployment redis	(spans come from the backend’s client)
SQLite	The “database”: one seeded row in a file	Inside the backend container	(manual span in the backend)
OTel Collector 0.162.0	OTLP receiver, log tailer, Prometheus scraper, exporter	DaemonSet in observability
The homelab’s mon cluster	OTLP gateway, Tempo, Loki, Mimir, Grafana 13.2 for every homelab cluster	Another k3d cluster	(filter on lab-otel)

Three different transport patterns are in play, and telling them apart is half the lesson:

• Push: traces (from every tier) and Traefik’s metrics are exported over OTLP to the Collector.
• Tail: nginx and the API write JSON lines to stdout; Kubernetes stores them in files under /var/log/pods; the Collector reads those files.
• Scrape: nginx’s counters are exposed by an exporter sidecar, and the Collector pulls them every 15 seconds.

Building Blocks

The lab is about OpenTelemetry, but it stands on a dozen other technologies. Here is each one in a paragraph: what it is, and why it’s in this lab. Follow the links for the deeper posts on this blog.

The application

Python. A general-purpose language with a huge library ecosystem, and the language both services are written in (on the python:3.12-slim image). Python suits this lab because its OpenTelemetry auto-instrumentation is mature: you can trace an app without editing it. More: Introduction to Python.

FastAPI and Uvicorn. FastAPI (0.115) is a Python web framework for building HTTP APIs from plain functions: @app.get("/api/demo") turns a function into an endpoint and its return value into JSON. Uvicorn (0.34) is the ASGI server that actually listens on port 8000 and runs it. Both services use them because they are small, fast to start and instrumented by opentelemetry-instrumentation-fastapi.

REST API and backend. An API is a contract one program offers another; a REST-style HTTP API exposes resources at URLs and answers with JSON. Here the API tier (/api/demo) is what the browser calls, and the backend tier (/data) owns the data. Splitting them gives the trace a service-to-service hop to follow, which is where distributed tracing earns its keep. More: Introduction to APIs and REST API.

HTTPX. The HTTP client library the API uses to call the backend. Its OpenTelemetry instrumentation creates a client span and injects the traceparent header, which is how the backend’s spans end up in the same trace.

Redis. An in-memory key-value store, used everywhere as a cache. The backend checks Redis before touching the database and stores the record for 60 seconds afterwards (SETEX). It’s here so you can see a cache hit and a cache miss as two different trace shapes.

SQLite. A complete SQL database in a single file, with no server process. It plays “the database” behind the cache. Python has no SQLite auto-instrumentation in this lab, so the backend creates the sqlite SELECT records span by hand, which shows you what manual instrumentation looks like.

The web tier

HTML, JavaScript and the browser SDK. The page is a static index.html plus otel.js, a bundle of the OpenTelemetry Web SDK made with esbuild (a JavaScript bundler) at image build time. In the browser it records page loads, clicks and fetch calls as spans, so the trace starts where the user actually is, not at the first server.

nginx. A web server and reverse proxy, here in the nginx:1.30-otel image that ships the OpenTelemetry module. It serves the page, creates a span per request, writes JSON access logs with the trace ID in them, and forwards the browser’s spans to the Collector on the same origin, so no CORS setup is needed.

Traefik. A cloud-native reverse proxy and the default ingress controller of k3s. Every request from your browser enters the cluster through it. It continues the browser’s trace with a gateway span and pushes its own request metrics over OTLP, which is where the lab’s per-status-code metrics come from. More: Kubernetes Ingress.

Containers and Kubernetes

Docker. Builds each tier into a container image from a Dockerfile: a two-stage build for nginx (Node to bundle otel.js, then nginx) and one for each Python service. Containers make the lab behave the same on any machine. More: Introduction to Docker.

Kubernetes. The orchestrator that runs the containers as Deployments, gives them stable names through Services (http://backend:8000), restarts them when probes fail and lets you scale them, which experiment 6 uses to take the backend down. The Collector runs as a DaemonSet, one per node, so it can read every pod’s log files. More: Introduction to Kubernetes.

k3d and k3s. k3s is a lightweight, certified Kubernetes distribution; k3d runs k3s inside Docker containers, so a full cluster starts in seconds on a laptop and disappears with one command. It also maps host ports (8081 for the site, 6451 for the API) into the cluster. More: Install k3d.

Kustomize and make. kubectl apply -k k8s applies the manifests with Kustomize, which also turns the Collector config into a ConfigMap. make wraps the whole build, deploy and experiment workflow into short, repeatable commands.

Observability

OpenTelemetry (OTel). A vendor-neutral standard and toolkit for telemetry: APIs and SDKs to create traces, metrics and logs, automatic instrumentation for common libraries, a wire protocol, and the Collector. It describes and moves the data; it doesn’t store or display it.

OTLP and W3C Trace Context. OTLP is OpenTelemetry’s protocol for sending telemetry, over gRPC (port 4317) or HTTP (4318). W3C Trace Context is the traceparent HTTP header that carries the trace ID from one service to the next. Together they are why four services built in two languages produce one trace.

OpenTelemetry Collector. A standalone pipeline: receivers take data in (OTLP, log files, Prometheus scrapes), processors batch and enrich it, exporters send it on. Apps only need to know the Collector’s address; the backend behind it can change without touching them.

Grafana, Loki, Tempo and Mimir on mon. Loki stores logs, Tempo stores traces, Mimir (a Prometheus-compatible store) keeps metrics, and Grafana queries all three. They run in the homelab’s mon cluster and hold every homelab cluster’s data for 15 days, so the lab’s queries filter on its cluster name, lab-otel. More: Grafana, Loki and Prometheus.

What You Will Learn

• How a trace ID travels from the browser through a gateway and two services, and what each instrumentation library adds: Experiment 2.
• How a trace’s shape reveals behaviour, such as a cache hit versus a miss: Experiment 3.
• The difference between pushed, tailed and scraped signals, and why each arrives at a different time: Experiment 1 and Experiment 4.
• How to correlate a log line with its trace using nothing but a shared ID: Experiment 5.
• How a failure shows up in all three signals, and where to look for the root cause: Experiment 6.

Check Your Understanding

Which component stores the traces, and which one only moves them?

Tempo stores traces. The OpenTelemetry Collector only receives, processes and forwards them; it keeps nothing once a batch has been exported.

Name one signal that is pushed, one that is tailed and one that is scraped in this lab.

Pushed: spans from any tier (OTLP), or Traefik’s metrics. Tailed: nginx and API log lines read from /var/log/pods. Scraped: nginx’s request counters from the exporter sidecar.

Why does the SQLite query need hand-written instrumentation when Redis doesn't?

Redis has an OpenTelemetry instrumentation library that the backend enables on its client. The SQLite access here is plain sqlite3 calls with no instrumentation in the lab’s dependencies, so the backend opens a span around the query itself.