Tag

OpenTelemetry End to End 7

Follow one click from the browser through nginx, Traefik, a FastAPI API, a backend with Redis and SQLite, and into traces, logs and metrics in Grafana, all on a local k3d cluster.

Built Level: Intermediate Time: about 90 minutes Runs on: k3d (lab-otel) on the homelab network Observability: OpenTelemetry Collector → homelab mon cluster (Grafana, Mimir, Loki, Tempo) Stack: nginx, Traefik, FastAPI (Python), Redis, SQLite, OpenTelemetry JS + Python Services: 5 traced services, 3 signals 7 of 7 parts written

Parts

Experiments

Requests in selected range stat panel after the reload
1 Reload the page

See one page load turn into nginx requests, JSON access-log lines and a browser documentLoad trace.

metricslogstraces
Tempo waterfall: browser, Traefik, API and backend spans with a Redis cache miss, a SQLite SELECT and a Redis SETEX
2 Follow a request on a cache miss

Watch one click cross four services in one trace (browser, Traefik, API, backend), and see the backend fall back to SQLite when Redis has no copy.

traceslogs
Tempo waterfall of a cache hit: the backend answers from Redis with no SQLite span
3 Compare a cache hit

Run the same request with a warm cache and spot what disappears from the trace.

traces
Animated Traefik requests stat panel counting up during the burst
4 Send a burst of traffic

See Traefik's request metrics climb while 150 requests arrive, and how pushed metrics differ from scraped ones.

metrics
Traefik requests per minute by status code with a 502 series during the outage
6 Take the backend down

Stop the backend, watch errors show up in all three signals, then watch it recover.

traceslogsmetrics