OpenTelemetry End to End: Deploy

One command creates the cluster, joins it to the homelab platform, builds three images and deploys the app tier, the Traefik settings and the Collector. This part shows what make up does step by step, how to tell everything is healthy, and what to check when it isn’t. At the end you’ll have the page open in one tab and the homelab’s Grafana in another.

Bring It Up

From the lab repo:

make up

up is five targets in a row:

up: cluster dns monitoring build deploy ## Create the cluster, join the platform, build and deploy everything
	@echo
	@echo "Site:    http://localhost:8081"
	@echo "Grafana: $(GRAFANA_UI) (mon, Keycloak sign-in)"

1. cluster creates the k3d cluster lab-otel from k3d/cluster.yaml, unless it already exists.
2. dns adds the CoreDNS forwarding for the homelab’s internal zone and restarts CoreDNS.
3. monitoring runs lab.sh register lab-otel dashboards (see Part 3) and writes the Collector’s and the Trace Lab’s tokens. On a first run this waits a few minutes while Argo CD installs the monitoring pack.
4. build builds static-site:dev, trace-lab-api:dev and trace-lab-backend:dev and imports them into the cluster’s containerd with k3d image import.
5. deploy applies everything with kubectl apply -k k8s, then waits for each piece in dependency order:

deploy: ## Apply all manifests and wait for rollouts
	$(KUBECTL) apply -k k8s
	$(KUBECTL) -n observability rollout status ds/otel-collector --timeout=5m
	$(KUBECTL) -n kube-system rollout status deploy/traefik --timeout=5m
	$(KUBECTL) -n demo rollout restart deploy/static-site
	$(KUBECTL) -n demo rollout restart deploy/api deploy/backend
	$(KUBECTL) -n demo rollout status deploy/static-site --timeout=5m
	# … api, redis, backend

Every kubectl in the Makefile is kubectl --context k3d-lab-otel, so nothing lands in another homelab cluster by accident. The explicit rollout restart of the app pods matters on re-deploys: the image tags stay :dev, so without a restart Kubernetes would keep running the old containers after make build.

Changed something under app/? Run make build deploy, not just make deploy.

Verify the Rollout

make status

On the run this part was checked against (2026-10-14), the lab’s namespaces looked like this:

$ kubectl --context k3d-lab-otel get pods -n demo
NAME                           READY   STATUS    RESTARTS   AGE
api-868cf77665-dlz96           1/1     Running   0          7m49s
backend-6fbd967fff-9k44q       1/1     Running   0          2m41s
redis-dc767847c-pcxcs          1/1     Running   0          8m8s
static-site-75ccc69ccd-h2qwn   2/2     Running   0          2m56s

$ kubectl --context k3d-lab-otel get pods -n observability
NAME                   READY   STATUS    RESTARTS   AGE
otel-collector-qnbtk   1/1     Running   0          8m8s

$ kubectl --context k3d-lab-otel get pods -n monitoring
NAME                                       READY   STATUS    RESTARTS   AGE
agent-collector-9gcls                      1/1     Running   0          8m47s
cluster-collector-0                        1/1     Running   0          8m47s
cluster-targetallocator-5cfcdbc6cb-pzxp4   1/1     Running   0          8m47s
kube-state-metrics-7dd9dc999-zl7hh         1/1     Running   0          16m

monitoring is the platform’s monitoring pack, not part of the lab’s own code. make status also shows the platform’s view of it:

$ infra-local/host/labs/lab.sh status lab-otel
https://172.30.0.139:6443
  cert-manager-lab-otel                Synced/Healthy
  prometheus-operator-crds-lab-otel    Synced/Healthy
  opentelemetry-operator-lab-otel      Synced/Healthy
  kube-state-metrics-lab-otel          Synced/Healthy
  monitoring-pack-lab-otel             Synced/Healthy

What to check:

• static-site shows 2/2: nginx and the metrics sidecar.
• Every pod is Running with READY full. Readiness is real here: each app pod is only ready once GET /healthz answers.
• All five Argo CD apps are Synced/Healthy.
• The Ingress has an address:

$ kubectl --context k3d-lab-otel get ingress -n demo
NAME          CLASS     HOSTS   ADDRESS      PORTS   AGE
static-site   traefik   *       172.30.0.139   80      8m8s

Smoke Test

Three checks, one per entry point:

$ curl -s -o /dev/null -w "site %{http_code}\n" localhost:8081/
site 200

$ curl -s localhost:8081/api/demo
{"tier":"api","backend":{"tier":"backend","record":{"id":"demo","message":"Hello from the database"},"source":"database"},"trace_id":"618404664486fc16987418f02f8fb780"}

$ curl -s 'localhost:8081/grafana-api/prometheus/api/v1/query?query=count(up%7Bcluster%3D%22lab-otel%22%7D)'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1791921946.841,"11"]}]}}

The API answer proves the whole request path works: Traefik routed /api, the API reached the backend, and the backend reached Redis and SQLite. The last check goes through the Trace Lab’s proxy to mon’s Mimir: 11 scrape targets in this lab report there, so the platform side works too. Copy its trace_id; you can paste it into Grafana’s Tempo search in Part 5.

Finally, open http://localhost:8081, reload once and click Call /api/demo. The result box shows the response, and the This request in Grafana panel fills in with the matching log line and trace a few seconds later.

Common Failures

Symptom	Likely cause	Fix
k3d cluster create fails with “port is already allocated”	Something else uses 8081 or 6451 (often another lab cluster)	Stop it, or change the ports in k3d/cluster.yaml and the card’s deploy.ports
You changed k3d/cluster.yaml but nothing changed	make cluster reuses an existing cluster	make rebuild
make monitoring times out on one Argo CD app	On a new cluster the first sync can race cert-manager’s CRDs; Argo CD retries with backoff	Re-run make monitoring; check the app in Argo CD
Collector logs 401 Unauthorized	Secret observability/otlp-token is missing or stale	make monitoring, then restart the Collector
Trace Lab panel says “Grafana returned HTTP 502”	nginx can’t reach or verify mon’s Grafana (DNS for the internal zone, or TLS)	make dns, and check the nginx container’s error log
App pods stuck in ErrImagePull / ImagePullBackOff	The :dev images were never imported; there’s no registry to pull from	make build, then make deploy
Code change not visible	Same :dev tag, old pods still running	make build deploy (deploy restarts the pods)
Grafana dashboard empty	Time picker shows a window before you sent traffic, or the Service filter excludes it	Set “Last 15 minutes”, choose All services; queries of your own need the lab’s cluster filter
Tempo search can’t find a trace you just made	Tempo’s search index lags behind ingestion; on this lab it took up to about a minute	Wait and search again; opening the trace by ID works sooner
Metrics panel flat after a click	nginx metrics are scraped every 15 s, and rates use a 5-minute window	Wait for the next scrape, or use the “Requests in selected range” stat
You edited dashboards/static-site.json and Grafana still shows the old one	The dashboard is loaded into mon by lab.sh register, not by make deploy	make monitoring (Grafana’s sidecar on mon picks it up within a minute)

Two of these rows were hit while capturing this lab’s own screenshots. A make verify-* check run 28 seconds after a click found nothing, and the same query succeeded a little later, so the verify script now retries for a minute before it gives up. And, with the earlier bundled-LGTM version of the lab, the Traefik panel was missing from Grafana although it was in the ConfigMap: the lgtm pod had been running since before the panel was added.

Check Your Understanding

Why does make deploy restart the app pods even when no manifest changed?

The images are always tagged :dev. A rebuilt image has the same tag, so Kubernetes sees no change in the Deployment. rollout restart forces new pods, which pick up the newly imported image.

The API returned a trace_id. What does that tell you before you open Grafana?

That the API had a valid span context, so its instrumentation is running. It doesn’t prove the span reached Tempo; for that, look it up (or run make verify-cache-miss).

Which single command shows that Traefik, the API, the backend, Redis and SQLite all work?

curl localhost:8081/api/demo: a 200 with a record in the body can only come back if the request went through every one of them (SQLite on a miss, Redis on a hit).