eBPF Lab: Deploy

One command creates the cluster, gives it a network, joins it to the platform and installs four eBPF tools and the app, in an order that matters. This part shows that order, how to tell every piece is healthy, and the failures this lab has actually hit. At the end the app, Hubble and the Falcosidekick UI are open in your browser.

Bring It Up

From the lab repo:

cd ~/gitrepos/observability/ebpf-lab
make up

up is eleven targets in a fixed order:

up: cluster node-prep cilium dns storage monitoring images app tetragon falco obi ## Everything, in order (about 15 min on a first run)
	@echo; echo "app $(APP)  Hubble UI http://127.0.0.1:8092  Falcosidekick UI http://127.0.0.1:8093"

Target	What it does	Why it’s here in the order
cluster	Creates the PV folders under /srv/homelab/labs/ebpf-lab and the k3d cluster (skipped if it exists)	Everything else needs it
node-prep	Mounts a shared bpffs in each node container	Cilium pins its maps there
cilium	Installs Cilium with k8sServiceHost set to the server node’s address, waits for CoreDNS	Until now the nodes are NotReady and no pod has a network
dns	CoreDNS upstreams for the homelab network, restart CoreDNS	Image and rule downloads in later steps need working DNS
storage	StorageClass lab-local and the two static PVs	Before any claim asks for them
monitoring	lab.sh register lab-ebpf dashboards, copies the OTLP token to namespace obi	Argo CD installs the monitoring pack; OBI needs the token
images	Builds the two app images and imports them	No registry
app	Applies k8s/app/app.yaml and waits for redis, api, web
tetragon	Installs Tetragon with the API server’s real address	Host-network pod: no ClusterIP
falco	Installs Falco, Falcosidekick and the UI	Downloads rules, so DNS must work
obi	Installs OBI, then patches in the host alias for the gateway	Last, so the app’s processes already exist

How long it takes depends mostly on what’s cached. The Makefile’s own estimate for a first run is about 15 minutes. On the run this part was checked against (2026-10-14), with every image already in the homelab’s Zot mirror, make up took about 4 minutes; experiment 7’s make rebuild (down, then up) took about 12 minutes on both of its runs (747 seconds on the first).

When it’s done:

app http://127.0.0.1:8091  Hubble UI http://127.0.0.1:8092  Falcosidekick UI http://127.0.0.1:8093

Verify the Rollout

make status is the preflight the screenshot script runs before every capture:

$ make status
node k3d-lab-ebpf-agent-0: Ready
node k3d-lab-ebpf-server-0: Ready
OK
app: ok
hubble ui: ok
falcosidekick ui: ok

The OK is cilium-dbg status --brief from inside the Cilium agent. The full status says the things this lab depends on:

$ kubectl --context k3d-lab-ebpf -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status
Kubernetes:              Ok         1.35 (v1.35.5+k3s1) [linux/arm64]
KubeProxyReplacement:    True   [eth0  172.30.0.136 (Direct Routing)]
Cilium:                  Ok   1.20.2 (v1.20.2-e0dc92bd)
IPAM:                    IPv4: 19/254 allocated from 10.100.0.0/24,
Routing:                 Network: Tunnel [vxlan]   Host: Legacy
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 21.72   Metrics: Ok
Cluster health:          2/2 reachable   (2026-10-13T21:44:07Z)   (Probe interval: 1m36.566274746s)
# … (trimmed)

KubeProxyReplacement: True, VXLAN routing and both nodes reachable are what you want. Hubble’s ring buffer is already full (4,095 flows) a few minutes after start; it keeps the most recent ones.

Every pod, with its node (trimmed to the lab’s own namespaces and the eBPF tools):

$ kubectl --context k3d-lab-ebpf get pods -A -o wide
NAMESPACE     NAME                                      READY  STATUS   NODE
demo          api-5df5d59bd7-lhbhh                      1/1    Running  k3d-lab-ebpf-agent-0
demo          api-5df5d59bd7-vss5p                      1/1    Running  k3d-lab-ebpf-server-0
demo          redis-bcc8fb84b-m6nfd                     1/1    Running  k3d-lab-ebpf-server-0
demo          web-57fbb6bb9f-zcggd                      1/1    Running  k3d-lab-ebpf-server-0
falco         falco-4thlb                               2/2    Running  k3d-lab-ebpf-server-0
falco         falco-rp8wr                               2/2    Running  k3d-lab-ebpf-agent-0
falco         falco-falcosidekick-66d8cdc879-gsqc7      1/1    Running  k3d-lab-ebpf-server-0
falco         falco-falcosidekick-ui-5dddc5ddb7-995hh   1/1    Running  k3d-lab-ebpf-server-0
falco         falco-falcosidekick-ui-redis-0            1/1    Running  k3d-lab-ebpf-server-0
kube-system   cilium-5r4h7                              1/1    Running  k3d-lab-ebpf-server-0
kube-system   cilium-7lxtx                              1/1    Running  k3d-lab-ebpf-agent-0
kube-system   cilium-envoy-fjfbx                        1/1    Running  k3d-lab-ebpf-server-0
kube-system   cilium-envoy-v5qts                        1/1    Running  k3d-lab-ebpf-agent-0
kube-system   cilium-operator-58ccc79878-rlkq5          1/1    Running  k3d-lab-ebpf-agent-0
kube-system   hubble-relay-bdbfb59-l8m6q                1/1    Running  k3d-lab-ebpf-agent-0
kube-system   hubble-ui-5d45b9c77b-jgqld                2/2    Running  k3d-lab-ebpf-agent-0
kube-system   tetragon-9cj8m                            2/2    Running  k3d-lab-ebpf-server-0
kube-system   tetragon-pkwsn                            2/2    Running  k3d-lab-ebpf-agent-0
kube-system   tetragon-operator-7b4dd7b-4dcvd           1/1    Running  k3d-lab-ebpf-server-0
monitoring    agent-collector-gxzqw                     1/1    Running  k3d-lab-ebpf-server-0
monitoring    agent-collector-sg2nb                     1/1    Running  k3d-lab-ebpf-agent-0
monitoring    cluster-collector-0                       1/1    Running  k3d-lab-ebpf-server-0
obi           obi-2m59b                                 1/1    Running  k3d-lab-ebpf-server-0
obi           obi-4lshq                                 1/1    Running  k3d-lab-ebpf-agent-0

What to check:

• The two api pods are on different nodes. The spread constraint is ScheduleAnyway, so if both land on one node, delete one and let it reschedule; experiments 1 and 4 need one per node.
• Redis is on the server node, where its PV is. If it’s Pending, its claim couldn’t bind: check kubectl get pv,pvc -A (below).
• Each eBPF tool has one pod per node, and Tetragon and Falco show 2/2 (the agent plus its export or rules sidecar).

The claims are bound to the static PVs:

$ kubectl --context k3d-lab-ebpf get pv
NAME                      CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                                                                   STORAGECLASS
ebpf-lab-falco-ui-redis   1Gi       RWO           Retain          Bound   falco/falco-falcosidekick-ui-redis-data-falco-falcosidekick-ui-redis-0  lab-local
ebpf-lab-redis            1Gi       RWO           Retain          Bound   demo/redis-data                                                         lab-local

And the platform sees the lab:

$ ~/gitrepos/iac/infra-local/host/labs/lab.sh status lab-ebpf
https://172.30.0.136:6443
  cert-manager-lab-ebpf                Synced/Healthy
  prometheus-operator-crds-lab-ebpf    Synced/Healthy
  opentelemetry-operator-lab-ebpf      Synced/Healthy
  kube-state-metrics-lab-ebpf          Synced/Healthy
  monitoring-pack-lab-ebpf             Synced/Healthy

There are no iptables Service rules on the nodes, only Cilium’s own chains:

$ docker exec k3d-lab-ebpf-server-0 sh -c 'iptables-save | grep -c KUBE-SVC; iptables-save | grep -c CILIUM'
0
42

Memory, as Docker counts it, a few minutes after the rebuild: 3.8 GiB for the server node and 1.8 GiB for the agent.

Smoke Test

$ curl -s -o /dev/null -w "page %{http_code}\n" 127.0.0.1:8091/
page 200
$ curl -s 127.0.0.1:8091/api/visits
{"served_by":"api-5df5d59bd7-vss5p","visits":7122}
$ curl -s 127.0.0.1:8091/api/items
[{"id":1,"name":"bpf_map"},{"id":2,"name":"kprobe"},{"id":3,"name":"xdp"}]
$ curl -s -o /dev/null -w "hubble ui %{http_code}\n" 127.0.0.1:8092/
hubble ui 200
$ curl -s -o /dev/null -w "falcosidekick ui %{http_code}\n" 127.0.0.1:8093/
falcosidekick ui 200

/api/visits returning a count proves the whole path: the NodePort, Cilium’s Service translation to web and then to an api pod, DNS for redis, and Redis on its PV. The count is high because the counter has survived every rebuild of the lab since it was first built.

Then open http://127.0.0.1:8091 and click Count a visit a few times; served_by changes between the two API pods. In http://127.0.0.1:8092, pick namespace demo and you’ll see the same calls as a service map.

Finally, the eBPF agents’ own checks:

make hubble-flows      # the last 20 flows in namespace demo
make obi-metrics       # the RED counters OBI has right now
make tetragon-events   # process events from demo in the last 2 minutes
make falco-events      # Falco alerts in the last 5 minutes

Common Failures

Most of these were hit while building or capturing this lab, and their fixes are already in the repo. The first two, the 401 and the last row are ones the setup invites but these runs didn’t meet.

Symptom	Cause	Fix
k3d cluster create fails: port already allocated	Something else uses 6450, 8091, 8092 or 8093	Free the port, or change k3d/cluster.yaml and the card’s deploy.ports, then make rebuild
Nodes NotReady for minutes	Cilium not installed or not healthy yet	kubectl -n kube-system get pods -l k8s-app=cilium; make cilium
After a k3d cluster start, Cilium crash-loops	Docker dropped the bpffs mounts when the node container restarted	make node-prep, then restart the Cilium pods
Pods reach the API server on its node IP but not on 10.101.0.1	Socket-level load balancing at a cgroup root that pod sockets aren’t under (nested containers)	socketLB.hostNamespaceOnly: true (already set)
Tetragon or OBI time out talking to the API	Host-network pods can’t use the API’s ClusterIP here	The Makefile passes KUBERNETES_SERVICE_HOST=<server node IP>; reinstall with make tetragon / make obi
OBI runs but produces nothing for demo	Same cause: without the API it has no pod metadata, so the namespace rule matches nothing	As above
Falco can’t download its rules; DNS lookups from pods time out	The node’s resolver is Docker’s, which pods can’t use	make dns
No OBI data in Grafana, OBI logs a 401	obi/otlp-token missing or stale	make monitoring, then kubectl -n obi rollout restart ds/obi
Grafana has OBI metrics but panels show “No data”	Short rate() windows: OBI only pushes every 60 s	The dashboard uses [3m]; for your own queries, use 2m or more
A Tetragon policy loads but never fires	Matching on the resolved binary: busybox’s /usr/bin/wget is a symlink to /bin/busybox	The lab matches execve’s first argument, the path as passed (see Part 6)
make down says the cluster is gone, but make cluster thinks it exists	Nodes full of eBPF programs take longer to stop than Docker’s kill timeout; k3d leaves containers behind	make down now removes leftovers itself and fails if any remain
Redis Pending after a rebuild	The PV wasn’t recreated before the app, or the node name changed	make storage, and keep the cluster name lab-ebpf

Check Your Understanding

Why must cilium run before dns and app?

Until Cilium is installed the nodes have no CNI: CoreDNS and every app pod would stay Pending or ContainerCreating because no pod can get a network. make cilium also waits for CoreDNS to be ready before the DNS settings are changed.

The cluster has no kube-proxy. What does iptables-save | grep -c KUBE-SVC returning 0 prove, and what doesn't it prove?

It proves no kube-proxy-style Service rules exist on the node. It doesn’t prove Services work; the working curl to /api/visits does, and cilium-dbg service list shows where the translation lives.

served_by keeps switching between two pod names. Who decides which pod answers?

Cilium’s eBPF program on the web pod’s network interface, when nginx connects to the api Service’s ClusterIP. It picks one of the Service’s two backends per connection, and the API opens a new connection per request, so consecutive requests can land on different pods.