eBPF Lab: Overview and Architecture

This lab swaps a k3d cluster’s networking for Cilium, gets request metrics and traces out of a Go service that has no telemetry code, and catches (then kills) a download attempt inside a container. All of it is done with eBPF. By the end of this part you’ll know what the lab builds, what your machine needs, and which program sees which part of a request.

What You Will Build

A two-node k3d cluster, lab-ebpf, running a deliberately plain app:

• web: nginx serving one page, proxying /api/ to the API.
• api: a Go HTTP service with two replicas, one per node. It counts visits in Redis. There’s no OpenTelemetry, no metrics library and no logging middleware in it, on purpose.
• redis: one Redis with append-only persistence, on a PersistentVolumeClaim bound to a static local PersistentVolume. The counter survives deleting the whole cluster.

Around the app, four eBPF tools watch from the kernel:

Tool	What it does here
Cilium	The cluster’s only network layer. It replaces flannel, kube-proxy and k3s’s network policy controller. Hubble shows every flow it forwards or drops
OpenTelemetry eBPF Instrumentation (OBI)	Reads the app’s HTTP and Redis traffic and turns it into request-rate, error and duration metrics and trace spans
Tetragon	Records process executions in the app’s pods and enforces a policy in the kernel: it kills any process that tries to start wget, curl or nc
Falco	Matches syscalls against rules and raises alerts, such as a shell started in a demo pod

The lab bundles no Grafana, Prometheus, Loki or Tempo. It’s a lab on the Cloudutsuk homelab platform (lab contract v2): make up registers the cluster with the platform, Argo CD installs the monitoring pack into it, and every metric, log and trace lands in the shared mon cluster under cluster="lab-ebpf". The lab’s own dashboard is in mon’s Grafana, folder Labs.

The repo is ~/gitrepos/observability/ebpf-lab. From there, one command brings it all up:

make up

Prerequisites

The lab was built and captured on a DGX Spark (GB10, arm64) running the homelab’s other clusters at the same time. These are the versions it ran with:

Tool	Version used	Check
Linux kernel	6.17 (6.17.0-1026-nvidia, aarch64)	uname -r -m
Docker	29.2.1	docker version
k3d	v5.9.0 (k3s v1.35.5-k3s1)	k3d version
kubectl	v1.36.2	kubectl version --client
Helm	v4.3.0	helm version --short
jq, curl, make, Python 3	jq 1.7, Python 3.12.3	jq --version
The homelab platform	core, platform and mon running; infra-local at ~/gitrepos/iac/infra-local (or HOMELAB_INFRA)	kubectl config get-contexts

What the lab takes from the host:

• Ports, all on 127.0.0.1: 6450 (Kubernetes API), 8091 (the app), 8092 (Hubble UI), 8093 (Falcosidekick UI).
• Pod and Service ranges 10.100.0.0/16 and 10.101.0.0/16, the first pair the platform reserves for labs. The OpenTelemetry lab uses the next pair, so both labs can run together.
• Memory: the lab card reserves 7 GB, which covers the lab’s stack plus the monitoring pack and cert-manager (about 1 GB). The platform’s lab runner checks this before it deploys a lab from the portal.
• Disk: the PV folders under /srv/homelab/labs/ebpf-lab are small. The images come through the homelab’s Zot mirror; the Helm charts are vendored in the repo’s charts/ folder.
• A read-only Grafana token at ~/.config/homelab/grafana-labs-token (infra-local host/monitoring/labs-grafana-token.sh) for the make verify-* checks that ask mon.

eBPF programs run in the host’s kernel, not in the node containers. On a shared host, Tetragon and Falco see processes from every cluster and from the host itself. Part 3 shows how the lab scopes them down to namespace demo.

Architecture

Step through one visit, then follow each tool’s signal to where you’ll read it:

1. A visit: curl or the Count a visit button calls /api/visits on 127.0.0.1:8091, which k3d maps to NodePort 30091 on the server node.
2. No kube-proxy: Cilium's eBPF programs translate the NodePort to the web pod. k3s runs with no kube-proxy and no iptables Service rules.
3. nginx proxies /api: nginx sends it to the api Service on HTTP :8080. Cilium picks one of two api pods on the pod's veth, sometimes on the other node, over VXLAN.
4. INCR in Redis: The Go API opens a connection, sends INCR visits to Redis :6379 in its own protocol and returns the count. It has no tracing, metrics or logging library.
5. Data on a static PV: Redis appends every write to its AOF on a PVC bound to a static local PV, so the counter outlives the whole cluster.
6. eBPF sees it all: Four eBPF tools watch from the kernel the k3d nodes share: packets (Cilium), HTTP and Redis calls (OBI), process executions (Tetragon, Falco).
7. Flows in Hubble: Cilium records every flow with its source and destination identity and verdict. Hubble Relay gathers both nodes for the UI.
8. OBI to mon, directly: OBI turns the traffic into RED metrics and spans and sends OTLP straight to mon's gateway, because a host-network pod can't reach the agent here.
9. Security events: Tetragon and Falco write JSON events to stdout; the platform's agent tails them like any pod log. Falcosidekick also shows Falco's in its UI.
10. The monitoring pack: The agent collector, installed by Argo CD when make up registered the lab, ships logs and metrics to the gateway with the labs token.
11. One cluster label: The gateway turns k8s.cluster.name into cluster=lab-ebpf; Mimir, Loki and Tempo keep the lab's data next to every other cluster's.
12. The lab's dashboard: Grafana on mon shows the eBPF lab dashboard in folder Labs, every query filtered on lab-ebpf.

Component	Role	Runs where
web (nginx 1.29)	Serves the page, proxies /api/	Deployment, 1 replica, NodePort 30091 (host 8091)
api (Go)	/api/visits, /api/items, /healthz	Deployment, 2 replicas spread over both nodes
redis (7.4.2)	The visit counter, append-only file on a PVC	Deployment, 1 replica, on the server node (where its PV is)
Cilium 1.20.2 + Hubble	CNI, kube-proxy replacement, network policy, flow visibility	DaemonSet cilium per node, Hubble Relay and UI (NodePort 30092)
OBI 0.13.0	HTTP and Redis RED metrics and spans from eBPF	DaemonSet obi on the host network, namespace obi
Tetragon 1.7.1	Process events and in-kernel enforcement	DaemonSet tetragon, events on its export-stdout sidecar
Falco 0.45 + Falcosidekick	Syscall rules, alerts and an event UI	DaemonSet falco (modern eBPF probe), UI on NodePort 30093
Monitoring pack	Agent and cluster collectors, kube-state-metrics	Namespace monitoring, installed by Argo CD from the platform
mon	Gateway, Mimir, Loki, Tempo, Grafana	The homelab’s monitoring cluster, shared by every cluster

The two node containers share the host’s kernel, so “per node” for an eBPF tool means “per k3d node container”: each DaemonSet pod loads its programs into the same kernel, and filters by the processes and interfaces of its own node.

Building Blocks

eBPF

eBPF lets you load small, verified programs into the Linux kernel and attach them to events: a packet arriving on an interface, a function being called, a syscall being made. The programs keep state in maps that user-space agents read. Every tool in this lab is such an agent. None of them needs a change to the app or a sidecar in its pods.

Cilium and Hubble

Cilium is a Kubernetes network layer built on eBPF. Here it does three jobs that k3s would otherwise do with flannel, kube-proxy and iptables: pod networking (VXLAN between the two nodes), Service load balancing, and network policy, enforced by security identity rather than IP address. Hubble is its observability layer. Part 3 walks through its settings. The blog’s Network Policies post covers the Kubernetes side of policy.

OpenTelemetry eBPF Instrumentation (OBI)

OBI is the OpenTelemetry project that Grafana Beyla was donated to. It attaches eBPF probes to processes it’s told to watch, decodes protocols such as HTTP and Redis on their sockets, and exports OpenTelemetry metrics and spans. Part 2 explains what that means for an app that has no telemetry code.

Tetragon

Tetragon, from the Cilium project, traces process execution and arbitrary kernel functions, adds Kubernetes metadata, and can act inside the kernel: send a signal, override a return value. The lab uses it to kill network tools before they start.

Falco

Falco, a CNCF project, watches syscalls with its modern eBPF probe and evaluates rules against them. It detects and reports; it doesn’t block. Falcosidekick forwards its alerts and runs a small web UI.

k3d, Redis and the platform

k3d runs k3s nodes as Docker containers. Redis keeps the counter. The platform pieces (Argo CD, the monitoring pack, mon) are the same ones every homelab cluster uses; the OpenTelemetry lab is the other lab built on them.

What You Will Learn

1. How a cluster with no kube-proxy still load-balances Services, and how to see every flow with Hubble (Experiment 1).
2. How Cilium tells pods apart by identity, not IP, and drops a connection the policy doesn’t allow (Experiment 2).
3. How OBI produces per-route request rate, latency and Redis metrics for a service with no metrics code (Experiment 3).
4. When OBI’s trace context joins web and API spans into one trace, and when it doesn’t (Experiment 4).
5. What detection (Falco) and enforcement (Tetragon) each record when the same command runs, and why Falco never sees the download Tetragon stopped (Experiments 5 and 6).
6. How static local PVs keep data through a full cluster rebuild without a hostPath in any pod (Experiment 7).

Check Your Understanding

The API has no OpenTelemetry SDK. Where can request metrics for it come from?

From outside the process. OBI’s eBPF probes see the API’s HTTP requests and responses and its Redis commands at the kernel and runtime level, time them, and export the result as OpenTelemetry metrics and spans. The API’s code never knows.

Why do Tetragon and Falco need filters for namespace demo on this machine?

Because their programs run in the one kernel every container on the host shares. Without a filter they would report processes from the other homelab clusters, Docker and the desktop too. Their Kubernetes metadata only covers pods of lab-ebpf, so the lab keeps events whose namespace is demo.

The lab has no Grafana of its own. How does a metric from OBI reach a dashboard?

OBI exports OTLP to the homelab’s gateway on mon, with the shared labs token and k8s.cluster.name=lab-ebpf. The gateway turns that into the label cluster="lab-ebpf" and writes the metric to Mimir. The lab’s dashboard in mon’s Grafana filters every query on that label.

What would you need to change to run a second copy of this lab next to the first, besides the cluster name?

All four host ports, 6450 (API), 8091, 8092 and 8093, and the pod and Service ranges. k3d fixes port mappings when it creates the cluster, and two clusters on the homelab network can’t share a range.