Cloudutsuk Platform: Technical Architecture

The blog is a static site on GitHub Pages, and that one fact shapes this whole architecture. A login, a cost dashboard, one-click deploys and a chatbot all need a backend, and GitHub Pages can’t run one. So the backend lives on the NVIDIA DGX Spark under my desk. Nothing on it is published on the internet: my own devices reach it over a private Tailscale network, and the blog only describes it. Everything below follows from that choice. The risk that matters most is that the whole platform shares one machine, and one pool of unified memory, with the GPU.

Living document. This project is in the planning stage. The architecture below is the design as approved before any code was written. Each delivery phase updates this post with what was actually built, and design changes arrive as new ADRs.

System context and component architecture

There are six zones:

• the edge (Cloudflare, plus the sources the platform reads)
• a long-lived platform cluster with the portal and its data stores
• a long-lived monitoring cluster
• a core cluster that boots first and holds identity (Keycloak), secrets (OpenBao), internal DNS (PowerDNS) and the chaos control plane (LitmusChaos)
• an ai cluster that serves the models on the GPU
• any number of disposable lab clusters, one per lab All of them are k3d clusters on one Docker network called homelab, so they reach each other by name. The diagram below opens one layer deep. Use the level buttons or click a cluster to open it, drag and zoom like a map, and hover an arrow for its protocol and why it exists. The tabs at the top add the networking, observability, security, CI/CD, and chaos and scaling connections. Pick a use case on the left and press play to follow a packet through it:

Ctrl + scroll (or pinch) to zoom

Laying out the diagram…

Cloudutsuk Platform system architecture

• You (phone, laptop)
• Tailscale (private network)
  • Policy (only my login)
  • Subnet router (tailscaled on the DGX)
• Homelab (DGX Spark · k3d)
  • DGX host services (systemd and Compose): Lab tooling, Host monitoring, Docker over mTLS
  • core cluster (boots first): Identity + secrets, Zot, PowerDNS, Backups + chaos
  • platform cluster (portal, GitOps, CI): Portal, Platform services, Delivery, platform-db
  • ai cluster (GPU): AI workloads, Monitoring pack, External Secrets
  • mon cluster (observability): OTLP gateway, Grafana LGTM
  • Lab clusters (contract v2): Lab app, OBI, Add-ons
• Clouds (read-only)
  • Azure
  • AWS
  • GCP
• GitHub (repos, Apps)
  • Blog + repos (content)
  • homelab-gitops (desired state)
  • gitops-writer App (contents:write)
• External services (SaaS)
  • Slack (alerts)
  • healthchecks.io (dead-man's switch)
  • Let's Encrypt (ACME · Cloudflare DNS-01)
  • Azure Blob (Velero backups)
• More capacity (later)

Sign in to the portal: How a device reaches the portal: a private tailnet, one Keycloak sign-in, nothing open on the internet.

1. You open the portal: From your phone or laptop on the tailnet; split DNS resolves the name to 172.30.0.2. The core cluster booted first, so Keycloak is ready.
2. Into the tailnet: Your device sends the request over WireGuard into the private tailnet. Nothing at home listens on the internet.
3. Policy and route: The policy lets only your login through, and tailscaled on the DGX routes the packets onto the homelab bridge to platform's Traefik, which ends TLS.
4. To Backstage: Traefik forwards the request to Backstage inside the platform cluster.
5. Sign in through Keycloak: Backstage sends you to Keycloak, which brokers your Entra ID login and maps your groups to roles. Network access alone gets nobody in.
6. Look up your account: Keycloak reads its realm, your Entra ID link and your groups from core-db, and gets them back.
7. Portal page returns: The portal page travels back the same way: through Traefik, the subnet router and the tailnet to your device.

Deploy a lab: One click on a lab tile: the runner runs make up, and the lab registers itself with the platform (contract v2).

1. Deploy clicked: The Labs plugin calls the runner on the DGX with its bearer token. The runner checks the card's ports and the memory budget and takes a per-lab lock.
2. Cluster created: The runner runs the lab's make up, which creates a k3d cluster such as lab-otel on the homelab network, pulling images through Zot.
3. lab.sh register: The end of make up runs infra-local's lab.sh register; registration is the lab's own job, not the runner's.
4. Access and token: lab.sh creates the argocd-manager ServiceAccount in the lab and copies in the shared labs token for the gateway.
5. Registered with Argo CD: The lab's cluster secret goes into platform, labelled for labs, so only cert-manager and the monitoring pack are installed.
6. Add-ons installed: The ApplicationSets install cert-manager and the monitoring pack; lab.sh waits until they are Healthy (up to 15 minutes on a first run).
7. Dashboards loaded: The lab's dashboard JSON is loaded into mon's Grafana, folder Labs; make down removes it again.
8. Lab ready: The runner reports the job as done (the OpenTelemetry lab took 8 min 48 s) and the tile shows its links.

Monitor a lab: Telemetry streams from the lab into mon's gateway, and you read it in Grafana filtered on the lab's cluster.

1. The app sends OTLP: The lab's app sends OTLP to the in-cluster agent collector, which adds pod metadata and the cluster name.
2. Out to mon: The agent forwards everything to otlp.mon.home.cloudutsuk.com over HTTPS with the shared labs bearer token.
3. Stored in LGTM: The gateway writes metrics to Mimir, logs to Loki and traces to Tempo, kept about 15 days.
4. You open the dashboard: grafana.mon.home.cloudutsuk.com is reached over the tailnet; Grafana signs you in through Keycloak.
5. Panels query Mimir: Every panel's query filters on the lab's cluster, because mon holds every cluster's signals.
6. Dashboard shown: The filled dashboard returns to your browser.

Alert to Slack: A rule fires in Mimir's ruler and reaches Slack; the Watchdog proves the alert path itself works.

1. Metrics arrive (fire): The embeddings model's pod isn't ready (on its first deploy it waited 15 minutes for its 22 GB image).
2. Stored in Mimir (fire): The gateway writes the ai cluster's metrics to Mimir with cluster="ai".
3. Rules evaluated: The ruler evaluates every alert rule from homelab-gitops against the stored metrics, once a minute.
4. Alert fires (fire): The condition has held for its 10 minutes, so the ruler sends the alert to Mimir's Alertmanager.
5. Posted to Slack (fire): Alertmanager posts the alert to the Slack channel; it posts again when the pod is ready and the alert resolves.
6. Watchdog fires (watchdog): The Watchdog rule always fires, so the alert path is exercised every minute.
7. Ping healthchecks.io (watchdog): If the pings stop (Mimir down, Alertmanager broken, mon gone), healthchecks.io emails me.

Nightly backup: At 02:00 Velero dumps the databases that can't be rebuilt, plus OpenBao, and uploads them to Azure Blob.

1. Dump Keycloak's database: Before core's backup, the backup-dumps hook takes a pg_dump of Keycloak's database through pg_read_all_data.
2. Snapshot OpenBao: The hook logs in with its own Kubernetes identity and a snapshot-only policy, and takes OpenBao's snapshot.
3. Upload to Azure: Velero uploads with Kopia to the Cool-tier account; nightly backups are kept 30 days, weeklies 12 weeks.
4. Dump platform-db: In platform, the same pattern dumps platform-db, where the portal keeps its data.
5. Upload platform: platform's Velero uploads its own backup to the same container.
6. Check in the UI: velero.platform.home.cloudutsuk.com lists core's, platform's and ai's backups with their size and status.

Code to cluster: A push to hello-world becomes a scanned, signed image and a digest commit that Argo CD deploys (M1: 8 min 25 s).

1. Commit pushed: A pushed commit (or a merged pull request) queues hello-world's workflow on GitHub.
2. Runner picks up the job: A runner pod in platform polls GitHub for jobs (no inbound connection). Ruff and pytest run first, in 13 s.
3. Log in to OpenBao: The runner logs in with GitHub's OIDC token and gets the Zot push key; nothing is stored as a GitHub secret.
4. Gates, then build: Gitleaks, Semgrep, Bandit, Trivy, Checkov, Conftest and Hadolint pass, then BuildKit builds and returns an OCI layout that is scanned and given an SBOM.
5. Push and sign: On main, crane pushes the scanned bytes to Zot, and Cosign signs them and attests the SBOM with the transit key.
6. Digest commit: The promote job mints a short-lived App token and commits the digest to apps/hello-world, then revokes the token.
7. Argo CD syncs: Argo CD notices the digest commit on its next poll and reads the changed manifests.
8. Admitted and applied: Kyverno verifies the signature and SBOM attestation in Zot and admits the Deployment, which Argo CD applies.
9. Image pulled: The new pod pulls the signed image from Zot by digest and starts.
10. Smoke test: The PostSync smoke test passes, so the sync succeeds and the app is Synced and Healthy on the new SHA.

Admission denied: Kyverno refuses an image that isn't signed in Zot, so an unsigned build can never run.

1. A change is synced: Argo CD picks up a commit that points at a platform/* image.
2. Admission review: Argo CD applies through the Kubernetes API, which sends the Pod spec to Kyverno's webhook for its seven policies.
3. Look up the signature: Kyverno fetches the image's referrers from Zot and checks them against the committed Cosign public key.
4. Denied (unsigned): No valid signature, so the API server refuses the object; the app shows the error and nothing runs.
5. Admitted (signed): The signature and SBOM verify, every other policy passes, and the object is created.

Resources and cost: Planned (phase 4): how the inventory collects cloud resources and cost, and how you see them.

1. Credentials from OpenBao: External Secrets fetches the read-only cloud credentials from OpenBao (HTTPS 443 through Traefik on core); no secret lives in git.
2. Read the clouds: Every 6 hours (or on Sync now) the inventory calls the cloud APIs over HTTPS and gets resources and cost back.
3. Store the results: Resources and daily cost go into platform-db.inventory (PostgreSQL, 5432), grouped by lab and repo tags.
4. You look at cost: The Inventory plugin in Backstage calls the inventory service and renders resources and cost per lab.

Price a lab on the DGX: Planned (phase 4): OpenCost on each k3d cluster splits the DGX's cost by cluster, namespace and lab.

1. Daily cost sync: Once a day the inventory asks each cluster's OpenCost for yesterday's cost by namespace, with a Keycloak token.
2. Read the usage: OpenCost reads only its own cluster's series from Mimir and prices them with the DGX's custom per-hour prices.
3. Allocation returns: The allocation comes back; GPU-hours go to the pods that requested the GPU.
4. Store it with the cloud costs: The rows are stored under the dgx provider, next to Azure, AWS and GCP, so a lab's local and cloud cost roll up together.
5. See the lab's cost: The portal's cost view shows each lab's DGX cost beside its cloud cost.

Join a new cluster: Planned (phase 1a): Karmada places the shared add-ons on a new lab cluster by label.

1. Register the cluster: At the end of make up, lab.sh also registers the cluster with Karmada in pull mode and labels it with its role.
2. The agent asks for work: The lab's karmada-agent dials out to Karmada's API server (default port 5443); propagation policies match the lab label.
3. Apply the add-ons: The agent applies the add-ons locally, with overrides such as the cluster's name.
4. Report status: The agent reports each add-on's status, which Karmada aggregates across every cluster.
5. One health in Argo CD: Argo CD reads the aggregated status from Karmada, so one app shows the add-ons' health on every cluster.

Run a chaos experiment: Planned (phase 2a): a fault in a lab, run from ChaosCenter in core, watched in Grafana and scored.

1. Start the experiment: You start an experiment from the library synced from homelab-gitops; ChaosCenter is never published to the internet.
2. Record the run: ChaosCenter records the run in its MongoDB (default port 27017) before any agent sees it.
3. The agent picks it up: The lab's chaos agent dials out to ChaosCenter in core and gets the experiment; nothing connects into the lab.
4. Inject the fault: The chaos operator injects the fault within the blast radius set in git, while its probes check the app keeps answering.
5. Watch the effect: The monitoring pack streams the lab's signals to mon as usual, next to the chaos exporter's metrics.
6. Report the result: The agent reports the probe results; ChaosCenter turns them into a pass or fail and a resilience score.
7. Read the score: You see the verdict and score next to the Grafana dashboards for the same time window.

Wake an idle model: Planned (phase 5): KEDA loads a rarely used model on demand and unloads it when idle.

1. A request for a sleeping model: A caller asks the gateway for an on-demand model; the gateway queues the request while the model is asleep.
2. KEDA sees the demand: KEDA's Prometheus scaler reads the gateway's per-model request rate from Mimir over HTTPS.
3. Load the model: If its memory slice fits what's free on the host, KEDA scales the model to one copy and vLLM loads it.
4. Serve the request: The queued request is answered. After 30 quiet minutes KEDA scales the model back to 0 and its memory returns.

Ask the chatbot: Planned (phase 5): a question goes through the RAG API, cache first, then retrieval and the local model.

1. Question sent: The chat plugin sends your question to the RAG API in the ai cluster over HTTPS.
2. Check the cache: The RAG API looks for a semantically similar question in Redis (6379).
3. Instant answer (hit): A cache hit (same index version) answers immediately, without retrieval or the model.
4. Retrieve documents (miss): On a miss, Qdrant (6333) returns the closest chunks, filtered by repo, lab or cloud.
5. Prompt to the model (miss): The prompt with the chunks goes to the LiteLLM gateway (4000), which routes it to vLLM chat (8000) on the GPU.
6. Answer streams back (miss): The answer streams back through the gateway to the RAG API.
7. Trace recorded (miss): The request's spans (cache check, chunks and scores, prompt, tokens) go to Phoenix.
8. Answer shown (miss): The answer, with citations and a View trace link, is shown in the portal.

Keep knowledge fresh: Planned (phase 5): every 6 hours, changed files are re-embedded into Qdrant.

1. Clone the repos: rag-ingest clones the blog and repos from GitHub over HTTPS with a read-only token; only pushed work is indexed.
2. Embed changed files: Only changed files are sent through the gateway (4000) to the embeddings model (8000), which already runs today.
3. Vectors come back: The embeddings come back through the gateway to rag-ingest.
4. Update Qdrant: New chunks are written to Qdrant and removed files deleted; the cache invalidates with the index version.

SRE agent handles an alert: Planned (phase 6): an alert becomes a root cause and a reviewed fix, never an unapproved change.

1. Alert fires: Alertmanager in mon POSTs to the SRE agent's /alerts webhook with the cluster, namespace and pod labels.
2. Read the cluster: The agent reads the lab cluster through the Kubernetes API with a 15-minute read-only token.
3. Find runbooks: It searches the knowledge base through the RAG API for matching runbooks.
4. Ask for a root cause: It asks the local model, through the gateway, for a root cause and a proposed fix.
5. Diagnosis returns: The model's diagnosis comes back through the gateway.
6. Post to Slack: The root cause and proposed fix are posted to Slack with approve and reject buttons.
7. Approve, open a PR: Approving opens a pull request to homelab-gitops; nothing changes without your approval.
8. Argo CD syncs the fix: Once merged, Argo CD syncs the fix, so every change is reviewed and reversible.

Component	Runs in	Job
Tailscale	tailscaled on the DGX host (subnet router), the app on my phone and laptop	Private path to every cluster’s address on the homelab bridge; split DNS for home.cloudutsuk.com; only my login is allowed in (ADR 0037)
Backstage (self-service-portal)	platform	The portal: inventory and cost, labs, chat, Argo CD and Kubernetes views
Inventory	platform (CronJob, 6 h + Sync now)	Pulls resources and cost from Azure, AWS, GCP and the DGX into Postgres
Lab runner	DGX host (systemd), called from platform	Runs a lab’s make up / make down, registers the cluster with Argo CD
Argo CD	platform	One instance that deploys to every cluster from homelab-gitops
Karmada control plane	platform	Places the shared add-ons (monitoring pack, External Secrets, ExternalDNS, cert-manager, chaos agent) on clusters by label, with per-cluster overrides (phase 1a, proposed)
Karmada agent	every member cluster, including labs	Pull mode: dials out to Karmada, applies the add-ons placed on its cluster and reports their status
RAG API, rag-ingest	ai cluster	Chat retrieval with a semantic cache; scheduled indexing of the blog and repos
Qdrant, Redis	ai cluster	Vectors and the semantic cache
Postgres	platform	Relational state: Backstage, inventory and cost, lab jobs
Arize Phoenix	ai cluster	Shows every chat and agent request as a trace: retrieved chunks and scores, prompt, tokens, latency per step
Model gateway (LiteLLM)	ai cluster	The one OpenAI-compatible endpoint for every model call: routing, queueing, limits, token counts, later overflow
vLLM chat + vLLM embeddings	ai cluster (GPU)	Local chat model and embedding model, each with its own memory slice
KEDA	ai cluster	Scales on-demand models from 0 to 1 when the gateway sees demand and back to 0 when idle, so an unused model frees its memory (phase 5)
SRE agent	platform	Alert-triggered diagnosis with human approval before any change
mon cluster	its own k3d cluster	OpenTelemetry gateway, Grafana LGTM, Alertmanager; ready-made dashboards for the fleet, Kubernetes, data stores, AI serving and the platform
Monitoring pack	every cluster, including labs	OTel collectors, kube-state-metrics and ServiceMonitor discovery, installed by one ApplicationSet
OpenCost	every k3d cluster, with the monitoring pack	Prices what each cluster, namespace and pod uses with the DGX’s own hourly cost; the inventory reads it for the DGX’s line in the cost view (phase 4)
Keycloak	core cluster	The one identity provider: brokers Google and Entra ID, issues tokens to people and services
OpenBao	core cluster	The one secret manager; External Secrets in every cluster pulls from it; issues short-lived Kubernetes tokens
Zot	core cluster	The container registry: our signed images, plus a cache of every upstream image the clusters pull
PowerDNS	core cluster	Internal zone home.cloudutsuk.com, fed by ExternalDNS in every cluster
LitmusChaos ChaosCenter + MongoDB	core cluster	The chaos control plane: the experiment library (synced from homelab-gitops), schedules, results and resilience scores (phase 2a)
Litmus chaos agent	each target cluster, labs first	Dials out to ChaosCenter, runs experiments through the chaos operator, exports their metrics to the monitoring pack
Lab clusters	one k3d cluster per lab	The lab’s app plus an OTel collector installed by Argo CD

Two more diagrams are planned for this playbook. Commit to deploy goes in the CI/CD post and RAG chat in the Data post. They’ll be added when those areas are written.

Technology and framework selection

The rule I used: reuse what I already run, and only add a new tool where nothing I have does the job.

Need	Choice	Why this and not the alternative
Portal	Backstage (existing self-service-portal)	SSO providers, a software catalog, a Kubernetes plugin and an Argo CD plugin already exist. A custom FastAPI + React app would be lighter but means building all of that again
Local Kubernetes	k3d	Already used by every lab. Clusters are cheap to create and destroy
Cluster lifecycle	Terragrunt units from Y-IaC: docker_network and k3d_config_cluster, which renders k3d’s own config file and runs the k3d CLI	The long-lived clusters get the same plan/apply workflow as the cloud infrastructure, and plan shows the exact k3d config. The older provider-based unit crashed whenever update.k3s.io was down and couldn’t set host aliases (ADR 0006)
GitOps	Argo CD	Its UI and Backstage plugin suit a portal-centric setup better than Flux
Multi-cluster placement	Karmada (CNCF), alongside Argo CD	Argo CD ApplicationSets already fan add-ons out, and they stay the fallback. Karmada adds placement by cluster label, per-cluster overrides and one API across every cluster, without changing a single manifest. Open Cluster Management and Rancher Fleet were the alternatives; Fleet would replace Argo CD, which this platform keeps
CI	GitHub Actions self-hosted runners (ARC) + BuildKit	The repos are already on GitHub, and self-hosted minutes are free
Kubernetes cost allocation	OpenCost (CNCF, Apache 2.0) on every k3d cluster	The open-source standard for “what does this namespace cost”. Kubecost adds a UI and savings reports on top, but its free tier is limited and it’s more to run. The cloud bills stay with the inventory’s own collectors, so OpenCost only prices the DGX
Observability	OpenTelemetry Collector + Grafana LGTM	The same stack the OpenTelemetry lab already teaches
LLM serving	vLLM	OpenAI-compatible API, continuous batching, runs on the GB10 GPU
Model gateway	LiteLLM proxy	One endpoint for all callers, so adding a model, a second machine or a hosted fallback doesn’t touch the services
Scale to zero	KEDA (CNCF graduated) in the ai cluster	Plain HPA can’t go to zero replicas, and CPU-based scaling is useless on one GPU. KEDA scales on a Prometheus query, so a rarely used model unloads when idle. Knative would also scale to zero, but needs its own serving layer
Vector store	Qdrant	Payload filters (repo, lab, cloud, path) and hybrid search
Semantic cache	Redis with RedisVL SemanticCache	Small, well understood, TTL and similarity threshold built in
RAG visualization	Arize Phoenix (self-hosted)	Understands LLM and retrieval spans (documents, scores, prompts), has an embeddings map and evals, and takes OpenTelemetry input like the rest of the platform. Langfuse is the fallback, but it also needs ClickHouse
Troubleshooting agent	existing sre-agent	It already has Kubernetes subagents, approval buttons and Slack
Testing	pytest + Testcontainers, Schemathesis, Playwright, Locust, Argo CD PostSync smoke tests	Locust scripts load in Python like the services, scales with workers, and its thresholds can fail a run. Schemathesis turns each OpenAPI spec into hundreds of API tests for free
Security scanning	Gitleaks, Semgrep + Bandit, Trivy (deps, IaC, images, licences), Checkov + Conftest, Hadolint, Syft, OWASP ZAP, Trivy Operator	All free and self-hosted. GitHub’s own code scanning needs a paid licence on private repos
Supply chain	Cosign signatures, SBOM attestations, Kyverno admission	Only signed images from our own registry can run, in every cluster
Chaos engineering	LitmusChaos (open source, CNCF), ChaosCenter in the core cluster	Chaos Mesh was the earlier note in ADR 0031 and is lighter, but Litmus adds a control plane for every cluster, a hub of ready-made experiments, probes that score each run, and experiments kept in git. Hosted tools such as Gremlin cost money and reach in from outside
Container registry	Zot in the core cluster	Harbor is the usual pick, but its official images are amd64 only and the DGX is arm64. Zot is a single multi-arch binary with access control, OIDC, a pull-through mirror, built-in CVE scanning and the OCI referrers API for signatures
Pod networking and policy	k3s defaults: flannel and its built-in NetworkPolicy controller	Enough for default-deny today. Cilium was considered and deferred: a lab first, then the long-lived clusters, see Cilium

GPU test, done 2026-10-11. vLLM runs inside the ai k3d cluster: a k3s image with NVIDIA’s container toolkit, the NVIDIA device plugin, and pods that ask for nvidia.com/gpu. A chat model and an embedding model (Qwen3-Embedding-0.6B) served requests at the same time on the one GPU. ADR 0010 is accepted. The details are under Scalability.

Chat model, chosen 2026-10-15 (ADR 0046). gpt-oss-20b, under Apache 2.0. Its MXFP4 weights take 13.8 GiB, so it runs in a slice of about 21 GB rather than the ~30 GB planned. Qwen3-30B-A3B in FP8 needs about 31 GB for its weights alone, more than the DGX had free that day. Callers reach it through LiteLLM as gpt-oss-20b, with OpenAI-style tool calls for the SRE agent.

Chat model, replaced 2026-10-16 (ADR 0049). Qwen3.5-35B-A3B in FP8, under Apache 2.0: a mixture of experts with 3B active parameters per token, so it decodes about as fast as gpt-oss on the GB10, with stronger tool calling and a 262k native context (served at 128k). Only one layer in four keeps a KV cache, so long contexts stay cheap. It runs in a slice of about 44 GB in the same vllm-chat app, as qwen3.5-35b-a3b behind LiteLLM. gpt-oss-20b had broken long agent runs on malformed tool-call output.

API and integration architecture

Backstage is the only thing a browser talks to. Its backend plugins call the internal services, which are plain HTTP + JSON APIs inside the platform cluster (the model gateway is in the ai cluster):

API	Consumers	Key endpoints
inventory	Backstage Inventory plugin, RAG API (tool)	GET /resources, GET /costs?group_by=lab, POST /sync
lab-runner	Backstage Labs plugin	GET /labs, POST /labs/{slug}/deploy, POST /labs/{slug}/destroy, GET /jobs/{id}/logs (stream)
rag-api	Backstage Chat plugin, SRE agent	POST /chat (streamed, with citations), POST /search, POST /reindex
Model gateway	RAG API, rag-ingest, SRE agent	OpenAI-compatible /v1/chat/completions, /v1/embeddings; routes to the vLLM instances
SRE agent	Alertmanager	POST /alerts (Alertmanager webhook payload)

Outbound integrations are read-only by default:

• Azure Resource Graph and Cost Management.
• AWS Resource Explorer and Cost Explorer.
• GCP Cloud Asset Inventory and the BigQuery billing export.
• GitHub, for repo content and CI.
• Slack, for approvals.

The lab registry isn’t a new API. It’s the lab cards already in this blog (_data/labs/*.yml). Each card gets a small deploy: block, so the blog and the portal read the same source:

deploy:
  targets: {up: up, down: down, status: status}   # plain Make target names, run in repo.path
  memory_gb: 4                                    # admission control reserves this much
  ports: [8081, 3002]                             # host ports the lab publishes, checked for clashes
  ttl_hours: 4                                    # cloud labs only: destroyed automatically at the TTL

The lab runner only accepts plain target names and an absolute repo.path under an allowed root, and skips any card that breaks those rules rather than failing. The learning-lab skill’s check.py applies the same rules, so a card that passes there is one the portal can deploy.

Data architecture and data models

Data lives in a handful of stores, and each entity has exactly one service that writes it. Other services read it through that service’s API, never straight from its tables. That rule keeps the ownership in the diagram below honest, and it’s why there are no foreign keys across databases.

Store	Cluster	Holds	Rebuildable?
Postgres platform-db	platform	Databases backstage, inventory, labs, agent, one owner role each	No: nightly pg_dump through Velero to Azure Blob
Postgres ai-db	ai	Schemas rag (ingest state, chat history) and phoenix (traces, evals)	Partly: rag rebuilds from git; chat history and evals are backed up
Postgres core-db	core	Keycloak’s realm, users’ links to Google/Entra, groups	No: nightly pg_dump through Velero, next to OpenBao’s snapshot
Qdrant	ai	Chunks of posts, lab cards and repos, with embeddings	Yes: re-ingest from git
Redis	ai	Semantic cache entries keyed by the index version	Yes: it’s a cache
Zot	core	Container images, signatures, SBOMs	Yes: rebuild from git (mirror refills itself)
MongoDB chaos-db	core	ChaosCenter’s experiment runs, results and resilience scores	Partly: experiment definitions rebuild from git; run history is an accepted loss
mon (Mimir, Loki, Tempo)	mon	Metrics, logs and traces for about 15 days	Accepted loss

The logical data model is drawn below as four views, one per owning service. Solid lines are real foreign keys inside one database. Dotted lines are soft references: an ID or slug held across stores and checked by the owning service. Entities owned by another service appear as small reference boxes with only their key. Each diagram has an expand button for full screen.

1. Labs and clusters. Written by the lab runner, in platform-db.labs.

erDiagram
    LAB ||--o{ LAB_DEPLOYMENT : "deployed as"
    CLUSTER ||--o{ LAB_DEPLOYMENT : hosts
    LAB_DEPLOYMENT ||--|{ LAB_JOB : runs
    LAB |o--o{ CLUSTER : "runs in"
    USER ||..o{ LAB_DEPLOYMENT : requests
    LAB {
        string slug PK "from the blog lab card"
        string lab_category
        string repo_path
        int memory_gb
        string ports
        bool uses_cloud
        int ttl_hours
    }
    CLUSTER {
        string name PK "k3d cluster name"
        string role "core platform mon ai lab"
        string lab_slug FK "lab clusters only"
        cidr pod_cidr
        cidr service_cidr
        bool argocd_registered
    }
    LAB_DEPLOYMENT {
        uuid id PK
        string lab_slug FK
        string cluster_name FK
        string requested_by "Keycloak subject"
        string status
        timestamp ttl_expires_at
        timestamp destroyed_at
    }
    LAB_JOB {
        uuid id PK
        uuid deployment_id FK
        string action "up down status"
        string status
        int exit_code
        string log_ref "Loki query"
    }
    USER {
        string sub PK "owned by Keycloak"
    }

2. Inventory and cost. Written by the inventory service, in platform-db.inventory.

erDiagram
    CLOUD_ACCOUNT ||--o{ RESOURCE : contains
    CLOUD_ACCOUNT ||--o{ SYNC_RUN : "synced by"
    RESOURCE ||--o{ COST_DAILY : costs
    LAB |o..o{ RESOURCE : "lab tag"
    LAB |o..o{ BUDGET : "limited by"
    CLOUD_ACCOUNT {
        uuid id PK
        string provider "azure aws gcp dgx"
        string account_ref "never published"
        string environment
    }
    RESOURCE {
        uuid id PK
        uuid account_id FK
        string native_id
        string type
        string region
        jsonb tags
        string lab_slug "from the lab tag"
        timestamp last_seen
        timestamp deleted_at
    }
    COST_DAILY {
        uuid resource_id PK "also FK to RESOURCE"
        date usage_date PK
        decimal amount
        string currency
    }
    SYNC_RUN {
        uuid id PK
        uuid account_id FK
        string kind "resources costs"
        string trigger "schedule manual"
        string status
    }
    BUDGET {
        uuid id PK
        string scope "cloud or lab"
        string scope_ref
        decimal monthly_limit
    }
    LAB {
        string slug PK "owned by the lab runner"
    }

3. Knowledge and chat. Written by rag-ingest and rag-api, in ai-db.rag, Qdrant and Redis.

erDiagram
    RAG_SOURCE ||--o{ RAG_DOCUMENT : contains
    RAG_DOCUMENT ||..|{ RAG_CHUNK : "split into"
    LAB |o..o{ RAG_DOCUMENT : about
    INDEX_VERSION ||..o{ CACHE_ENTRY : keys
    INDEX_VERSION ||--o{ CHAT_MESSAGE : "answered at"
    USER ||..o{ CHAT_MESSAGE : asks
    RAG_SOURCE {
        string repo PK
        string branch
        string last_indexed_commit
    }
    RAG_DOCUMENT {
        uuid id PK
        string repo FK
        string path
        string content_hash
        string lab_slug
        bool deleted
    }
    RAG_CHUNK {
        uuid id PK "Qdrant point"
        uuid document_id
        int chunk_index
        vector embedding
        jsonb payload "repo lab cloud path commit"
    }
    INDEX_VERSION {
        int id PK
        timestamp finished_at
        int chunks_changed
    }
    CACHE_ENTRY {
        string key PK "Redis, question vector"
        int index_version
        text answer
        timestamp expires_at
    }
    CHAT_MESSAGE {
        uuid id PK
        string user_sub "Keycloak subject"
        text question
        bool cache_hit
        int index_version FK
        string trace_id "Phoenix and Tempo"
    }
    LAB {
        string slug PK "owned by the lab runner"
    }
    USER {
        string sub PK "owned by Keycloak"
    }

4. Incidents. Written by the SRE agent, in platform-db.agent.

erDiagram
    CLUSTER |o..o{ ALERT_EVENT : "fires in"
    ALERT_EVENT ||--o{ INVESTIGATION : triggers
    INVESTIGATION ||--o{ PROPOSED_ACTION : proposes
    USER |o..o{ PROPOSED_ACTION : decides
    ALERT_EVENT {
        string fingerprint PK "Alertmanager"
        string cluster_name
        string alertname
        jsonb labels
        timestamp starts_at
    }
    INVESTIGATION {
        uuid id PK
        string alert_fingerprint FK
        string status
        text rca
        string trace_id "Phoenix and Tempo"
    }
    PROPOSED_ACTION {
        uuid id PK
        uuid investigation_id FK
        string pr_url "homelab-gitops PR"
        string status "proposed approved rejected"
        string decided_by "Keycloak subject"
    }
    CLUSTER {
        string name PK "owned by the lab runner and infra-local"
    }
    USER {
        string sub PK "owned by Keycloak"
    }

Who owns what. Each entity has one writer; everything else reads it through that service’s API.

Entity	Written by	Store	Source of truth	Main readers	Retention
LAB	lab runner (read-only copy)	platform-db.labs	The lab cards in this blog (_data/labs/*.yml), refreshed every sync	Backstage Labs plugin, inventory, RAG	Follows the cards
CLUSTER	infra-local (long-lived), lab runner (labs)	platform-db.labs	k3d itself, reconciled every sync	Argo CD registration, SRE agent, inventory	While the cluster exists, then 90 days
LAB_DEPLOYMENT, LAB_JOB	lab runner	platform-db.labs	Itself	Backstage, budgets, the TTL reaper	1 year
CLOUD_ACCOUNT, RESOURCE, COST_DAILY, SYNC_RUN, BUDGET	inventory	platform-db.inventory	The cloud providers’ APIs and billing exports	Backstage Inventory plugin, RAG tools, budget alerts	Resources 1 year after deletion; daily costs 2 years
ALERT_EVENT, INVESTIGATION, PROPOSED_ACTION	SRE agent	platform-db.agent	Alertmanager (alerts), GitHub (PR state)	Backstage incident view, Slack	1 year
RAG_SOURCE, RAG_DOCUMENT, INDEX_VERSION	rag-ingest	ai-db.rag	The git repos	rag-api	Rebuilt on demand
RAG_CHUNK	rag-ingest	Qdrant	The git repos	rag-api, SRE agent	Deleted with its document
CACHE_ENTRY	rag-api	Redis	Derived	rag-api	24-hour TTL, invalidated by a new index version
CHAT_MESSAGE	rag-api	ai-db.rag	Itself	You (history), Phoenix evals	90 days
USER	Keycloak	core-db	Google / Entra ID (brokered)	Every service, as a token subject only	While the account exists

Personal data stays minimal: only Keycloak stores emails. Every other table holds the opaque subject ID, so removing a user is a Keycloak change plus deleting their chat history.

Costs are grouped by the tags the infra-* repos already apply, plus one new optional tag, lab. Each Qdrant point’s payload carries source, repo, lab, cloud, path and commit, so the chatbot can filter (“only the Azure lab”) and ingest can delete the chunks of a removed file.

_posts/private/ and .env* files are never indexed.

The RAG pipeline is observable step by step. rag-api and rag-ingest are instrumented with OpenInference, and the collector sends their traces to Arize Phoenix as well as to Tempo. For any answer, Phoenix shows:

• whether the semantic cache hit
• which chunks Qdrant returned, with their scores and source paths
• the exact prompt the model saw
• how long each step took

Each answer in the chat has a “View trace” link. Phoenix also projects the stored embeddings onto a map, which shows the topics the knowledge base covers well and where it’s thin.

Phoenix and Grafana overlap a little, so each has a clear job:

• Grafana in mon is where the AI stack is operated. It holds every dashboard and alert for the GPU, vLLM’s queue and time to first token, the gateway, cache hit rate and ingest lag.
• Phoenix is where answer quality is investigated. Use it when a reply is wrong, or to compare a chunking, prompt or model change with evals over a golden question set.

Phoenix gets no alerts. Both receive the same OTel traces, and one trace ID links a slow request in Grafana to its readable version in Phoenix. Phoenix arrives with the RAG pipeline in phase 5.

rag-ingest clones the repos from GitHub with a fine-grained, read-only token. Only pushed work is indexed, so drafts and half-finished experiments stay out of the answers.

Identity and access architecture

Keycloak, running in the core cluster, is the one identity provider for everything. It doesn’t hold accounts of its own. It brokers my Google and Entra ID logins and adds groups, such as platform-admins and viewers, that map to roles in every tool.

There are still two gates for people, so one misconfiguration isn’t enough to expose anything:

1. Tailscale decides who reaches the network at all. Only my Tailscale login can reach the homelab bridge (172.30.0.0/24), and only on the ports the platform publishes: 80/443 for the UIs, 53 for DNS and 6441-6444 for the Kubernetes APIs. Nothing is reachable from the internet.
2. Every tool signs you in again through Keycloak OIDC: Backstage, Grafana, Argo CD, OpenBao and Phoenix. So does kubectl on every k3d cluster, through the k3s OIDC flags. Roles come from groups, so one group change applies everywhere.

The break-glass route doesn’t depend on Keycloak: if core is down, I can still SSH to the DGX over the tailnet (or from the home network) and fix it from there. If Tailscale itself is down, the platform keeps running; only access from outside the house waits.

ChaosCenter is the exception to the public hostnames. A tool that can delete pods across the fleet isn’t published through the tunnel at all. It’s reached only from the home LAN at chaos.core.home.cloudutsuk.com, and only platform-admins may run experiments.

TODO: Can ChaosCenter sign in through Keycloak? Litmus’s single sign-on goes through Dex, so the phase 2a spike checks Dex with Keycloak as its OIDC upstream. If it doesn’t work, ChaosCenter keeps local accounts, with the admin password in OpenBao and the LAN-only rule above.

Services authenticate to each other with short-lived credentials, never shared passwords:

Call	How
Backstage, sre-agent and RAG tools → RAG API, inventory, lab runner	OAuth2 client credentials from Keycloak: 5-minute JWTs with an audience and roles, checked against Keycloak’s signing keys
AI services → LiteLLM gateway	A JWT, or one virtual key per client with its own limits (depends on LiteLLM’s free edition)
Alertmanager → SRE agent, collectors → mon	OAuth2 client credentials, which both support natively
SRE agent and lab runner → a cluster’s Kubernetes API	15-minute ServiceAccount tokens from OpenBao’s Kubernetes secrets engine, read-only for the agent
Argo CD → every cluster	One ServiceAccount token per cluster, refreshed from OpenBao
Karmada agents → Karmada API server	Each agent’s own client certificate from pull-mode registration; Karmada holds no member-cluster credentials
Every cluster → OpenBao	Per-cluster Kubernetes auth (below)

Every cross-cluster call is HTTPS on a *.home.cloudutsuk.com certificate, and NetworkPolicies allow only the calls above. Service-mesh mTLS is a later option. Keycloak’s events and OpenBao’s audit log go to Loki.

Machine permissions stay least-privilege:

Identity	Permissions
Inventory	Azure Reader + Cost Management Reader; AWS ReadOnly + ce:Get*; GCP Viewer + BigQuery Data Viewer
Lab runner	Per-lab deploy and destroy credentials, only for labs with cloud resources
SRE agent	Read-only Kubernetes access on every cluster through short-lived tokens. Fixes go out as PRs to homelab-gitops after approval
CI runners	Log in to OpenBao with GitHub OIDC; push to Zot with the ci-pusher key (only from main); commit image digests to homelab-gitops with a short-lived GitHub App token
Clusters pulling images	Anonymous pulls of mirrored public images; a zot-pull secret from OpenBao (through External Secrets) for our own images, added to pods by a Kyverno policy

Secrets: OpenBao and External Secrets

No secret value lives in git. OpenBao, the open-source fork of Vault, runs in the core cluster and is the only place secrets are stored. Every cluster (core, platform, mon, ai and each lab) runs the External Secrets Operator. Apps keep reading ordinary Kubernetes Secrets, and ExternalSecret objects in homelab-gitops say which OpenBao path each one comes from.

• Per-cluster identity. OpenBao has one Kubernetes-auth mount per cluster, so no cluster holds a static token. Each cluster’s policy reads only its own paths (platform/*, labs/<slug>/*, …) plus shared/*. A lab cluster can’t read the portal’s OAuth secrets.
• Labs. The lab runner creates a lab’s mount and policy on deploy and removes them on destroy.
• Outside Kubernetes.
  • The lab runner uses an OpenBao agent with AppRole.
  • CI runners log in with GitHub’s own OIDC token, so nothing is stored as a GitHub secret.
• What’s stored:
  • the OAuth clients
  • read-only cloud credentials (rotated every 90 days)
  • per-lab deploy credentials
  • GitHub App and repo tokens
  • Slack tokens
  • LiteLLM keys
  • database and Grafana passwords
  • the Cloudflare DNS-01 token
  • backup keys
• Unseal and backup. After a reboot, OpenBao is unsealed from a root-only key file on the DGX. A nightly raft snapshot goes out with Velero’s backup of core, encrypted, and the unseal keys have an offline copy.

Hosting and deployment model

Everything runs on the DGX Spark, so hosting costs $0. The only cloud spend is the labs I choose to deploy. The repos are split by how often they change:

Repo	Holds
infra-local (new)	The core, mon, platform and ai k3d clusters, the homelab network and each node’s registry settings, through the shared module’s docker_network and k3d_config_cluster units, plus the Helm bootstrap of OpenBao, PowerDNS, External Secrets and Argo CD
homelab-gitops (new)	Argo CD app-of-apps, Helm values, image tags, ADRs
homelab-services (new)	Python services: inventory, lab-runner, rag-ingest, rag-api
self-service-portal	Backstage plus the new plugins, built into a custom image
sre-agent	The alert-driven agent

The three new repos were created on 2026-10-11 through infra-github, like every other repo: a new homelab group holds one resources.yaml per repo, and a Terragrunt plan showed only additions before the apply. Each repo is private (ADR 0019: self-hosted runners only serve private repos), squash-merge only, deletes merged branches, and carries the same custom properties as the other platform repos (portfolio platform-engineering, cloud on-premises) plus the topic cloudutsuk-platform. infra-local also gets a production environment, so its applies can be gated later.

# infra-github/environments/github/homelab/homelab-gitops/resources.yaml (abridged)
github_repository:
  - name: homelab-gitops
    description: "Cloudutsuk Platform - Argo CD desired state (app-of-apps, Helm values, dashboards, alert rules) and the platform ADR log"
    allow_auto_merge: false
    topics: ["cloudutsuk-platform", "homelab", "dgx-spark"]
github_repository_collaborators:
  - name: homelab-gitops
    collaborators:
      - teams:
          - team_name: "team-cloudutsuk"
            permission: admin

A commit flows:

1. A self-hosted runner on the DGX builds the image and pushes it to Zot in core.
2. The new tag is committed to homelab-gitops.
3. Argo CD syncs it.

Argo CD is a single hub instance. Lab clusters are registered when the lab runner creates them, and an ApplicationSet installs the OTel collector on every one. From phase 1a, if the Karmada trial passes, that fan-out moves to Karmada (below) and the lab runner also registers each new lab with it.

Multi-cluster management with Karmada

The platform starts with four long-lived clusters and adds one per lab, and most of what each cluster runs is the same: the monitoring pack, OpenCost, External Secrets, ExternalDNS, cert-manager and, from phase 2a, the chaos agent. Karmada (a CNCF project) manages that shared layer. It’s a Kubernetes-style control plane for many clusters: you apply ordinary manifests to its API server, and policies decide which member clusters get them.

How it fits with Argo CD. Argo CD stays the GitOps engine and keeps deploying each cluster’s own apps directly. For the shared add-ons, Argo CD syncs them once, to Karmada’s API server, as if it were one more cluster:

Karmada object	What it does here
PropagationPolicy	Places each add-on on clusters by label, for example every cluster with cloudutsuk.com/role in core, platform, mon, ai, lab
OverridePolicy	Per-cluster values: the OTel cluster attribute, each cluster’s OpenBao auth mount, the ExternalDNS domain (<cluster>.home.cloudutsuk.com)
Cluster	One per member, labelled with its role; the lab runner adds a lab’s on deploy and removes it on destroy
Status aggregation	Each add-on’s status comes back from every cluster, so Argo CD shows one health for “monitoring pack everywhere”

Placement. The control plane (API server, etcd, controller manager, scheduler, webhook) runs in platform next to Argo CD, installed by Argo CD. It isn’t needed at boot, so it doesn’t belong in core. Members join in pull mode: a karmada-agent in each cluster dials out to the Karmada API server and applies what’s placed on it, so Karmada holds no cluster credentials. That’s the same outbound pattern as the chaos agents.

What I’m deliberately not using. Karmada can split a Deployment’s replicas across clusters, fail over between them and autoscale across them. On one DGX those add nothing: every cluster shares the same machine, so failing over from one to another doesn’t survive the failures that matter (ADR 0015). They become worth turning on when a second machine joins.

# Sketch: the monitoring pack on every cluster, with each cluster's name as an override
apiVersion: policy.karmada.io/v1alpha1
kind: ClusterPropagationPolicy
metadata:
  name: monitoring-pack
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: DaemonSet
      name: otel-agent
      namespace: monitoring
  placement:
    clusterAffinity:
      labelSelector:
        matchExpressions:
          - key: cloudutsuk.com/role
            operator: In
            values: [core, platform, mon, ai, lab]

TODO: The Karmada trial (phase 1a) has to show four things before ADR 0037 is accepted: arm64 images for every component, Argo CD reporting health correctly through Karmada’s API server, a lab joining and leaving in pull mode within the lab runner’s deploy time, and the memory it really uses. If any fails, the ApplicationSets stay and Karmada is dropped.

Cost of the local clusters with OpenCost

The cloud bills tell the inventory what Azure, AWS and GCP cost, but the DGX sends no bill. Its cost is electricity plus the machine itself, and the question that matters is which cluster, lab or namespace is using it. OpenCost, the open-source CNCF project for Kubernetes cost allocation, answers that. It runs on every k3d cluster as part of the monitoring pack, placed like the other shared add-ons, so a new lab is priced from its first minute.

How it fits on this platform:

• No new metrics pipeline. OpenCost needs Prometheus-style CPU, memory and pod metrics, and the monitoring pack already ships them to Mimir in mon with their Prometheus names (ADR 0026). Each cluster’s OpenCost queries Mimir over PromQL, filtered by its own cluster label, instead of running its own Prometheus.
• The DGX’s own prices. There’s no cloud price list for a desk, so OpenCost runs with custom pricing: a cost per CPU core-hour, per GB of RAM per hour and per GPU-hour, derived from the DGX’s hourly cost. That hourly cost is the electricity tariff times its measured power draw, plus the machine’s price spread over its expected life.
• One source of truth for cost. The inventory service reads OpenCost’s allocation API once a day, per cluster and namespace, and writes the result into COST_DAILY under the dgx provider. So the portal shows the DGX next to Azure, AWS and GCP, and a lab’s local and cloud cost roll up together. OpenCost itself keeps no history: Mimir holds about 15 days, and the daily rows in platform-db keep the rest. Its own cloud-cost feature stays off, because the inventory already reads the cloud bills.
• Dashboards. OpenCost exports its own metrics (cost per node, allocation per container), which the monitoring pack scrapes into Mimir, so Grafana gets a Cost dashboard per cluster and lab.
• UI off. Each cluster runs only the OpenCost API and exporter. People see cost in the portal and Grafana, not in a dozen separate UIs.

The one-host catch is the GPU. On the GB10, GPU memory is system memory and isn’t charged to any container, and models share the GPU by time-slicing. OpenCost can only allocate the GPU by nvidia.com/gpu requests, so the GPU-hour goes to whichever vLLM pods requested it, not by real use. That’s good enough to show what the ai cluster costs; it isn’t a per-request cost.

TODO: The custom prices need two numbers that aren’t recorded yet: the electricity tariff per kWh (already an open TODO in Product Planning) and the DGX’s purchase price and the years to spread it over. Until then OpenCost shows usage shares with placeholder prices.

TODO: Mimir is multi-tenant. Confirm in phase 4 how each OpenCost authenticates to Mimir’s query API and that its cluster-label setting picks out only its own cluster’s series.

Internal DNS and bootstrap order

Containers on the homelab network can already reach each other by container name, but nothing gives services stable, readable names, or lets the DGX resolve them. PowerDNS in core is the only server for home.cloudutsuk.com, an internal-only zone that’s never published in Cloudflare.

• Records. ExternalDNS in every cluster writes its Ingress and Service names, such as grafana.mon.home.cloudutsuk.com or otel.lab-otel.home.cloudutsuk.com. A lab’s records disappear with the lab.
• Resolution.
  • Every cluster’s CoreDNS forwards the zone to PowerDNS: in core to PowerDNS’s own fixed service address, in the other clusters to a separate port that core publishes on the gateway. Not the gateway’s port 53: inside every k3d node that address belongs to Docker’s own resolver, which I only found out by testing from a pod (lesson L009).
  • The DGX uses split DNS in systemd-resolved, the same trick as the Azure private endpoint lab.
• TLS. Because the zone sits under a domain I own, cert-manager gets real Let’s Encrypt certificates through a Cloudflare DNS-01 challenge. Only the _acme-challenge TXT records ever reach Cloudflare.

The boot order follows from what depends on what:

1. The network.
2. core, then OpenBao is unsealed.
3. mon, platform and ai, each with External Secrets.
4. Argo CD, whose own repo credentials come from External Secrets.

After that, Argo CD’s ApplicationSets keep External Secrets, ExternalDNS, cert-manager and the OTel agent on every cluster, including each new lab cluster.

Commit to deploy

Every change takes the same path, and every security and test gate sits on it:

• Every PR and main run: Ruff and pytest, then Gitleaks, Semgrep and Bandit, Trivy (dependencies, IaC and the image), Checkov, Conftest and Hadolint. Any failure stops the run before an image exists.
• On main: the scanned image is pushed to Zot, signed and SBOM-attested with a key in OpenBao, and its digest is committed to homelab-gitops.
• Every deploy: Kyverno’s signature and pod policies at admission, and a PostSync smoke test.
• Every night (Phase 3a): Playwright, OWASP ZAP and a Locust load run against what’s deployed.

The animated diagram, the full testing and security matrix with tools, triggers and thresholds, and the measured timings are in CI/CD and Release Management.

GitHub, the registry and the local clusters

GitHub hosts the code and runs the workflows, but every job executes on the DGX and nothing connects in:

• Runners. Actions Runner Controller in the platform cluster registers arm64 runners with the GitHub org through a GitHub App. Each runner pod polls GitHub over outbound HTTPS, takes one job and is deleted.
• CI stops at git. No cluster credentials ever go to GitHub. A main build ends by committing the new image digest to homelab-gitops.
• CD pulls. Argo CD polls that repo every 3 minutes, also outbound, and applies the change to the right k3d cluster over the local Docker network.

If GitHub is down, nothing new ships, but everything already running keeps running.

Images live in Zot, a small OCI registry in the core cluster at registry.core.home.cloudutsuk.com, on HTTPS with a Let’s Encrypt certificate. I chose it over Harbor because Harbor’s official images don’t support arm64.

Who	How they authenticate
CI pushing an image	The runner logs in to OpenBao with its GitHub OIDC token (only main builds of our repos are allowed), reads the ci-pusher key, and BuildKit and Cosign push with it. PR builds can’t push
Clusters pulling public images	No credentials. Every node’s containerd is configured to fetch docker.io, ghcr.io, quay.io and registry.k8s.io through Zot’s mirror, which caches them
Clusters pulling our images	A zot-pull secret that External Secrets fetches from OpenBao into each namespace; a Kyverno policy attaches it to every pod
Kyverno checking signatures	The same zot-pull secret; signatures and SBOMs sit next to each image (OCI referrers)
People	Zot’s web UI through Keycloak

Deployments reference images by digest, so what runs is exactly what was scanned and signed.

Network layout

The same platform, seen as networks: what talks to what, on which port, and where TLS starts and ends. Tick Simulate a cloud lab to see the WireGuard path a cloud lab adds.

1. Nothing listens at home: The home router has no port forwards and nothing is published on the internet. tailscaled runs on the DGX host itself, outside every cluster.
2. Out through NAT: tailscaled keeps an outbound WireGuard connection (UDP 41641) through the router's NAT, falling back to Tailscale's relays over 443 when a direct path isn't possible.
3. You join the tailnet: Your phone or laptop joins the tailnet. The policy lets only your login reach 172.30.0.0/24, and split DNS sends home.cloudutsuk.com to PowerDNS. Cloudflare keeps only public DNS.
4. Into the bridge: The DGX routes you onto the homelab bridge: portal.platform.home.cloudutsuk.com is 172.30.0.2, where platform's Traefik serves 443 with a Let's Encrypt certificate.
5. To the pod: Traefik routes through the Backstage Service (10.21.0.0/16) to its pod (10.20.0.0/16) on 7007. NetworkPolicies only allow ingress from Traefik.
6. Internal names: Pods look up *.home.cloudutsuk.com through their cluster's CoreDNS, like any other name.
7. One DNS for all clusters: CoreDNS forwards home.cloudutsuk.com to PowerDNS in core, never via the gateway's port 53 (k3d keeps that for Docker's DNS). The DGX uses systemd-resolved split DNS.
8. Calls between clusters: Clusters reach each other only through their k3d load balancers on the homelab network: HTTPS 443 with a JWT to ai, and HTTPS 443 to OpenBao and Keycloak in core.
9. Cluster APIs: Argo CD and the SRE agent reach each cluster's Kubernetes API on 6443 with short-lived tokens. APIs listen only on the bridge gateway: reachable from the tailnet with a Keycloak login, never from the internet.
10. Telemetry and images: Collectors send OTLP gRPC with TLS on 4317 to mon. Nodes pull from Zot in core over HTTPS (published on 172.30.0.1:443, a k3d host alias); public images come through its mirror.
11. Out to the internet: cert-manager gets Let's Encrypt certificates with DNS-01 (a TXT record in Cloudflare). Cloud APIs, GitHub and model downloads leave through NAT on 443.
12. A cloud lab joins (cloudlab): A cloud lab such as the Azure private endpoint lab adds a WireGuard tunnel (UDP 51820) from the DGX to a VM in Azure, and split DNS sends only Azure's names there.

The address plan is fixed up front (ADR 0030). The point is that no two clusters overlap, so a service mesh or direct pod routing can be added later without renumbering anything:

Network	Range	Notes
Home LAN	a /24 in 192.168.0.0/16 (Wi-Fi)	No port forwards. The DGX only accepts SSH from the LAN
homelab Docker bridge	172.30.0.0/24, gateway 172.30.0.1	Every k3d node and load balancer. The gateway address publishes core’s PowerDNS (:53) and Zot registry (:443)
core pods / services	10.10.0.0/16 / 10.11.0.0/16
platform pods / services	10.20.0.0/16 / 10.21.0.0/16
mon pods / services	10.30.0.0/16 / 10.31.0.0/16
ai pods / services	10.40.0.0/16 / 10.41.0.0/16
Lab clusters	10.100.0.0/16 upwards, two /16s per lab	Assigned by the lab runner from the lab card

k3s’s defaults (10.42/16 pods, 10.43/16 services) would be the same in every cluster. They’re overridden with --cluster-cidr and --service-cidr when each cluster is created.

Path	Protocol and port	TLS
My device → DGX	WireGuard through Tailscale (direct UDP, or Tailscale’s relays when NAT blocks it), outbound only on both ends	WireGuard encryption; the tailnet policy allows only my login
DGX → a cluster	HTTPS 443 to the cluster’s own bridge address (core .1, platform .2, mon .3, ai .4)	Let’s Encrypt certificate for *.<cluster>.home.cloudutsuk.com, the same one the DGX sees
Cluster → cluster	HTTPS 443 through each cluster’s k3d load balancer and Traefik	*.<cluster>.home.cloudutsuk.com, plus a JWT (ADR 0029)
→ OpenBao, Keycloak	HTTPS 443 through core’s Traefik (bao.core.home.cloudutsuk.com)	Let’s Encrypt certificates in core
→ Kubernetes API	HTTPS 6443 on the load balancer, never published to the LAN	k3s’s own CA; short-lived tokens
Collectors → mon	OTLP gRPC 4317	TLS, plus OAuth2 client credentials
Pods → internal DNS	DNS 53 to CoreDNS, forwarded to PowerDNS (its service address in core, a separate gateway port from other clusters)	Plain DNS, on the bridge only
OpenCost → Mimir	PromQL over HTTPS 443 to mimir.mon.home.cloudutsuk.com, from every cluster	Let’s Encrypt certificate, plus OAuth2 client credentials like the collectors
Inventory → OpenCost	HTTPS 443 to each cluster’s OpenCost API, once a day	*.<cluster>.home.cloudutsuk.com, plus a JWT (ADR 0029)
Karmada agents → Karmada API server	HTTPS 5443 (Karmada’s default) on the platform load balancer, outbound from each member	Karmada’s own CA; per-agent client certificates
Chaos agents → ChaosCenter	HTTPS 443 to chaos.core.home.cloudutsuk.com, outbound from each target cluster	Let’s Encrypt certificate; each agent has its own infrastructure ID and access key from OpenBao
Clusters → registry	HTTPS 443 to registry.core.home.cloudutsuk.com, which resolves to the gateway address through a k3d host alias on every node	Let’s Encrypt certificate. Anonymous pulls of mirrored public images; our images need the zot-pull secret
cert-manager → Let’s Encrypt	HTTPS 443, DNS-01 TXT records through the Cloudflare API	ACME
Egress (cloud APIs, GitHub, model downloads)	HTTPS 443 through the router’s NAT	Provider certificates
Cloud lab (optional)	WireGuard, UDP 51820, from the DGX to the lab’s VM	WireGuard

Inside every cluster, NetworkPolicies default to deny. They allow ingress from Traefik, the calls in the identity table above, DNS, and egress to the targets in this table. Inside a cluster, Traefik terminates TLS and talks plain HTTP to the pod. Pod-to-pod mTLS waits for the service mesh option.

Cilium: considered, deferred

Cilium (eBPF networking, policy and observability) fits where this platform is heading, but not yet. I’m recording it so the reasoning isn’t lost.

Why it fits:

• The address plan already leaves room for it: unique pod and service CIDRs per cluster are exactly what Cilium’s ClusterMesh needs.
• It could fill the “pod-to-pod mTLS waits for the service mesh” gap with mutual authentication and no sidecars.
• Hubble shows every flow, including the ones a policy dropped, and its metrics would go to Mimir like everything else. That makes default-deny NetworkPolicies far easier to debug.
• Policies by HTTP path and DNS name, for example the inventory limited to its three cloud APIs by name instead of “HTTPS anywhere”.
• arm64 is supported.

Why not now:

• k3d makes it fiddly. Each cluster has to be created without flannel and k3s’s policy controller, and each node container needs the BPF filesystem mounted and cgroup settings adjusted. That’s fine for four long-lived clusters, but it adds steps and failure points to every lab’s make up.
• One shared kernel. Every k3d cluster runs on the DGX’s one Linux kernel. Some Cilium features hook in at the host level (its kube-proxy replacement and socket load balancing), so one cluster’s Cilium might affect another’s traffic. A risk to test, not a known blocker.
• Memory. An estimated 0.3–0.5 GB per cluster (agent and operator, more with Hubble), about 2–3 GB with labs, on a budget that’s already tight.
• ClusterMesh buys little on one machine. Cross-cluster calls already go over HTTPS through Traefik with a JWT (ADR 0029), and every cluster shares the same host.
• Today’s need is met. k3s’s built-in controller already enforces default-deny NetworkPolicies.

The path, if and when it’s worth it:

Step	What	Trigger
1. A lab	“Cilium on k3d”: Hubble, an HTTP-path or DNS-name policy, two clusters joined by ClusterMesh. Answers the k3d and shared-kernel questions cheaply, and it’s a blog lab in its own right	Any time; no platform change
2. Long-lived clusters	core, platform, mon and ai move to Cilium in one change, with its own ADR. Labs keep k3s defaults unless the lab is about Cilium	The lab shows no cross-cluster interference and the memory fits
3. ClusterMesh and mutual authentication	Direct pod routing and mTLS between clusters	A second machine joins, the same point where Karmada’s failover starts to matter

Checked 2026-10-11. The home LAN is a /24 inside 192.168.0.0/16 and the DGX joins it over Wi-Fi. Docker’s existing networks use 172.17 and 172.18, and no host route touches 172.30.x or 10.x, so the plan stands. infra-local/host/check-prereqs.sh repeats this check before any cluster is created.

Scalability and performance strategy

This platform can’t scale out. It has one GB10 GPU and about 121 GB of unified memory, shared between the CPU and GPU. Moving vLLM to another k3d cluster doesn’t add capacity. Every cluster on this box draws on the same chip and the same memory. So the budget is the main design constraint:

Component	Range	Planned (mid-range)
ai cluster: vLLM chat model (~20–30B MoE, FP8/MXFP4)	~35–40 GB	~30 GB to start
ai cluster: vLLM embeddings + model gateway	~3–5 GB	~4 GB
ai cluster: KEDA operator and metrics server (on-demand models use 0 GB while idle)	~0.1–0.3 GB	~0.2 GB
ai cluster: RAG API, rag-ingest, Qdrant, Redis, Phoenix	~4–6 GB	~5 GB
core cluster (Keycloak, OpenBao, PowerDNS, Zot, ESO, ExternalDNS, LitmusChaos ChaosCenter + MongoDB)	~4–5.5 GB	~4.5 GB
Litmus chaos agents (one per target cluster, from phase 2a)	~0.2–0.4 GB each	~1 GB
Karmada control plane in platform (API server, etcd, controllers), plus one agent per cluster (from phase 1a)	~1.5–2.5 GB, agents ~0.1 GB each	~2.5 GB
OpenCost, one per k3d cluster (API and exporter, no UI; from phase 4)	~0.1–0.2 GB each	~1 GB
mon cluster	~6–8 GB	~7 GB
platform cluster (portal, Postgres, inventory, Argo CD, CI, SRE agent)	~7–9 GB	~8 GB
CI runners + BuildKit (in bursts)	~4–8 GB	~5 GB
Lab clusters	~3–6 GB each, 2–3 of them	~10 GB in total
k3s servers (4 long-lived clusters)	~0.5–1 GB each	~3 GB
OS, Docker, desktop use (VS Code, browser, the local blog server)	~15 GB	~25–35 GB

Measured on 2026-10-11: with the DGX in use as a desktop and no platform running yet, about 81 GB of the 121 GB was free. The top of every range needs about 97 GB, so the plan works to the mid-range column instead: about 81 GB for the platform (75 GB before LitmusChaos, Karmada and OpenCost were added), which leaves 25–35 GB for the desktop and less headroom than before. The Litmus, Karmada and OpenCost figures are estimates until they’re measured on the DGX; if the total gets too tight, Karmada is the first to go, because Argo CD ApplicationSets already do its job. The chat model starts with a smaller memory slice and only grows if measurements in Phase 5 leave room. The lab runner’s admission control checks the memory actually free at deploy time (ADR 0017), not a fixed number, so a busy desktop day means fewer labs at once rather than an out-of-memory model.

What the GPU test changed. On the GB10, GPU memory is ordinary system memory, and it isn’t charged to any container. With two small vLLM pods running, the host’s used memory rose by about 29 GB, while Docker counted 7.5 GB for the whole cluster and Kubernetes about 3 GiB per vLLM pod. Three rules follow:

• A model’s slice is set by --gpu-memory-utilization, as a fraction of the whole 121 GB: 0.1 took about 12 GB, so the chat model’s ~30 GB start is about 0.25. Pod memory limits don’t cap it.
• There’s no MIG, so models share the GPU by time-slicing, with no memory isolation between them. A wrong utilization setting can starve the other model or the host.
• Admission control reads the host’s free memory, never the sum of container metrics, and the Fleet dashboard shows host memory next to per-cluster memory so the ai cluster doesn’t look smaller than it is.

The whole AI stack lives in its own ai k3d cluster (ADR 0022): the RAG API, rag-ingest, Qdrant, the Redis semantic cache, Phoenix, the model gateway and the vLLM instances. They’re upgraded and tuned together. The SRE agent stays in platform on purpose: when the models are down, it still has to post its raw evidence to Slack.

On one host that buys isolation, not capacity:

• Only this cluster needs the custom GPU-enabled k3s image.
• A failure anywhere in the AI stack (vLLM, the GPU driver, Qdrant) can’t take the portal, Postgres or GitOps down with it.
• It’s the obvious place for a second machine to join later.

Nothing calls vLLM directly. The RAG API, rag-ingest and the SRE agent all call one OpenAI-compatible model gateway (LiteLLM proxy) by model name. The gateway routes each request, queues bursts, applies per-client limits and records token use. That gateway is what makes the scaling steps below possible:

Need	What I do	Why not just add replicas
More requests to the same model	Tune the one instance: continuous batching, FP8/FP4 weights, prefix caching, a cap on concurrent sequences, plus the semantic cache in front	A second replica on the same GPU copies the weights and halves the KV cache each copy gets, so throughput goes down
More kinds of model (embeddings, a reranker, a coding model)	Each vLLM instance gets a fixed slice of memory (--gpu-memory-utilization) from the budget. Small models stay loaded; rarely used large ones are loaded on demand and unloaded when idle	The GB10 can’t be split into hardware partitions (no MIG, confirmed in the GPU test), so pods share it by time-slicing and memory is the only lever
More than one box can serve	Add a backend behind the gateway: a second DGX Spark, another GPU machine joining as a k3s node, or a hosted model for overflow	The callers don’t change, only the gateway’s routing table

Autoscaling in the usual Kubernetes sense doesn’t apply with one GPU. CPU-based HPA would only start copies that compete for the same chip. Instead, mon watches vLLM’s waiting-request count and time to first token. When those stay high, that’s the signal to add a backend or turn on overflow. Real autoscaling, such as KEDA on queue depth, becomes possible once there’s a second GPU node.

On-demand models with KEDA

There’s one kind of scaling that does pay off on one box: scaling to zero. The table above says rarely used models are “loaded on demand and unloaded when idle”, because memory is the only lever. KEDA (Kubernetes event-driven autoscaling, a CNCF graduated project) does exactly that, and it runs only in the ai cluster:

• Which models. Only the on-demand ones, such as a reranker or a coding model if they’re added. The chat model and the embedding model keep minReplicaCount: 1 and are never scaled to zero, because the portal chat and ingest need them all the time.
• The trigger. KEDA’s Prometheus scaler queries Mimir for the model gateway’s request metrics for that model. Demand scales its vLLM Deployment from 0 to 1; a long quiet spell (the cooldown) scales it back to 0 and hands its memory slice back to the desktop and the labs.
• Cold starts. Loading a model takes time, so the first request after a quiet spell waits. The gateway queues and retries that request, and its callers already expect a slow first answer from an on-demand model. That trade is the whole point: memory when nobody’s using it, a wait when someone is.
• Admission. Before the model scales up, its memory slice must fit what’s free on the host, the same rule the lab runner follows (ADR 0017). If it doesn’t fit, it stays at 0 and the gateway answers with a clear “model unavailable” instead of starving the chat model.

# Sketch: an on-demand model that sleeps when idle
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-reranker
  namespace: serving
spec:
  scaleTargetRef:
    name: vllm-reranker
  minReplicaCount: 0
  maxReplicaCount: 1                 # one GPU: never more than one copy
  cooldownPeriod: 1800               # 30 quiet minutes before it unloads
  triggers:
    - type: prometheus
      metadata:
        serverAddress: https://mimir.mon.home.cloudutsuk.com/prometheus
        query: sum(rate(litellm_requests_total{model="reranker"}[5m]))   # metric name checked in phase 5
        threshold: "0.01"
      authenticationRef:
        name: mimir-reader           # OAuth2 token from OpenBao through External Secrets

KEDA isn’t used for the services: CPU-based scaling on one host only starts copies that compete for the same chip. Its other uses wait for a second GPU node: vLLM replicas on queue depth, and the overflow backend described above.

TODO: Confirm in phase 5 how the gateway behaves while a model scales from 0 (queue, retry or fail fast), the gateway’s real metric names, and how long the on-demand models take to load on the GB10.

Elsewhere:

• The lab runner refuses or queues a deploy that would push free memory below the headroom.
• rag-api rate-limits per user.
• The services are stateless, so they move to another node as easily as the models do.

TODO: Set performance targets after the first load test. A starting proposal is time to first token under 3 s at p95.

Resilience and disaster recovery

One host can’t be highly available, so I’m optimising for recovery and for knowing when it’s down:

• Dead-man’s switch. A systemd timer on the DGX pings healthchecks.io (free plan) every five minutes while OpenBao in core answers healthy, and pings the check’s failure URL with the reason when it doesn’t. If the DGX, power or the ISP goes down, the pings stop and I get an email, even though monitoring died with the host. From Phase 2, Alertmanager’s always-firing Watchdog pings it too.
• Isolation. mon is a separate cluster that doesn’t depend on Argo CD at runtime. Resource limits and PriorityClasses stop vLLM or a runaway lab starving it.
• Restart. After a reboot, a systemd unit starts Docker, then core, then unseals OpenBao, then starts mon, platform and ai. External Secrets retries until OpenBao answers. Lab clusters stay down after a reboot.
• Remote access. Tailscale runs on the DGX host, not in a cluster, so remote access survives any cluster failing; the boot unit checks it’s up after a reboot.
• Degradation:
  • Inventory shows the last known data with a “stale since” badge.
  • Chat falls back to search results when the LLM is down.
  • The agent posts its raw evidence when it can’t reach the model.
• Backups. Only Postgres and the secrets, with Velero (ADR 0039): before each nightly backup, a hook in a small dump pod takes a pg_dump and an OpenBao snapshot; Velero uploads them encrypted to an Azure Blob container, with each cluster’s objects (minus Secrets). R2 was the first plan, but it asks for a payment method. Everything else rebuilds from git. Targets are RTO 1 hour, RPO 24 hours, tested once a quarter by restoring into an empty cluster. A web UI at velero.platform.home.cloudutsuk.com shows what was backed up, when, and how big it was.
• Chaos experiments. The claims above (degradation, retries, recovery) are only true once they’ve been broken on purpose. LitmusChaos does that, below.

Chaos engineering with LitmusChaos

LitmusChaos, the open-source CNCF project (Apache 2.0), is the platform’s chaos tool. It arrives in phase 2a, once mon can show what a fault does. Its control plane, ChaosCenter (web UI, GraphQL server, auth server and a small MongoDB), runs in the core cluster:

• It survives the thing it breaks. core boots first and doesn’t depend on platform, mon or ai, so an experiment that takes down a target never takes the chaos tool with it. That’s the same reason identity and secrets live there (ADR 0025).
• Hub and spoke, like Argo CD. Each target cluster gets a chaos agent: a subscriber, the chaos operator and the chaos exporter, installed by an Argo CD ApplicationSet. The agent dials out to ChaosCenter over HTTPS 443, so nothing connects into a lab.
• Experiments as code. Each experiment is a reviewed file in homelab-gitops, and ChaosCenter’s GitOps sync keeps its library equal to git. A one-off run from the UI is fine for exploring, but anything scheduled goes through a PR.

Running chaos on one shared machine needs stricter rules than in a real cloud, because every k3d node is a container on the same DGX, drawing on the same memory as the GPU:

Guardrail	Rule
Targets	Lab clusters first. platform, mon and ai only from phase 2a’s exit, one cluster at a time. core only in a planned game day, with the break-glass login ready
Fault types	Pod-level only: pod-delete, container-kill, pod network latency and loss, pod CPU and memory hog inside the pod’s own limits. No node-level stress or node drain: on k3d a “node” is the whole DGX
Blast radius	One namespace per experiment; Litmus’s annotation check, so only workloads labelled as chaos targets can be hit
Who	Only platform-admins; ChaosCenter is reachable only from the LAN and the tailnet
Memory	A running experiment counts against lab admission control (ADR 0017), like a temporary lab
Abort	Stop the experiment in ChaosCenter, or set the ChaosEngine’s engineState: stop

Every experiment carries probes (an HTTP check, a Prometheus query, a Kubernetes check), and ChaosCenter turns their results into a pass or fail and a resilience score. The chaos exporter’s metrics go through the monitoring pack to mon, so a Grafana dashboard shows each fault next to its effect. The faults also feed phase 6: a run that trips an alert is exactly the “injected lab fault” that milestone M6 grades the SRE agent on.

A sketch of a first experiment, killing the portal’s pod while a probe checks the portal still answers (Backstage’s health endpoint on its default port, 7007):

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: backstage-pod-delete
  namespace: backstage          # sketch: set to the portal's namespace
spec:
  engineState: active
  annotationCheck: "true"
  appinfo:
    appns: backstage
    applabel: app=backstage
    appkind: deployment
  chaosServiceAccount: pod-delete-sa
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "20"
        probe:
          - name: portal-still-answers
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: http://backstage.backstage.svc:7007/healthcheck
              method:
                get:
                  criteria: ==
                  responseCode: "200"
            runProperties:
              probeTimeout: 5s
              interval: 5s
              retry: 1

The first experiments, each matched to a claim this post makes:

Experiment	Target	Claim it tests
pod-delete on Backstage	platform	The portal recovers on its own; the PostSync smoke test passes again
pod network loss to vLLM	ai	Chat falls back to search results when the model is down
pod-delete on the inventory	platform	The portal shows the last data with a “stale since” badge
pod network latency to OpenBao	core (game day)	External Secrets retries until OpenBao answers
pod-delete on a lab’s app	lab	The alert fires, and the SRE agent’s RCA cites the right lab doc (M6)

TODO: Check the Litmus images and its MongoDB on arm64 in the phase 2a spike, like every other chart (see the arm64 risk in Product Planning). If its MongoDB chart has no arm64 build, pick a multi-arch MongoDB image and pin it in homelab-gitops.

TODO: k3s uses containerd at a non-default socket. Litmus’s network and container faults need the runtime set to containerd and the socket path set to k3s’s (/run/k3s/containerd/containerd.sock); confirm both on a k3d node in the spike.

Architectural decision records

Decisions are recorded as MADR files in homelab-gitops/docs/adr/, in the same PR as the change that implements them. A changed decision gets a new ADR that supersedes the old one, so the history stays intact. The full text of every ADR (context, decision, the options that lost, consequences) is in the Architecture Decision Records post, generated from the repo. One is still proposed: 0020 waits for the first load-test baseline (the SLO targets). 0010 was accepted after the GPU test on 2026-10-11.

LitmusChaos in core is recorded as ADR 0036 (proposed until the phase 2a spike). It replaces the “Chaos Mesh in a lab cluster later” line in ADR 0031. It joins the log below once it’s merged in homelab-gitops and synced. Karmada for multi-cluster placement is ADR 0037, also proposed until its phase 1a trial. It adds to ADR 0007 rather than replacing it: Argo CD stays the single GitOps hub. OpenCost on every k3d cluster for the DGX’s cost is ADR 0038, proposed until phase 4. KEDA in the ai cluster for on-demand models is ADR 0039, proposed until phase 5.

ADR	Decision	Status
0001	Record architecture decisions as MADR in homelab-gitops	Accepted
0002	Self-host the platform on the DGX Spark; GitHub Pages stays static	Accepted
0003	Public access via Cloudflare Tunnel and Access; move DNS to Cloudflare	Accepted
0004	Extend the Backstage self-service portal, built as a custom image	Accepted
0005	Repo layout: infra-local, homelab-gitops and homelab-services, created through infra-github	Accepted
0006	Long-lived k3d clusters as code with the shared module (k3d config files); lab clusters stay Makefile-owned	Accepted
0007	A single central Argo CD (hub and spoke) manages every cluster	Accepted
0008	CI on GitHub Actions with ARC runners on the DGX; CD by Argo CD pulling homelab-gitops	Accepted
0009	A central mon cluster with Grafana LGTM and Alertmanager for every cluster	Accepted
0010	Local LLM serving with vLLM on the DGX; model choice and memory budget	Accepted
0011	Qdrant for vectors and Redis (RedisVL) for the semantic cache	Accepted
0012	Six-hour schedules with manual sync for inventory and RAG ingest	Accepted
0013	Cost attribution through the existing common_tags plus an optional lab tag	Accepted
0014	Extend sre-agent for alert-triggered troubleshooting; human approval for every write	Accepted
0015	Single host by design: optimise for recovery, not high availability	Accepted
0016	Back up only Postgres and secrets, encrypted to the R2 free tier	Accepted
0017	Lab runner admission control (memory and ports) and a TTL for cloud labs	Accepted
0018	Graceful degradation contract	Accepted
0019	Self-hosted runners only for private repos; ephemeral runner pods	Accepted
0020	Platform SLOs and burn-rate alerting	Proposed
0021	Arize Phoenix for RAG and LLM trace visualisation, fed by the same OTel pipeline as Tempo	Accepted
0022	The whole AI stack in its own ai cluster behind a LiteLLM gateway	Accepted
0023	OpenBao as the local secret manager; External Secrets Operator in every cluster	Accepted
0024	Internal DNS with PowerDNS for home.cloudutsuk.com and Let's Encrypt via DNS-01	Accepted
0025	A dedicated core k3d cluster for shared services, booted first, on a fixed homelab subnet	Accepted
0026	Monitoring collection with the OTel Operator, keeping Prometheus metric names	Accepted
0027	Dashboards and alert rules as code, vendored and pinned in homelab-gitops	Accepted
0028	Keycloak in core as the single OIDC issuer, brokering Google and Entra ID	Accepted
0029	Service-to-service authentication	Accepted
0030	Network plan: fixed homelab bridge, unique CIDRs per cluster, no inbound ports	Accepted
0031	Testing strategy	Accepted
0032	Security scanning in the pipeline	Accepted
0033	Supply chain: Cosign signing with an OpenBao transit key and Kyverno admission	Accepted
0034	Zot in core as the only container registry	Accepted
0035	Data ownership: one writing service per entity	Accepted
0036	Bootstrap charts through the shared module's Helm layer; Argo CD for the rest	Accepted
0037	Private remote access through Tailscale instead of Cloudflare Tunnel and Access	Accepted
0038	Argo CD: installed by Terraform, everything else through an app-of-apps in homelab-gitops	Accepted
0039	Backups with Velero: application dumps through file-system backup to Azure Blob	Accepted
0040	A second, repo-scoped runner scale set for infra-local's IaC jobs	Accepted
0041	Monitoring stack on mon: monolithic Mimir, Loki and Tempo, one OTLP gateway, rules from git	Accepted
0042	Monitoring pack: OTel Operator collectors per cluster, Prometheus job names for the mixins, a token per cluster	Accepted
0043	CI build path: shared rootless BuildKit, OpenBao roles bound to the reusable workflow, digest commits by a GitHub App	Accepted
0044	Own runner image with the CI tools baked in, rolled out by hand	Accepted
0045	Lab UIs by name through one labs gateway on the host	Accepted
0046	First chat model: gpt-oss-20b (Apache 2.0) in a 0.17 slice of the GB10	Superseded by 0049
0047	Argo CD application groups: one AppProject per area, mirrored as the label homelab/group	Accepted
0048	Hermes Agent on platform: non-root without s6, the local model at 64k, dashboard for platform-admins only	Accepted
0049	Chat model: Qwen3.5-35B-A3B (FP8, Apache 2.0) replaces gpt-oss-20b	Accepted

Checklist

• All three animated diagrams approved before any infrastructure work
• GPU-in-k3d test done and ADR 0010 accepted: vLLM in k3d; no MIG, so time-slicing (2026-10-11)
• Memory budget confirmed on the DGX with the chosen model loaded
• ADRs 0001–0035 written in homelab-gitops and synced to the blog (2026-10-11)
• The three new repos created through infra-github (2026-10-11)
• Internal names resolve on the DGX and from pods, and bao.core.home.cloudutsuk.com serves a Let’s Encrypt certificate (2026-10-11)
• Keycloak brokering Entra ID, and kubectl signing in through it (mon, 2026-10-11); Google deferred
• My phone on mobile data reaches the internal UIs over Tailscale (2026-10-11); the laptop kubectl test and the “another Tailscale user is denied” test are deferred
• A service call without a valid token is rejected
• Backup and restore of Postgres tested once
• ADR 0036 (LitmusChaos in core) merged in homelab-gitops and synced to the blog
• Litmus spike done: arm64 images, k3s containerd socket, Keycloak sign-in through Dex, measured memory
• The first chaos experiments pass on a lab, then on platform, with results in Grafana
• ADR 0037 (Karmada) merged in homelab-gitops and synced to the blog
• ADR 0039 (KEDA in ai) merged in homelab-gitops and synced; an on-demand model scales 0 → 1 → 0 with its memory returned
• ADR 0038 (OpenCost on every k3d cluster) merged in homelab-gitops and synced to the blog
• OpenCost running on every cluster with custom DGX prices, and the DGX’s daily cost in the portal per cluster and lab
• Karmada trial done: arm64 images, Argo CD health through Karmada, a lab joining and leaving in pull mode, measured memory

• This post updated with what was actually built at the end of each phase