Post

Cloudutsuk Platform: Architecture Decision Records

Every architecture decision behind the Cloudutsuk Platform on a DGX Spark: context, the options that lost, and the consequences of each.

Posted
Cloudutsuk Platform: Architecture Decision Records - Projects hero image
Cloudutsuk Platform: Architecture Decision Records - Projects hero image
By Sakharam Shinde
111 min read
Cloudutsuk Platform: Architecture Decision Records

Every significant decision in the Cloudutsuk Platform is written down as an architecture decision record (ADR): what forced the decision, what was decided, which options lost and why, and what it costs. Most were made before any code existed, which is exactly when they’re easiest to forget. This post is the full log, generated from the platform’s GitOps repo (homelab-gitops/docs/adr), so it always matches what the platform is built to.

How to read this log

  • Proposed decisions are still waiting on evidence, such as a spike or a baseline; accepted ones are what the platform is built to. A changed decision never edits an accepted ADR: a new ADR replaces it and the old one is marked superseded, with a link forward, so the history stays.
  • Each ADR names the repos it affects. The summary table is in the Technical Architecture post.

ADR-0001: Record architecture decisions as MADR in homelab-gitops

Accepted 2026-10-10 · homelab-gitops, sakharams.github.io

Context

The platform spans at least eight repos (three new, five existing) and most decisions were made during planning, before any code. Without a written record, the reasons behind choices such as “why Zot and not Harbor” are lost, and the public playbook on cloudutsuk.com can drift from what is actually built.

Decision

  • Every significant decision is an ADR in homelab-gitops/docs/adr/NNNN-kebab-title.md, in MADR form with front matter (id, title, status, date, deciders, related_repos) and the sections Context, Decision, Considered options and Consequences.
  • An ADR is added in the same PR as the change that implements it. A changed decision gets a new ADR and the old one becomes superseded by NNNN; accepted ADRs are never rewritten.
  • Decisions about other repos are recorded here too, so there is one log.
  • scripts/check_adrs.py validates every ADR and the index in CI.
  • The blog shows the full text, generated by sync-adrs.py in the blog repo, which scrubs private data and fails --check when the blog copy is behind.

Considered options

  1. ADRs in each repo: decisions that span repos have no single home, and the blog sync would read five places.
  2. A wiki or Notion page: not reviewed in PRs and not versioned with the change.
  3. Blog posts only: the blog is public and can’t hold the private detail; it isn’t part of the PR flow.
  4. MADR files in homelab-gitops, synced to the blog (chosen): reviewed with the change, one log, a public copy.

Consequences

  • Good: One reviewable log for the whole platform.
  • Good: The public playbook is generated, so it can’t silently drift.
  • Bad: The blog sync has to be run each phase.
  • Bad: The scrubber’s deny-list must be maintained.

ADR-0002: Self-host the platform on the DGX Spark; GitHub Pages stays static

Accepted 2026-10-10 · infra-local, sakharams.github.io

Context

cloudutsuk.com is a Jekyll site on GitHub Pages, which can’t run a backend. The platform needs long-running services (portal, inventory, RAG, LLM serving, monitoring) and the owner wants no additional hosting cost. A DGX Spark (GB10, 20 cores, about 121 GB unified memory, 3.3 TB free disk) is available at home.

Decision

  • All platform compute runs on the DGX Spark at $0 hosting cost; the only cloud spend is the labs the owner chooses to deploy.
  • The blog stays a static site on GitHub Pages and only links to the portal.
  • Public access goes through Cloudflare (ADR 0003).

Considered options

  1. Cloud PaaS or managed Kubernetes: monthly cost, and GPU capacity for a local LLM is expensive.
  2. GitHub Codespaces or Actions as hosting: not meant for long-running services.
  3. Self-host on the DGX Spark (chosen): hardware already owned, local GPU, no hosting bill.

Consequences

  • Good: $0 hosting.
  • Good: A local GPU makes the local LLM and RAG possible.
  • Good: Full control of the stack.
  • Bad: A single host: availability is designed for fast recovery, not HA (ADR 0015).
  • Bad: Memory is the hard limit and needs admission control (ADR 0017).
  • Bad: Electricity is the real running cost.

ADR-0003: Public access via Cloudflare Tunnel and Access; move DNS to Cloudflare

Accepted 2026-10-10 · infra-local, homelab-gitops

Context

The DGX sits on a home network. Opening inbound ports exposes the home IP and needs a reverse proxy plus its own auth. The portal, Grafana, Argo CD and Phoenix must be reachable from anywhere behind SSO. cloudutsuk.com’s DNS is at GoDaddy and holds only GitHub Pages records.

Decision

  • A Cloudflare Tunnel (cloudflared, two replicas in platform) connects outbound only; no inbound port is opened.
  • Cloudflare Access (free plan) guards portal., grafana., argocd. and phoenix.cloudutsuk.com, with Keycloak as its identity provider and Google as break-glass (ADR 0028).

  • cloudutsuk.com’s DNS moves from GoDaddy to Cloudflare. Every record is exported first and compared with dig after the move.

  • On hold (2026-10-11): the Cloudflare Zero Trust team asks for a payment method, which the owner declined. ADR 0037 (accepted the same day) replaces the tunnel and Access with Tailscale; the DNS move above is done and stays.

Considered options

  1. Port forwarding plus a reverse proxy: inbound exposure of the home network and its IP.
  2. A VPN or tailnet: every viewer needs a client; no browser SSO in front of the apps.
  3. A paid tunnel service: custom domains cost money.
  4. Cloudflare Tunnel and Access (chosen): free, outbound only, SSO at the edge; needs DNS on Cloudflare.

Consequences

  • Good: No inbound ports.
  • Good: SSO before any request reaches the DGX.
  • Good: Free.
  • Bad: DNS has to move, with a risk of losing records (mitigated by exporting first).
  • Bad: Cloudflare becomes a dependency for remote access (break-glass covers Keycloak, not Cloudflare).

ADR-0004: Extend the Backstage self-service portal, built as a custom image

Accepted 2026-10-10 · self-service-portal, homelab-services

Context

A portal is needed for inventory and cost, the labs catalog with deploy and destroy, chat, and links into Grafana and Argo CD. The self-service-portal Backstage app already exists. Backstage compiles plugins into the app, so it isn’t a base image with runtime plugins.

Decision

  • Extend self-service-portal with three plugins: plugins/inventory, plugins/labs, plugins/chat, plus the Argo CD and Kubernetes plugins.
  • Plugin backends call the homelab-services APIs with service tokens (ADR 0029).
  • The app is built into its own arm64 image by CI and deployed by Argo CD like any other app.
  • Image as built (2026-10-13, Phase 3 step 1): Backstage 1.55.2. A multi-stage Dockerfile installs, type-checks and bundles inside the build (node 24.21.0, pinned by digest), so the shared build-image.yaml builds it from a clean checkout like any other image. The runtime stage is the slim image, runs as uid 1000 with a read-only root, and drops npm. The portal’s test job (tsc, lint, Prettier, unit tests) runs first. Deployed as apps/portal in homelab-gitops, with the homelab’s settings in a ConfigMap loaded after the image’s own config files.

Considered options

  1. A new custom React app: rebuilds what Backstage already gives: auth, catalog, plugins.
  2. Grafana only: good for dashboards, not for actions such as deploy, or for chat.
  3. Extend Backstage (chosen): reuses the existing app and its plugin model.

Consequences

  • Good: Reuses an existing app.
  • Good: One place for catalog, labs, cost and chat.
  • Bad: Every plugin change is a rebuild.
  • Bad: Backstage upgrades are regular work.
  • Bad: Needs an arm64 Node base image.

ADR-0005: Repo layout: infra-local, homelab-gitops and homelab-services, created through infra-github

Accepted 2026-10-10 · infra-local, homelab-gitops, homelab-services, infra-github

Context

The platform has infrastructure (clusters), desired state (what Argo CD deploys) and service code. They change at different rates and need different permissions: CI writes image tags into the desired state, but must never touch the cluster definitions.

Decision

  • Three new private repos in the cloudutsukpilot org, created through infra-github (group environments/github/homelab/), created on 2026-10-11:
    • infra-local: the homelab network, the long-lived k3d clusters, host services.
    • homelab-gitops: Argo CD desired state and this ADR log.
    • homelab-services: Python services, one image each.
  • Existing repos are extended: self-service-portal, sre-agent, sakharams.github.io, infra-github, terragrunt_shared_module and the infra-* repos (the lab tag).

Considered options

  1. One monorepo: mixes infra state, deploy state and code; CI triggers and permissions get coarse.
  2. One repo per service: too many repos for one owner.
  3. Split by change rate and permission (chosen).

Consequences

  • Good: CI only needs write access to homelab-gitops.
  • Good: Repos are created and governed as code like every other repo.
  • Bad: A change can span repos (code, then a tag commit).
  • Bad: Needs a GitHub App for cross-repo commits.

ADR-0006: Long-lived k3d clusters as code with the shared module (k3d config files); lab clusters stay Makefile-owned

Accepted 2026-10-10 · infra-local, terragrunt_shared_module

Context

The platform needs several isolated Kubernetes clusters on one host, and every lab needs a throwaway cluster of its own. The shared Terragrunt module already has cloud/k3d/.../k3d_cluster, and the lab repos already have make up/down/status.

Decision

  • core, platform, mon and ai are k3d clusters created by infra-local with the shared module, following the same Terragrunt workflow as the cloud repos.
  • The units are docker_network (kreuzwerker/docker) for the homelab network and k3d_config_cluster, which renders k3d’s own config file (k3d.io/v1alpha5) and runs the k3d CLI from terraform_data. plan shows the rendered config. Any change to it recreates the cluster (data on /srv/homelab survives), so settings that change after creation (OIDC flags, registry mirrors) go in k3s drop-ins under /srv/homelab/k3s/<cluster>/, which need only k3d cluster stop/start (not docker restart, which drops the host aliases).
  • Not the shared module’s original k3d_cluster unit: its moio/k3d provider calls update.k3s.io at start-up and panics when that fails (it did on 2026-10-11, so no plan could run), can’t set host aliases, and hasn’t been released upstream since 2023.
  • k3d writes host aliases into a node’s /etc/hosts only at create or k3d cluster start; Docker drops them on every container start, including after a reboot. infra-local/host/k3s/node-hosts.sh re-applies them from the node label k3d keeps them in, and the boot unit runs it (lesson L026).
  • Lab clusters (lab-<slug>) stay owned by each lab repo’s Makefile, because they’re short-lived.
  • Exception (2026-10-11): ports on a cluster’s load balancer can be added or removed in place, so they go in load_balancer_ports, which is kept out of the rebuild hash and applied with k3d node edit <cluster>-serverlb --port-add/--port-delete (only that container restarts). core’s port 80 (HTTP to HTTPS redirect) was added this way; the DNS port for the other clusters will be too.

Considered options

  1. One cluster with namespaces: a lab or the GPU stack could take down everything; labs need cluster-level resources.
  2. k3s straight on the host: only one cluster per host.
  3. kind or minikube: k3d is lighter and the module already exists.
  4. The k3d_cluster unit (moio/k3d provider): plans break whenever update.k3s.io is unreachable; no host aliases.
  5. k3d clusters as code (chosen).

Consequences

  • Good: Failure isolation per cluster.
  • Good: Same IaC workflow as the clouds.
  • Bad: Each cluster’s k3s server costs about 0.5-1 GB.
  • Bad: Cross-cluster calls need DNS, TLS and auth (ADRs 0024, 0029).

ADR-0007: A single central Argo CD (hub and spoke) manages every cluster

Accepted 2026-10-10 · homelab-gitops, infra-local

Context

Four long-lived clusters and any number of lab clusters need deployments, and lab clusters need the same add-ons (collector, External Secrets, cert-manager) as soon as they exist.

Decision

  • One Argo CD in platform manages all clusters through cluster secrets at https://k3d-<name>-server-0:6443 on the homelab network.
  • infra-local registers the long-lived clusters; the lab runner registers lab clusters after make up and removes them after make down.
  • ApplicationSets with a cluster generator (label homelab/role=lab) install the shared add-ons on every lab cluster.
  • Argo CD manages its own config from homelab-gitops (app-of-apps).

Considered options

  1. Argo CD in every cluster: about 1 GB per cluster and one UI per cluster.
  2. Flux in every cluster: same per-cluster cost; no central UI.
  3. One central Argo CD (chosen).

Consequences

  • Good: One UI and one config.
  • Good: New lab clusters get their add-ons automatically.
  • Bad: If platform is down nothing new deploys (running workloads keep running).
  • Bad: Argo CD needs a credential for every cluster (short-lived, ADR 0029).

ADR-0008: CI on GitHub Actions with ARC runners on the DGX; CD by Argo CD pulling homelab-gitops

Accepted 2026-10-10 · homelab-gitops, homelab-services, self-service-portal, sre-agent

Context

The repos live on GitHub. Builds must reach the local registry and must not give GitHub any credential that can change a cluster. The DGX has no inbound ports (ADR 0003).

Decision

  • GitHub Actions orchestrates; every job runs on Actions Runner Controller (ARC) runner pods in the platform cluster, which connect outbound to GitHub. Images are built with rootless BuildKit and pushed to Zot (ADR 0034) from main only.
  • The last step of a main build commits the new image digest to homelab-gitops with a short-lived GitHub App token. Argo CD polls homelab-gitops every 3 minutes and syncs.
  • No kubeconfig ever goes to GitHub. Images are deployed by digest.
  • arm64 checked 2026-10-10: the runner image, the scale-set controller (0.12.1-0.13.1, no latest tag, so pin a version) and moby/buildkit:rootless all ship arm64.
  • Gitea with Gitea Actions is the documented fully local fallback, not built in v1.
  • Faster pickup (2026-10-15): Argo CD still has no inbound webhook from GitHub. The build workflow’s promote job, which runs on the homelab runners inside platform, posts a GitHub-style push event to Argo CD’s webhook right after the digest commit, so the apps refresh at once instead of at the next 3-minute poll. The webhook has no secret yet (it only triggers a refresh); signing it is a follow-up.

Considered options

  1. GitHub-hosted runners pushing to the cluster: needs inbound access and a kubeconfig in GitHub; can’t reach Zot.
  2. Gitea Actions: fully local, but the repos are on GitHub; kept as the fallback.
  3. Jenkins or Tekton: heavier to run and maintain. Tekton was re-checked when Phase 1 started (2026-10-11) and still lost. Its triggers (Tekton Triggers, Pipelines as Code) need GitHub to send webhooks into the cluster, but the platform has no inbound ports (ADR 0030, ADR 0037). The only documented relay, gosmee, is a hosted service that isn’t meant for production. Every repo’s workflows, the shared Terragrunt module’s IaC pipeline among them, would have to be rewritten as Tasks. Pull requests would lose native checks unless the Pipelines as Code GitHub App were also run. What Tekton does better, Chains signing and SLSA provenance through a KMS key, Cosign with the OpenBao transit key already covers (ADR 0033).
  4. ARC runners plus Argo CD pull (chosen).

Consequences

  • Good: Nothing inbound; no cluster credentials in GitHub.
  • Good: Self-hosted runner minutes are free.
  • Good: Deploys are exactly the scanned and signed digest.
  • Bad: A GitHub outage stops new builds and deploys (running apps continue).
  • Bad: Runner images and tools must be arm64.

ADR-0009: A central mon cluster with Grafana LGTM and Alertmanager for every cluster

Accepted 2026-10-10 · homelab-gitops, infra-local

Context

Each existing lab bundles its own LGTM stack, which duplicates memory and gives no fleet view. The alert-driven agent (ADR 0014) needs one alert source.

Decision

  • A long-lived mon cluster runs Mimir, Loki, Tempo, Grafana and Alertmanager with an OTel gateway collector; retention is about 15 days on a host-path volume.
  • Every cluster, labs included, sends OTLP to the gateway with k8s.cluster.name set.
  • Argo CD isn’t in mon’s runtime path, so monitoring keeps working if platform breaks.
  • Labs move to this model through lab contract v2 (no bundled LGTM).

Considered options

  1. LGTM per lab: duplicated memory; no fleet view or shared alerts.
  2. A hosted observability free tier: limits, and telemetry leaves the box.
  3. One central mon cluster (chosen).

Consequences

  • Good: One Grafana and one rule set for all clusters.
  • Good: Lower memory per lab.
  • Bad: Existing labs must migrate.
  • Bad: mon is a shared dependency; the dead-man’s switch covers its failure (ADR 0015).

ADR-0010: Local LLM serving with vLLM on the DGX; model choice and memory budget

Accepted 2026-10-10 · homelab-gitops, infra-local

Context

The chatbot and the SRE agent need an LLM and an embedding model. Running them locally keeps data on the box and costs nothing per token. The GB10 has one GPU and unified memory shared with everything else.

Decision

  • Serve the chat model and the embedding model with vLLM (OpenAI-compatible API) behind the LiteLLM gateway (ADR 0022).
  • Chat: a ~20-30B mixture-of-experts model in FP8 or MXFP4 (candidates gpt-oss-20b, Qwen3-30B-A3B), starting with about 30 GB (--gpu-memory-utilization about 0.25) and growing toward 35-40 GB only if Phase 5 measurements leave room. Embeddings: bge-m3 or Qwen3-Embedding-0.6B.
  • Memory is budgeted at mid-range: on 2026-10-11, with the DGX also used as a desktop, about 81 GB of 121 GB was free, so the platform plans for about 75 GB and leaves 25-35 GB for the desktop.
  • Where it runs (decided by the 2026-10-11 spike, infra-local/spikes/gpu-k3d): inside the ai k3d cluster, on a k3s image with the NVIDIA container toolkit (runtimeClassName: nvidia) and the NVIDIA device plugin. Image nvcr.io/nvidia/vllm (26.07, vLLM 0.24.0) works on the GB10.
  • Sharing the GPU: the GB10 has no MIG, so the device plugin time-slices it (4 replicas). That gives no memory isolation; each vLLM’s --gpu-memory-utilization is a fraction of the whole 121 GB unified pool (0.1 measured at about 12 GB), so the chat model starts at about 0.25.
  • Embeddings: Qwen3-Embedding-0.6B (tested: 1024-dim vectors, served next to the chat model). BAAI/bge-small crashed in this vLLM build; bge-m3 is untested.
  • Still to choose: the first chat model and its licence, from the candidates above, in Phase 5, recorded as its own ADR.
  • Chosen (2026-10-15, Phase 5 step 1): gpt-oss-20b, Apache 2.0, in a smaller slice than planned above, because the host had far less free memory than this ADR budgeted (ADR 0046).
  • Replaced (2026-10-16): Qwen3.5-35B-A3B in FP8, the newer generation of the Qwen candidate, for tool calling and context (ADR 0049).

Considered options

  1. Ollama or Docker Model Runner: simpler, but less batching and throughput for concurrent users.
  2. TensorRT-LLM or NIM: fast, but heavier to build and licence for this use.
  3. llama.cpp server: light, but weaker continuous batching.
  4. vLLM (chosen): continuous batching, prefix caching, OpenAI API, arm64 builds.

Consequences

  • Good: No per-token cost, data stays local.
  • Good: The OpenAI API keeps clients portable.
  • Bad: The model takes about a third of the box’s memory.
  • Bad: Time-slicing has no memory isolation; a mis-set utilization can starve the other model or the host.
  • Bad: GPU memory doesn’t show in pod or container memory metrics (see ADR 0026).

ADR-0011: Qdrant for vectors and Redis (RedisVL) for the semantic cache

Accepted 2026-10-10 · homelab-services, homelab-gitops

Context

RAG needs a vector store with metadata filters (source, repo, lab, cloud, path, commit) and a cache that returns earlier answers to semantically equal questions.

Decision

  • Qdrant, one collection with payload filters and HNSW plus payload indexes; its web UI’s Visualize tab gives a 2D map of the vectors.
  • Redis with RedisVL SemanticCache: a hit needs cosine similarity of at least 0.92; TTL 24 hours; the key includes the index version, so a re-index invalidates stale answers.
  • Both run in the ai cluster and are rebuildable (re-ingest, cold cache), so neither is backed up.

Considered options

  1. pgvector: fewer moving parts, but weaker filtered and hybrid search and no vector UI.
  2. Chroma: simpler, less suited to filtered production search.
  3. Weaviate or Milvus: heavier to run on one box.
  4. Qdrant plus RedisVL (chosen).

Consequences

  • Good: Fast filtered search.
  • Good: Repeated questions skip the LLM.
  • Bad: Two more stateful services to run.
  • Bad: Threshold 0.92 must be tuned with the golden question set.

ADR-0012: Six-hour schedules with manual sync for inventory and RAG ingest

Accepted 2026-10-10 · homelab-services

Context

Inventory and RAG need refreshing, but cloud APIs have limits and costs (AWS Cost Explorer charges $0.01 per call) and cost data lags up to 24 hours anyway.

Decision

  • The inventory CronJob and the rag-ingest CronJob run every 6 hours.
  • “Sync now” and “Re-index now” in the portal (and task sync:inventory, task rag:ingest) create a one-off Job from the CronJob.
  • AWS Cost Explorer’s scheduled pull runs once a day; the manual button can force it.
  • Every view shows “last synced” and the provider’s cost lag.

Considered options

  1. Every 15 minutes or hourly: more API calls and cost for data that lags anyway.
  2. Manual only: data goes stale unnoticed.
  3. Six hours plus manual (chosen).

Consequences

  • Good: Low API cost.
  • Good: Fresh data on demand.
  • Bad: Up to 6 hours stale between runs (shown in the UI).

ADR-0013: Cost attribution through the existing common_tags plus an optional lab tag

Accepted 2026-10-10 · terragrunt_shared_module, infra-azure, infra-aws, infra-gcp, homelab-services

Context

Cost APIs can group spend by tag. The infra-* repos already apply common_tags and the OPA policy required_tags.rego enforces them. The portal should show cost per lab.

Decision

  • Add one optional tag, lab: <slug>, to the common_vars.yaml of folders that create lab resources, and to lab scripts that use the CLI directly. Add it to the policy’s allowed set; it isn’t required.
  • One-time billing settings: activate lab, environment and github-repository as AWS cost allocation tags; GCP uses lowercase labels.
  • Untagged resources group by github-repository and environment.
  • DGX cost is estimated: GPU power from DCGM plus a configured baseline for the rest of the box, times the tariff (calibrated once with a plug meter if available).

Considered options

  1. A new tagging scheme: duplicates what already exists and is enforced.
  2. Cost per account only: can’t attribute a lab that spans clouds.
  3. Existing tags plus lab (chosen).

Consequences

  • Good: Almost no IaC change.
  • Good: Cross-cloud cost per lab.
  • Bad: AWS ignores tags until they’re activated.
  • Bad: The DGX figure is an estimate.

ADR-0014: Extend sre-agent for alert-triggered troubleshooting; human approval for every write

Accepted 2026-10-10 · sre-agent, homelab-gitops

Context

The existing sre-agent has Kubernetes diagnosis subagents, human-in-the-loop approvals and Slack. Alerts from mon should start an investigation automatically. An agent reading logs and docs can be prompt-injected.

Decision

  • Alertmanager posts to sre-agent POST /alerts (OAuth2 client credentials, ADR 0029). The cluster label selects the target cluster.
  • The agent uses the existing subagents plus new RAG and mon query tools, and the local models through the gateway. It writes an RCA to Slack and the portal.
  • Its cluster access is read-only (short-lived tokens from OpenBao). Every write needs human approval; fixes to GitOps-managed apps are PRs to homelab-gitops.
  • Alert storms: Alertmanager groups by cluster and alertname, the agent de-duplicates by fingerprint and runs at most one investigation per cluster, with a bounded queue.
  • It runs in platform, not ai, so it can still post raw evidence when the models are down.
  • Lab experiments become its eval cases.

Considered options

  1. A new agent: rebuilds the subagents, approvals and Slack integration.
  2. Alert routing to Slack only: no investigation.
  3. Extend sre-agent (chosen).

Consequences

  • Good: Reuses working code.
  • Good: Read-only plus PRs keeps injection risk contained.
  • Bad: Agent quality depends on the local model; needs evals.

ADR-0015: Single host by design: optimise for recovery, not high availability

Accepted 2026-10-10 · infra-local, homelab-gitops

Context

Everything runs on one DGX. Power, ISP or hardware failure takes the whole platform down, and monitoring goes with it, so nothing would tell the owner.

Decision

  • Targets: RTO 1 hour, RPO 24 hours.
  • A dead-man’s switch: Alertmanager’s always-firing Watchdog pings a free external heartbeat service; a missed ping notifies the owner.
  • Two cloudflared replicas; Cloudflare shows a friendly offline page.
  • A systemd boot unit starts Docker, core, unseals OpenBao, then mon, platform and ai, then the lab runner. Labs stay down after a reboot.
  • A DR runbook and a quarterly restore test from scratch.

Considered options

  1. A second host for HA: doubles hardware and complexity for a personal platform.
  2. Cloud failover: ongoing cost against ADR 0002.
  3. Single host, recovery first (chosen).

Consequences

  • Good: Simple; failures are visible even when everything is down.
  • Bad: Downtime during outages is accepted.

ADR-0016: Back up only Postgres and secrets, encrypted to the R2 free tier

Accepted 2026-10-10 · infra-local, homelab-gitops

Context

Most state is rebuildable from git: clusters (infra-local), apps (Argo CD), Qdrant (re-ingest), Redis (cache), dashboards, and images in Zot. Some isn’t: Postgres data (inventory history, lab jobs, agent investigations, Keycloak), OpenBao, and a few config files.

Decision

  • Nightly: pg_dump of every Postgres (platform-db, ai-db, core-db), OpenBao raft snapshot, Zot config and htpasswd. Encrypted with age and uploaded to a Cloudflare R2 bucket (free tier 10 GB).
  • Persistent volumes live under /srv/homelab, so recreating a cluster keeps the data.
  • An offline copy of the OpenBao unseal keys, root token and age key.

  • A restore test every quarter.

  • Amended (2026-10-11, proposed in ADR 0039): R2 needs a payment method, which the owner declined. The scope above stays; the transport becomes Velero (dump hooks into an emptyDir, file-system backup with Kopia) to an Azure Blob container, and Kopia’s encryption replaces the age key.

Considered options

  1. Back up every volume: large and mostly rebuildable.
  2. No off-box backup: a disk failure loses history and identities.
  3. Postgres and secrets only (chosen).

Consequences

  • Good: Small, cheap backups.
  • Good: Recovery path is rebuild plus restore.
  • Bad: Rebuilding Qdrant means a full re-ingest.
  • Bad: Depends on the R2 free tier.

ADR-0017: Lab runner admission control (memory and ports) and a TTL for cloud labs

Accepted 2026-10-10 · homelab-services, sakharams.github.io

Context

Labs compete with the LLM for memory on one box; two labs can collide on host ports; a forgotten cloud lab keeps billing; a double click could deploy the same lab twice.

Decision

  • Each lab card declares deploy.memory_gb and deploy.ports. The runner refuses or queues a deploy that would push free memory below the headroom or that clashes on a port. Free memory is the host’s MemAvailable, not the sum of container metrics, because GPU memory isn’t charged to containers (ADR 0026). The portal shows the capacity left.
  • One operation per lab at a time (a per-lab lock); jobs are idempotent and recorded.
  • Cloud labs get a TTL (default 4 hours, extendable). The runner warns 30 minutes before and destroys with confirmation, or automatically if enabled. Budget alerts per cloud and per lab.
  • The runner only runs the allowlisted Make targets from the lab card, listens only on the Docker bridge, needs a bearer token and audit-logs every job.

  • As built (2026-10-11): homelab-services/services/lab-runner, a host systemd service running as the owner on 172.30.0.1:8095 (lab Makefiles need Docker, k3d, the lab repos and cloud CLI logins). The portal’s labs backend is its only client, through lab-runner-api.yaml in the portal repo. Target names and repo paths from the card are validated before they reach make (no shell, minimal environment, own process group, job timeout). State is in SQLite until platform-db exists, then Postgres through the same database URL. Cloud labs are destroyed automatically at the TTL.

  • Installed (2026-10-14, Phase 3 step 4): the runner is a systemd service on the DGX (lab-runner, 172.30.0.1:8095), records in SQLite for now; its token is in OpenBao kv/platform/lab-runner and the portal reads it through External Secrets. Argo CD registration is the lab’s own make up and make down (lab contract v2), not the runner’s. M3 passed from the portal with the OpenTelemetry lab: deploy in 8 min 48 s, destroy in 2 s. The runner only knows labs it started; reconciling labs started by hand through their status target is a follow-up.
  • Lab contract v2 (2026-10-14, Phase 2 step 5): deploy.ports lists every host port the lab’s k3d cluster publishes, its Kubernetes API port included; the blog’s check.py fails a card that misses one, because two clusters can’t bind the same host port.

Considered options

  1. First come, first served: a lab can starve the LLM or monitoring.
  2. Kubernetes quotas only: labs are separate clusters; quotas don’t span them.
  3. Runner-side admission and TTL (chosen).

Consequences

  • Good: Predictable memory use.
  • Good: No forgotten cloud spend.
  • Bad: Lab cards need two more fields (lab contract v2).

ADR-0018: Graceful degradation contract

Accepted 2026-10-10 · homelab-services, self-service-portal, sre-agent

Context

Cloud APIs rate-limit and fail, the LLM can be down or out of memory, and the platform runs on one host. A failure in one dependency shouldn’t blank the whole portal.

Decision

  • Inventory: timeouts, retries with backoff and a circuit breaker per provider; show the last known data with a “stale since” badge instead of an error.
  • Chat: if the LLM is down, return retrieval-only results with their sources.
  • Agent: if the LLM is down, post the raw evidence bundle to Slack without an RCA.
  • Every view shows “last synced”.

Considered options

  1. Fail fast with an error: the portal becomes unusable during any provider outage.
  2. Degrade gracefully (chosen).

Consequences

  • Good: Partial outages stay partial.
  • Bad: Each service needs explicit fallback paths and tests for them.

ADR-0019: Self-hosted runners only for private repos; ephemeral runner pods

Accepted 2026-10-10 · infra-github, homelab-gitops

Context

Self-hosted runners on a public repo can run untrusted code from fork PRs on the DGX, which sits on the home network next to every cluster.

Decision

  • The ARC runner scale set is attached only to private repos (the three new repos, the portal, the agent and lab repos).
  • The public blog repo keeps GitHub-hosted runners for Pages, with no access to anything local.
  • Runners are ephemeral: one job per pod, then the pod is deleted.
  • PR builds get a read-only OpenBao role and never push; only main pushes (ADR 0034).

Considered options

  1. Runners for every repo: fork PRs could run code on the DGX.
  2. Private repos only, ephemeral (chosen).

Consequences

  • Good: No untrusted code on the host.
  • Good: No state leaks between jobs.
  • Bad: Public repos can’t use local resources in CI.

ADR-0020: Platform SLOs and burn-rate alerting

Proposed 2026-10-10 · homelab-gitops

Context

Without targets, “up” is undefined and alerts are either noisy or missing.

Decision

  • Initial targets: portal availability 99% per month; chat time to first token under 3 s at p95; alert to RCA under 5 minutes.
  • SLO dashboards and multi-window burn-rate alerts in mon.
  • Why proposed: the targets are confirmed or adjusted after the first Locust baseline (Phase 3a), then this ADR is accepted.

Considered options

  1. No SLOs, threshold alerts only: noisy and not tied to what users feel.
  2. SLOs with burn-rate alerts (chosen).

Consequences

  • Good: Alerts tied to user impact.
  • Good: A baseline for performance work.
  • Bad: Targets need data first.

ADR-0021: Arize Phoenix for RAG and LLM trace visualisation, fed by the same OTel pipeline as Tempo

Accepted 2026-10-10 · homelab-services, homelab-gitops, sre-agent

Context

Grafana and Tempo show latency and errors, but not what was retrieved, the prompt, tokens, or answer quality. RAG tuning needs those views and evals.

Decision

  • rag-api, rag-ingest and sre-agent use OpenInference instrumentation. The OTel Collector sends those traces to both Phoenix and Tempo; one trace ID links them.
  • Phoenix runs in the ai cluster on the ai-db Postgres (schema phoenix, ADR 0035), about 1-2 GB, at phoenix.cloudutsuk.com behind Access. Each chat answer has a “View trace” link.
  • Split of duties: Grafana covers operations (AI dashboards, SLOs, every alert); Phoenix covers answer quality (LLM-aware traces, retrieval and hallucination evals, embeddings map).
  • Added in Phase 5. Licence: Elastic License 2.0, free to self-host for this use.

Considered options

  1. Grafana and Tempo only: no retrieval or prompt view, no evals.
  2. Langfuse: MIT core, but needs ClickHouse as well, heavier on this box; the fallback.
  3. Phoenix (chosen).

Consequences

  • Good: See exactly why an answer was good or bad.
  • Good: Evals after each re-index.
  • Bad: Another UI next to Grafana.
  • Bad: Not an OSI licence.

ADR-0022: The whole AI stack in its own ai cluster behind a LiteLLM gateway

Accepted 2026-10-10 · infra-local, homelab-gitops, homelab-services

Context

The AI stack (models, embeddings, vector store, cache, RAG services, Phoenix) is upgraded and tuned together and is the most likely to exhaust memory. There is one GPU, so more replicas on the same box don’t add capacity.

Decision

  • An ai k3d cluster, the only one built on the GPU-enabled k3s image, runs rag-api, rag-ingest, Qdrant, the Redis cache, Phoenix with ai-db, the LiteLLM gateway, vLLM chat and vLLM embeddings.
  • Callers use only the gateway’s OpenAI-compatible endpoint, by model name.
  • The GB10 has no MIG: vLLM pods share the GPU through the device plugin’s time-slicing, and each one’s --gpu-memory-utilization is its memory slice (ADR 0010).
  • As built (2026-10-11): ai runs on k3s-gpu (infra-local/images/k3s-gpu: k3s plus the NVIDIA container toolkit, the device plugin baked in) with gpus: all; k3s registers the nvidia RuntimeClass and the node advertises 4 time-sliced nvidia.com/gpu. The image tag ends in a hash of its build files, because a stale image once hid behind an unchanged tag (lesson L027). The cluster’s memory cap (16 GB) covers only CPU-side memory: GPU allocations aren’t charged to the container.
  • As built (2026-10-15, Phase 5 steps 1 and 2): LiteLLM v1.103.1 in namespace litellm, at https://llm.ai.home.cloudutsuk.com (tailnet only), serving qwen3.5-35b-a3b (ADR 0049; gpt-oss-20b until 2026-10-16, ADR 0046) and qwen3-embedding-0.6b from the models namespace; the model list is only in git. Its keys and spend live in ai-db (Postgres 17, namespace ai-db), which Phoenix will share. Qdrant 1.19.1 (qdrant) and Redis 8.10.2 (redis-cache) run next to it, each with its own key or password from OpenBao and a NetworkPolicy that admits only the RAG namespace.
  • The SRE agent stays in platform so it can report when ai is down.
  • Scaling path: tune the single instance (batching, FP8/FP4, prefix caching) and lean on the cache; give each extra model a fixed memory slice, loading rarely used ones on demand; add a second box or a hosted model as another gateway backend. KEDA only once a second GPU node exists. mon alerts on vLLM queue depth and time to first token.

Considered options

  1. AI services in platform: an out-of-memory model could take down the portal and GitOps.
  2. Callers talk to vLLM directly: every caller changes when a model or backend changes.
  3. A dedicated ai cluster plus gateway (chosen).

Consequences

  • Good: Failure isolation.
  • Good: Models can change without touching callers.
  • Bad: One more cluster’s overhead.
  • Bad: The gateway is a hop to monitor.

ADR-0023: OpenBao as the local secret manager; External Secrets Operator in every cluster

Accepted 2026-10-10 · infra-local, homelab-gitops

Context

Secrets are needed in every cluster (OAuth clients, cloud credentials, tokens, database passwords) and by CI. They must never be in git, must work offline, and must cost nothing.

Decision

  • OpenBao (KV v2) in core is the source of truth, laid out as core/*, platform/*, mon/*, ai/*, labs/<slug>/* and shared/*.
  • Every cluster runs External Secrets Operator with a ClusterSecretStore and its own Kubernetes-auth mount; each cluster’s policy reads only its paths plus shared/*. The lab runner creates and removes a lab’s mount and policy.
  • Outside Kubernetes: the lab runner uses an OpenBao agent with AppRole; CI uses GitHub OIDC.
  • After a reboot the boot unit unseals from a root-only key file; nightly raft snapshots go to R2; the unseal keys have an offline copy.
  • This replaces the earlier SOPS plus KSOPS idea; at most one SOPS-encrypted seed file remains for DR.
  • As built (2026-10-11):
    • One KV v2 mount, kv/, with the folders above. Kubernetes auth mounts are named kubernetes-<cluster>; in core, role external-secrets gets policy core-eso (kv/core/*, kv/shared/*).
    • The other clusters (mon first, 2026-10-11) are set up by `host/openbao/configure-cluster.sh ` with the admin token: policy `-eso`, a mount pointed at that cluster's API on the gateway, and `disable_local_ca_jwt`, because OpenBao's own service account (in `core`) can't review another cluster's tokens. OpenBao sends each client's own JWT to that cluster's TokenReview API instead, so External Secrets' service account there is bound to `system:auth-delegator`. Their ClusterSecretStores use `https://bao.core.home.cloudutsuk.com`. Checked in `mon`: its own path syncs and `kv/core/*` is refused (403).
    • Shamir keys, 5 shares with threshold 3, in /etc/homelab/openbao/init.json (root, 0600), with an offline encrypted copy. infra-local/host/openbao/{init,unseal,configure-core}.sh call the API through a localhost port-forward and never print a key or token.
    • The root token is break-glass only. Day-to-day changes use a periodic homelab-admin token (30-day period; no seal, rekey, root generation or audit changes) in a 0600 file of the owner. The root token is revoked once an admin login through Keycloak exists.
    • The audit device is declared in the server config: OpenBao 2.x refuses audit devices created through the API.
    • The ClusterSecretStore is a raw resource in a small Helm release, because Terraform’s kubernetes_manifest can’t type the External Secrets v2 Vault schema (ADR 0036).

Considered options

  1. HashiCorp Vault: same API, but the BSL licence.
  2. SOPS with KSOPS: secrets in git; rotation means commits; needs an Argo CD plugin.
  3. Sealed Secrets: per-cluster keys don’t suit disposable lab clusters.
  4. A cloud secret manager: cost and an internet dependency.
  5. OpenBao plus ESO (chosen).

Consequences

  • Good: No secrets in git.
  • Good: Short-lived and dynamic credentials possible.
  • Good: Same pattern in every cluster.
  • Bad: OpenBao must be unsealed after every reboot.
  • Bad: core becomes critical.

ADR-0024: Internal DNS with PowerDNS for home.cloudutsuk.com and Let’s Encrypt via DNS-01

Accepted 2026-10-10 · infra-local, homelab-gitops

Context

Services in different clusters need stable names, the DGX must resolve them, and every cross-cluster call should use trusted TLS without distributing a private CA.

Decision

  • PowerDNS Authoritative (SQLite) in core is the only server for the internal zone home.cloudutsuk.com, never published in Cloudflare.
  • ExternalDNS in every cluster (pdns provider) writes Ingress and Service names such as grafana.mon.home.cloudutsuk.com; ownership TXT records stop clusters overwriting each other.
  • The core load balancer publishes DNS on the gateway address 172.30.0.1:53, and the DGX uses systemd-resolved split DNS to send only this zone there.
  • Pods can’t use 172.30.0.1:53: k3d’s DNS fix redirects that address inside every node to Docker’s embedded resolver (found while building step 8, 2026-10-11). So core’s CoreDNS forwards the zone to PowerDNS’s fixed ClusterIP service (10.11.0.53), and the other clusters forward to a second port that core’s load balancer publishes on the gateway: 172.30.0.1:1053 (added 2026-10-11 with mon; k3d’s redirect only matches port 53).
  • Ingresses report their cluster’s published address (Traefik’s ingress endpoint), so ExternalDNS points names there: 172.30.0.1 for core, and each other cluster’s own bridge address (ADR 0030).
  • cert-manager issues Let’s Encrypt certificates through a Cloudflare DNS-01 challenge; only _acme-challenge TXT records are written to Cloudflare, with a token scoped to DNS edit.
  • Public names stay in Cloudflare behind the tunnel.

Considered options

  1. Hosts files: manual and per machine.
  2. CoreDNS with etcd for ExternalDNS: needs an etcd to run.
  3. A private CA: every client and containerd must trust it.
  4. PowerDNS plus Let’s Encrypt DNS-01 (chosen).

Consequences

  • Good: Readable names everywhere.
  • Good: Real certificates trusted by default.
  • Bad: Depends on Cloudflare for issuance and renewal.
  • Bad: One more service in core.

ADR-0025: A dedicated core k3d cluster for shared services, booted first, on a fixed homelab subnet

Accepted 2026-10-10 · infra-local

Context

Secrets, DNS, the registry and identity are needed by every other cluster, including before Argo CD exists. Putting them in platform would make every cluster depend on the busiest one.

Decision

  • A long-lived core cluster holds OpenBao, PowerDNS, Zot, Keycloak with core-db, External Secrets, ExternalDNS and cert-manager.
  • The homelab Docker network has a fixed subnet 172.30.0.0/24; core’s load balancer publishes DNS (53) and the registry (443) on the gateway 172.30.0.1.
  • Bootstrap order: network, core and its Helm releases, OpenBao init/unseal/seed, then mon, platform and ai with ESO, then Argo CD, which then manages everything including core’s apps.
  • core keeps working when platform is down.

Considered options

  1. Shared services in platform: every cluster depends on the busiest cluster.
  2. Host containers outside k3d: loses Kubernetes tooling and GitOps for them.
  3. A dedicated core cluster (chosen).

Consequences

  • Good: Clear dependency order.
  • Good: Small, stable cluster.
  • Bad: About 3-4 GB more memory.
  • Bad: A core outage affects new pulls, logins and secret refresh.

ADR-0026: Monitoring collection with the OTel Operator, keeping Prometheus metric names

Accepted 2026-10-10 · homelab-gitops, infra-local

Context

Every cluster, labs included, needs node, pod, workload and control-plane metrics plus logs and events. Community dashboards and mixins expect Prometheus metric names.

Decision

  • An ApplicationSet installs on every cluster: the OpenTelemetry Operator, the Prometheus Operator CRDs only, a collector DaemonSet (kubelet and cAdvisor, container logs, app OTLP) and a collector Deployment (kube-state-metrics, API server, Kubernetes events to Loki).
  • The Target Allocator reads ServiceMonitors and PodMonitors that charts already ship.
  • Every series gets cluster=<name>; unused histograms are dropped; about 0.3-0.5 GB per cluster.
  • GPU memory on the GB10 is unified system memory that isn’t charged to any container, so cAdvisor and the kubelet undercount the ai cluster by the size of the vLLM slices (spike, 2026-10-11). The Fleet dashboard shows node-exporter’s host memory next to per-cluster memory, and per-model memory comes from vLLM’s configured utilization and vllm:kv_cache_usage_perc.
  • Host exporters run as containers from infra-local: node-exporter, DCGM exporter, cAdvisor for Docker (memory per k3d cluster), blackbox exporter for public URLs and certificate expiry.

Considered options

  1. Prometheus server per cluster: more memory and a second pipeline next to OTel.
  2. OTel with OTel-native names: breaks community dashboards and mixins.
  3. OTel Operator with Prometheus names (chosen).

Consequences

  • Good: One pipeline for metrics, logs and traces.
  • Good: Community dashboards work unchanged.
  • Bad: The pack’s memory counts against each lab’s budget.

ADR-0027: Dashboards and alert rules as code, vendored and pinned in homelab-gitops

Accepted 2026-10-10 · homelab-gitops

Context

Dashboards built by hand in the UI are lost on rebuild and drift between versions.

Decision

  • Dashboards live in dashboards/<folder>/*.json, loaded by the Grafana sidecar from ConfigMaps; community JSON is vendored and version-pinned, never imported at runtime. Every dashboard has cluster and datasource variables.
  • Folders: Fleet, Kubernetes, Data, AI, Platform, Apps, SLOs and cost (plus Performance and Security from Phase 3a).
  • Alert rules (kubernetes and node mixins, GPU, vLLM, databases, OpenBao sealed, cert expiry, ExternalSecret failures, Argo CD drift, Watchdog) live in alerts/ and load into the Mimir ruler.
  • As built (2026-10-13, Phase 2 steps 4 and 4a): Kubernetes, Fleet, AI and Platform folders, 30 dashboards. Platform holds Argo CD, External Secrets, Kyverno, Traefik and cert-manager (upstream, at the versions that run) and two written here (Velero backups, OpenTelemetry collectors). 195 rules: the two mixins, cert-manager-mixin, GPU, vLLM, blackbox, Watchdog and the platform alerts above, except databases, which come with the first database’s monitoring. scripts/panel_check.py runs every panel per cluster; panels that are empty by design are listed with the reason. Data, Apps, SLOs and cost folders arrive with their components.

Considered options

  1. Dashboards edited in the UI: lost on rebuild.
  2. Import by ID at runtime: versions change under you; needs internet.
  3. Vendored, pinned, as code (chosen).

Consequences

  • Good: Rebuildable and reviewable.
  • Good: Upgrades are diffs.
  • Bad: Vendored dashboards must be refreshed by hand (Renovate can’t).

ADR-0028: Keycloak in core as the single OIDC issuer, brokering Google and Entra ID

Accepted 2026-10-10 · infra-local, homelab-gitops, self-service-portal

Context

People sign in to Backstage, Grafana, Argo CD, OpenBao, Phoenix, Zot and kubectl on five or more clusters. Services also need tokens to call each other. The owner uses Google and Entra ID accounts.

Decision

  • Keycloak in core (its own Postgres, core-db) is the only OIDC issuer.
  • It brokers Google and Entra ID. Cloudflare Access uses it as the IdP; every UI and every k3s API server trust it. Groups (platform-admins, viewers) map to roles in each tool and ClusterRoles in each cluster.
  • Break-glass: Cloudflare Access keeps Google as a direct IdP limited to the owner, and local admin accounts for OpenBao and Argo CD are sealed in OpenBao and the offline backup.
  • Only Keycloak stores email addresses (ADR 0035).
  • As built (2026-10-11): Keycloak 26.7.4 in core (bootstrap release, ADR 0036) on its own Postgres (core-db, Postgres 17.10, ADR 0035), at https://auth.core.home.cloudutsuk.com; with no fixed hostname it builds URLs from Traefik’s forwarded headers, so the public auth.cloudutsuk.com works through the tunnel without a change. Realm homelab, groups platform-admins and viewers, and the Entra ID provider (entra, OIDC v2, PKCE) come from infra-local/host/keycloak/configure-realm.sh; the Entra app (cloudutsuk-keycloak, single-tenant, tenant-wide consent for openid, profile, email, User.Read) from host/entra/keycloak-app.sh, with its client secret written straight to OpenBao (kv/core/keycloak). Google is not configured yet. The bootstrap admin in the master realm is for automation and emergencies only.
  • Cluster sign-in (2026-10-11): each k3s API server trusts the issuer through a drop-in (infra-local/host/k3s/oidc.yaml: client kubernetes, username claim email, groups claim groups, both prefixed oidc:), and oidc:platform-admins is bound to cluster-admin, oidc:viewers to view. Each node needs the host alias auth.core.home.cloudutsuk.com -> 172.30.0.1, because the API server resolves the issuer through the node, not CoreDNS. Done in mon.

Considered options

  1. Google or Entra directly in each tool: N integrations and no single group model; no service tokens.
  2. Dex: lighter, but fewer features for service clients and group mapping.
  3. Authentik or Zitadel: workable alternatives; Keycloak is the most widely documented with these tools.
  4. Keycloak (chosen).

Consequences

  • Good: One group change applies everywhere.
  • Good: Client credentials for services.
  • Bad: About 1 GB plus a database.
  • Bad: If core is down, people use break-glass.

ADR-0029: Service-to-service authentication

Accepted 2026-10-10 · homelab-services, self-service-portal, sre-agent, homelab-gitops

Context

Services in different clusters call each other over the homelab network. A network position alone mustn’t grant access.

Decision

  • App to app (Backstage, sre-agent, RAG tools to inventory, RAG API, lab runner): Keycloak client credentials, 5-minute JWTs with aud=<service> and role claims, checked against Keycloak’s JWKS.
  • Callers to the LiteLLM gateway: JWT if the free edition supports it (checked in Phase 0), otherwise one virtual key per client with budgets. Checked (2026-10-15, Phase 5 step 1): it doesn’t; LiteLLM v1.103.1 refuses enable_jwt_auth without an Enterprise licence. Every caller gets a virtual key limited to its models, created with the master key and kept in OpenBao kv/ai/litellm-clients; the master key is used by no service.
  • Alertmanager to sre-agent and OTel collectors to mon and Phoenix: OAuth2 client credentials.
  • sre-agent, lab runner and Argo CD to cluster APIs: short-lived ServiceAccount tokens from OpenBao’s Kubernetes secrets engine (15 minutes for the agent, read-only).
  • Every cluster to OpenBao: per-cluster Kubernetes auth; CI to OpenBao: GitHub OIDC.
  • Queries to Mimir and Loki go through Grafana’s API with a Keycloak JWT.
  • All cross-cluster traffic is HTTPS on *.home.cloudutsuk.com; NetworkPolicies allow only these calls. Keycloak events and OpenBao audit logs go to Loki. A service mesh is a later option.

Considered options

  1. Trust the network: any compromised pod can call anything.
  2. Static API keys everywhere: long-lived and hard to rotate.
  3. Service mesh mTLS now: heavier than needed for v1.
  4. Short-lived tokens per call type (chosen).

Consequences

  • Good: No long-lived credentials in pods.
  • Good: Every call is attributable.
  • Bad: Every service validates JWTs.
  • Bad: More moving parts to debug.

ADR-0030: Network plan: fixed homelab bridge, unique CIDRs per cluster, no inbound ports

Accepted 2026-10-10 · infra-local, homelab-gitops

Context

Several k3d clusters share one Docker host. Overlapping pod or service ranges make cross-cluster routing and debugging ambiguous; published API ports widen the attack surface.

Decision

  • homelab bridge 172.30.0.0/24, gateway 172.30.0.1 (publishes DNS 53 for the host and registry and ingress 443; pods reach DNS another way, see ADR 0024).
  • Pod and service CIDRs: core 10.10/10.11, platform 10.20/10.21, mon 10.30/10.31, ai 10.40/10.41, labs from 10.100 upward (allocated by the lab runner).
  • Traefik plus cert-manager TLS on every cluster; default-deny NetworkPolicies.
  • No inbound ports on the home router; the k3s API is never published to the LAN: each long-lived cluster’s API listens only on the homelab gateway (core 6441, platform 6442, mon 6443, ai 6444 on 172.30.0.1). Docker assigns container addresses from 172.30.0.128/25, leaving .2-.127 for fixed addresses.

  • Each long-lived cluster except core gets its own host address on the bridge for its ingress (2026-10-11, with mon): platform 172.30.0.2, mon 172.30.0.3, ai 172.30.0.4. Only one load balancer can publish 172.30.0.1:443, and that’s core’s. A systemd unit (infra-local/host/network) adds the addresses to br-homelab at boot and after a Docker restart, and net.ipv4.ip_nonlocal_bind=1 lets Docker publish on them even if it starts first. Each cluster’s load balancer publishes 80/443 there, and its Traefik reports that address, so ExternalDNS points *.mon.home.cloudutsuk.com at 172.30.0.3. The API ports stay on the gateway.

  • Labs gateway (2026-10-15, ADR 0045): 172.30.0.5 on the bridge is the labs gateway’s address; every lab UI is reached by name through it (*.labs.home.cloudutsuk.com).
  • Labs as built (2026-10-14, lab contract v2): a lab’s pod and service ranges are a fixed pair in its own k3d/cluster.yaml (eBPF 10.100/10.101, OpenTelemetry 10.102/10.103, new labs from 10.104), chosen when the lab is scaffolded, not allocated by the lab runner: make up creates the cluster from that file. Each lab’s API listens on 127.0.0.1 on its own port (6450 up); Argo CD reaches it on the server node’s homelab address.

Considered options

  1. k3d defaults: every cluster gets the same ranges.
  2. A fixed plan (chosen).

For the other clusters’ ingress:

  1. SNI passthrough through core’s Traefik: one address, but every UI depends on core’s ingress.
  2. Another port on the gateway (e.g. 8443): every URL carries the port.
  3. One host address per cluster (chosen): independent of core, at the cost of a host unit.

Consequences

  • Good: Unambiguous addresses and routes.
  • Good: Small attack surface.
  • Bad: The lab runner must allocate ranges.
  • Bad: The per-cluster addresses are host configuration outside Docker; without the unit, those clusters’ ingresses are unreachable.

ADR-0031: Testing strategy

Accepted 2026-10-10 · homelab-services, self-service-portal, sre-agent, homelab-gitops

Context

Changes deploy automatically after merge, so tests are the gate. The platform has APIs, a UI, an LLM and load-sensitive components on one box.

Decision

  • Every PR: unit tests (pytest; Jest via backstage-cli test), integration tests with Testcontainers (Postgres, Redis, Qdrant), Schemathesis API tests from each OpenAPI spec (fast profile). All block the merge.
  • Every deploy: an Argo CD PostSync smoke-test Job (health, Keycloak redirect, one inventory call, one RAG question, one gateway call); failure marks the app Degraded and rolls back. blackbox exporter probes public URLs continuously.
  • Nightly: Playwright journeys, the full Schemathesis profile, Locust load scenarios in a perf namespace (limits set, thresholds fail the run, results to a Performance dashboard).

  • On change: vllm bench serve, Phoenix evals over a golden set of about 30 questions, sre-agent evals from lab experiments. Chaos Mesh in a lab cluster later.

  • Unit tests as built (2026-10-13, Phase 1 step 4c): each service’s workflow in homelab-services has a test job (Ruff lint and format check, pytest, from uv.lock, on the homelab runners) that the image job needs, so a failure stops the run before the reusable build workflow starts. The tests stay in the caller rather than in build-image.yaml because each stack tests differently (pytest now, Jest for the portal). Until then hello-world had tests that no CI job ran. Testcontainers come with the lab runner (Phase 3), the first service with a database; Schemathesis with Phase 3a.
  • PostSync smoke tests as built (2026-10-12, Phase 1 step 6): a Job in the app’s folder with argocd.argoproj.io/hook: PostSync and hook-delete-policy: BeforeHookCreation (the last run’s Job and log stay until the next sync). hello-world’s (apps/hello-world/smoke-test.yaml) uses a pinned curl image and checks /readyz, / and that revision is a commit SHA. It is written to pass every Kyverno policy, including the ones still in Audit.
    • A failing test fails the sync operation, not the app’s health. Argo CD shows the failed hook and retries; there is no automatic rollback, because Argo CD can’t roll back an app with automated sync. The rollback is a git revert (or the next good digest commit).
    • Apps with a smoke test set syncPolicy.retry.refresh: true. Without it, Argo CD retried the broken revision for up to an hour and ignored the fix commit. With it, the next retry synced the fix about a minute after it was pushed.
    • A hook doesn’t count in Argo CD’s diff, so adding or changing only the test doesn’t trigger a sync; it runs with the next real change (normally the next digest commit) or a manual sync.
    • Tested by pointing the Service at a wrong port: the Deployment stayed Healthy, the smoke test failed (FAIL: GET /readyz) and the sync failed.

Considered options

  1. Unit tests only: misses integration, API contract and load issues.
  2. The layered strategy above (chosen).

Consequences

  • Good: Every deploy is verified.
  • Good: Performance baselines before SLOs are set.
  • Bad: Nightly runs cost memory while they run (counted like a temporary lab).

ADR-0032: Security scanning in the pipeline

Accepted 2026-10-10 · homelab-services, self-service-portal, sre-agent, homelab-gitops, infra-local

Context

The repos are private, so GitHub Advanced Security (CodeQL, push protection, SARIF code scanning) isn’t free. Scanning must be free, open source and run on arm64.

Decision

  • Secrets: Gitleaks in pre-commit and CI (full history on the first run).
  • SAST: Semgrep CE plus Bandit. SCA: Trivy fs plus Renovate. IaC: Trivy config, Checkov and Conftest with the shared OPA policies. Dockerfiles: Hadolint.
  • Images: Trivy image scan before push; Syft CycloneDX SBOM; licence scan.
  • DAST: nightly OWASP ZAP baseline of the portal and API scans of each service.
  • Posture: Trivy Operator in every long-lived cluster, kube-bench monthly.

  • Gates: secrets any finding; SAST high; SCA and images critical or high with a fix; IaC high or an OPA deny; ZAP new high alerts. Reports are CI artifacts plus a PR comment; Trivy Operator feeds a Security dashboard.

  • Pipeline gates as built (2026-10-12, Phase 1 step 4): in the reusable build-image.yaml, on every run including pull requests, before anything is pushed. Tools are pinned in homelab-gitops/ci/tools and baked into the runner image (ADR 0044).
    • Gitleaks scans the full history on every run (repos are small), not only the first.
    • Semgrep CE uses the registry’s p/default rules with metrics off; the rules aren’t pinned. The gate is severity HIGH or ERROR.
    • Bandit runs on tracked .py files and gates on HIGH severity.
    • Trivy fs (dependencies) and the image scan gate on HIGH or CRITICAL with a fixed version; config gates on HIGH or CRITICAL.
    • Checkov’s open-source checks carry no severity, so any failed check blocks. CKV_DOCKER_2 (HEALTHCHECK) is skipped by default because Kubernetes uses probes; a repo’s .checkov.yaml replaces that default.
    • Conftest runs the shared Dockerfile rules in homelab-gitops/policies/conftest (digest-pinned bases, non-root final USER, no ADD of URLs, no curl | sh); any deny blocks.
    • Hadolint gates on error or warning.
    • The image is built into an OCI layout on the runner, scanned there, and the same bytes are pushed.
    • Licences: reported (Trivy), not gated.
    • Exceptions live in the caller’s repo: .gitleaksignore, # nosemgrep, # nosec, .trivyignore, .checkov.yaml, .hadolint.yaml.
    • Reports are a security-reports-<image> artifact (30 days) and a gate table in the job summary, not a PR comment: a comment would need pull-requests: write in every caller.
    • A caller can narrow the SAST, dependency and IaC scans with scan-path. The runner image sets image-gate: report (ADR 0044).

Considered options

  1. GitHub Advanced Security: paid for private repos.
  2. A commercial scanner: cost.
  3. Open-source toolchain (chosen).

Consequences

  • Good: Free and runs locally.
  • Good: Covers every layer.
  • Bad: Several tools to keep updated.
  • Bad: No central findings store until DefectDojo (later).

ADR-0033: Supply chain: Cosign signing with an OpenBao transit key and Kyverno admission

Accepted 2026-10-10 · homelab-gitops, homelab-services

Context

Only images built and scanned by our pipeline should run in our namespaces, and public images should be pinned.

Decision

  • main builds sign images with Cosign using a key in OpenBao’s transit engine and attach the SBOM as an attestation (stored in Zot through the referrers API).

  • Kyverno in every cluster enforces: images only from registry.core.home.cloudutsuk.com; a valid signature on platform/* and labs/*; digest-pinned mirror/*; no :latest; non-root, read-only root filesystem and resource limits; Pod Security “restricted” except for documented exceptions (such as the GPU device plugin).

  • Signing as built (2026-10-12, Phase 1 step 4a): transit key cosign (ECDSA P-256, not exportable) in OpenBao. Only role main, which is bound to the reusable build-image.yaml (ADR 0043), may call transit/sign/cosign/*; Cosign puts the hash in the path (…/sha2-256). Cosign v3.1.3 signs with --key hashivault://cosign, no transparency log (--tlog-upload=false --use-signing-config=false; the first flag is deprecated in v3 in favour of a signing-config file). It stores a Sigstore bundle (v0.3) as an OCI referrer in Zot, with no .sig tag. The public key is homelab-gitops/apps/kyverno-policies/cosign.pub; check with cosign verify --key cosign.pub --insecure-ignore-tlog <image>@<digest>. Kyverno’s verifyImages (v1.19.1) still has to be checked against this bundle format (step 5).
  • SBOM attestation as built (2026-10-12, Phase 1 step 4): after signing, cosign attest --type cyclonedx with the same key attaches Syft’s CycloneDX SBOM of the pushed digest, also as an OCI referrer. Check with cosign verify-attestation --key cosign.pub --insecure-ignore-tlog --type cyclonedx <image>@<digest>. The image is scanned before it is pushed (ADR 0032), and crane pushes the scanned OCI layout unchanged, so the signed digest is the scanned one.
  • Admission as built (2026-10-12, Phase 1 step 5): Kyverno 1.19.1 (chart 3.9.1) on platform, mon and ai, with CEL ValidatingPolicy and ImageValidatingPolicy resources (apps/kyverno-policies).
    • verify-image-signatures checks the Cosign signature and the signed CycloneDX SBOM attestation of every platform/* and labs/* image against cosign.pub, with the transparency log off. It reads Cosign v3’s bundle referrers from Zot with a zot-pull secret in the kyverno namespace. Tested: signed with SBOM admitted, unsigned denied, signed without SBOM denied.
    • Kyverno resolves an image index to linux/amd64, which our arm64 images don’t have, so the verification errored. build-image.yaml now builds without BuildKit’s provenance attestation, and every image is a single manifest.
    • Deny: disallow-latest-tag, restrict-image-registries and verify-image-signatures.
  • As built (2026-10-13, Phase 1 step 5a): all seven policies deny on platform, mon and ai.
    • Every add-on now meets them through its chart values, in homelab-gitops and in infra-local’s releases. Public images are pinned by the digest of their upstream index; Zot converts Docker-format manifest lists to OCI when it mirrors a tag, so the digest a running pod reports can differ while the layers are the same. Restricted security contexts, read-only roots (emptyDirs for scratch) and requests with a memory limit apply everywhere, Helm hook jobs included. Argo CD’s Redis comes from Docker Hub through Zot, and Mimir’s data reaches the pod through a PV instead of a hostPath.
    • Four written exceptions (PolicyExceptions in templates/exceptions.yaml, only in the kyverno namespace): rootless BuildKit (Pod Security, ADR 0043), the log-tailing agent collector (Pod Security: root, hostPath), Velero’s node agent (Pod Security and read-only root) and the Actions runners’ writable home (read-only root).
    • Kyverno’s chart ships with PolicyExceptions off. Background reports still honour them, so the reports were clean while admission denied the runner pods. features.policyExceptions is now on.
    • failurePolicy is Ignore (one admission replica), so a pod created while Kyverno is down isn’t checked at admission. An evaluation error, such as an unreadable signature, denies.

Considered options

  1. No admission control: anything pullable can run.
  2. Keyless Cosign: needs a public transparency log and OIDC from every signer.
  3. Cosign with a transit key plus Kyverno (chosen).

Consequences

  • Good: Only scanned, signed images run.
  • Good: The signing key never leaves OpenBao.
  • Bad: About 0.3 GB per cluster for Kyverno.
  • Bad: Exceptions must be maintained.
  • Bad: Every new chart needs digests, security contexts and resources set before it can run.

ADR-0034: Zot in core as the only container registry

Accepted 2026-10-10 · infra-local, homelab-gitops, homelab-services

Context

CI needs a local registry with authentication, and clusters should pull public images through a local cache. The host is arm64. Harbor’s official images are amd64 only (checked against goharbor/harbor-core:v2.13.0).

Decision

  • Zot (CNCF, multi-arch) in core, served as registry.core.home.cloudutsuk.com on HTTPS 443 with a Let’s Encrypt certificate; storage on /srv/homelab/zot; retention keeps the last 20 tags per repo, anything referenced by homelab-gitops, and mirror entries used in 30 days.
  • Repositories: platform/* and labs/* (read puller, write ci-pusher), mirror/* (anonymous read, written only by Zot sync of docker.io, ghcr.io, quay.io, registry.k8s.io and nvcr.io; side-loading the 22 GB vLLM image into k3d took 4.5 minutes, the mirror makes it a one-time pull), charts/*.
  • CI logs in to OpenBao with GitHub OIDC (bound to main) and reads the ci-pusher key; PR builds never push. Clusters pull public images anonymously through registries.yaml mirrors and our images with a zot-pull secret from ESO, added to pods by a Kyverno mutate policy.

  • Nodes resolve the name through a k3d host alias to 172.30.0.1. People log in to the UI through Keycloak. The k3d-managed plain-HTTP registry is dropped.

  • As built (2026-10-11): chart 0.1.125 (Zot 2.1.21) as a bootstrap release in core (ADR 0036), data on /srv/homelab/zot/data, served at https://registry.core.home.cloudutsuk.com with a Let’s Encrypt certificate. Accounts admin, puller and ci-pusher come from kv/core/zot; External Secrets builds the bcrypt htpasswd file, so plain passwords never reach Zot. On-demand sync mirrors docker.io, ghcr.io, quay.io, registry.k8s.io and nvcr.io under mirror/<registry>/. http.compat: [docker2s2] lets the Docker CLI push Docker-format manifests; BuildKit pushes OCI.
  • Anonymous pulls of mirror/* work for containerd (the k3d nodes), which asks for the manifest directly. The Docker CLI first calls /v2/, which Zot answers with 401 without credentials, so people pull with the puller account.

  • registries.yaml for the other clusters is one host file, infra-local/host/k3s/registries.yaml, installed to /srv/homelab/k3s/registries.yaml and bind-mounted into each node, so a change needs a k3d cluster stop/start of that cluster, not a new cluster (ADR 0006). It rewrites each upstream to mirror/<registry>/; containerd falls back to the upstream when Zot is down. core itself has no mirror. Checked 2026-10-11: mon pulled its pause image through the mirror; ci-pusher pushed a test image that puller could read and an anonymous client couldn’t (401).

  • charts/* (2026-10-11): anonymous read like mirror/*, write by ci-pusher and admin. Third-party charts that aren’t published as OCI, or whose download fails from the DGX, are checked against their index digest and pushed here (first: charts/velero 12.2.0 and charts/velero-ui 0.16.0, ADR 0039).

  • CI sign-in as built (2026-10-12, ADR 0043): JWT auth github-actions/ in OpenBao. Role main reads the ci-pusher key only for homelab-gitops’ build-image.yaml at refs/heads/main. Role pr reads the puller key. Both keys are copies in kv/ci/, so CI never reads kv/core/zot.

  • Cluster pulls as built (2026-10-12, Phase 1 step 3): the puller password is also copied to kv/shared/zot-pull, which every cluster’s store can read. Each app namespace gets a zot-pull ExternalSecret (a dockerconfigjson template) and names it in imagePullSecrets. That stays until the Kyverno mutate policy adds it automatically; first used by apps/hello-world.

Considered options

  1. Harbor: no arm64 images.
  2. registry:2: no auth model, UI, scanning or sync.
  3. A cloud registry: cost and internet dependency for every pull.
  4. Zot (chosen).

Consequences

  • Good: Local, authenticated, arm64.
  • Good: A pull-through cache avoids Docker Hub limits.
  • Good: Built-in CVE rescans.
  • Bad: core must be up for new pulls.
  • Bad: Images aren’t backed up (rebuilt from git).

ADR-0035: Data ownership: one writing service per entity

Accepted 2026-10-10 · homelab-services, self-service-portal, sre-agent, homelab-gitops

Context

Several services share Postgres instances. If more than one service writes the same table, schema changes and bugs couple them, and backups and privacy are hard to reason about.

Decision

  • Each entity has exactly one writing service; others read through that service’s API.
  • Stores: platform-db (databases backstage, inventory, labs, agent, each with its own owner role), ai-db in the ai cluster (schemas rag and phoenix), core-db (Keycloak).
  • References across stores are soft (IDs checked by the owning service); no cross-database foreign keys.
  • Only Keycloak stores email addresses; everything else keeps an opaque user ID. Deleting a user is a Keycloak change plus deleting their chat history.
  • The ER diagram and ownership table are in the playbook’s Technical Architecture post.
  • platform-db as built (2026-10-13, Phase 3 step 1): Postgres 17.10 in platform, deployed by Argo CD (homelab-gitops apps/platform-db), data on the host through a static PV. Only backstage exists so far; Backstage keeps each plugin in its own schema inside it (pluginDivisionMode: schema) instead of creating a database per plugin. The other databases are added with their services. Velero’s pre-backup hook dumps every database (pg_dump -Fc) for the nightly backup.

Considered options

  1. A shared schema: tight coupling between services.
  2. A database per service, one writer per entity (chosen).

Consequences

  • Good: Services change their schema independently.
  • Good: Clear backup and privacy scope.
  • Bad: Joins across services happen in code.
  • Bad: More databases and roles to manage.

ADR-0036: Bootstrap charts through the shared module’s Helm layer; Argo CD for the rest

Accepted 2026-10-11 · infra-local, terragrunt-shared-module, homelab-gitops

Context

core boots first and must run OpenBao, External Secrets, cert-manager, PowerDNS, ExternalDNS and Zot before Argo CD or any other cluster exists (ADR 0025). Those releases need a tool other than Argo CD. The owner already installs platform charts with the Terragrunt shared module’s Helm layer in other application repositories, and wants the same workflow here.

Decision

  • Bootstrap charts are Terragrunt stacks in infra-local/environments/k3d/homelab/helm/<cluster>/, one folder per release, applied with entrypoint.py --cloud k3d --environment homelab --provider helm --stack <cluster> --resource <release>. plan comes first: only plan generates the stack file.
  • Releases use the shared helm_release: key with an explicit release-external-secrets: list. The external-secrets and cert-manager keys keep their own units (secret stores; switching off the module’s Sectigo issuers). One release per folder, named after the folder, because the entrypoint filters releases by that name.
  • Every release sets shared_values: false: the module’s default values for well-known release names target another platform’s managed clusters (node pools, workload identity, image pins).
  • Charts come from OCI registries where they exist, with the version at least 14 days old and the digest recorded in resources.yaml. Releases that hold only cluster-scoped objects live in kube-system, because the module creates and owns every other release namespace.
  • Things without a chart (PowerDNS, ClusterIssuers, the ClusterSecretStore, CoreDNS and Traefik settings) use the bjw-s app-template chart and its raw resources, not Terraform’s kubernetes_manifest.
  • Everything after bootstrap (apps, monitoring, the AI stack) is deployed by Argo CD from homelab-gitops with app-of-apps (ADR 0007, 0008). Argo CD itself is the last bootstrap release.

Considered options

  1. Argo CD for everything: Argo CD’s own secrets and registry would depend on releases it deploys.
  2. Helmfile or plain helm scripts: a second workflow next to the Terragrunt one used everywhere else, without the plan, state and policy checks.
  3. Terraform kubernetes_manifest for custom resources: fails on CRDs with deep optional schemas (External Secrets v2) and needs the CRDs at plan time.
  4. The shared module’s Helm layer for bootstrap, Argo CD after (chosen).

Consequences

  • Good: the same plan/apply workflow, state backend and scans as every other infra-* repo.
  • Good: core can be rebuilt from infra-local alone, before anything else exists.
  • Bad: two delivery paths to keep apart; a component must not be managed by both.
  • Bad: chart downloads depend on upstream registries until Zot mirrors them (ADR 0034).

ADR-0037: Private remote access through Tailscale instead of Cloudflare Tunnel and Access

Accepted 2026-10-11 · infra-local, homelab-gitops

Context

ADR 0003 planned to publish portal., grafana., argocd. and phoenix.cloudutsuk.com through a Cloudflare Tunnel guarded by Cloudflare Access. Setting up the Cloudflare Zero Trust team asks for a payment method even on the free plan, and the owner doesn’t want to give one (2026-10-11).

Without remote access, the platform is usable only from a browser on the DGX itself: every service is on the homelab Docker bridge (172.30.0.0/24), and home.cloudutsuk.com resolves only on the DGX through split DNS (ADR 0024). Other machines on the home LAN can’t reach it either.

The platform has one user (the owner, with a phone and a laptop) and no page that anonymous visitors need. Every service already has an internal HTTPS name with a Let’s Encrypt certificate (*.home.cloudutsuk.com), and Keycloak (ADR 0028) already handles sign-in for every UI.

Decision

  • Tailscale (free Personal plan) gives the owner’s devices private access; nothing is published on the internet.
  • The DGX is a subnet router. tailscaled runs on the host, not in a cluster, because the bridge is a host network. It advertises 172.30.0.0/24 (the clusters’ published addresses 172.30.0.1-.4 and the k3s API ports). The route is approved once in the admin console, and the DGX’s key expiry is switched off so the router doesn’t drop out after 180 days.
  • Split DNS. The tailnet’s DNS settings add 172.30.0.1 as a nameserver restricted to home.cloudutsuk.com, so a phone or laptop resolves grafana.mon.home.cloudutsuk.com through PowerDNS in core and gets the same Let’s Encrypt certificate as the DGX. No new names, no certificate changes.
  • Access policy (tailnet policy file). Only the owner’s identity can reach the advertised range, on 80/443 (UIs), 53 (DNS) and 6441-6444 (k3s APIs with OIDC sign-in, ADR 0028). The DGX node is tagged (tag:homelab) and can’t reach the owner’s devices. Device approval is on.
  • Sign-in stays in Keycloak. Tailscale decides who can reach the network; Keycloak decides who can use each app. The browser reaches auth.core.home.cloudutsuk.com over the tailnet, so Keycloak needs no public name (auth.cloudutsuk.com from ADR 0028 is dropped).
  • Fallback. If Tailscale’s coordination service is down, existing connections keep working; new devices wait. On the home LAN, SSH to the DGX with a port-forward still works.

  • Cloudflare stays for DNS and DNS-01 only (ADR 0024). ADR 0003’s DNS move is kept; its tunnel and Access parts are replaced by this ADR once it’s accepted.

  • As built (2026-10-11): the DGX (tag:homelab, key expiry off) advertises 172.30.0.0/24, approved by autoApprovers; split DNS home.cloudutsuk.com -> 172.30.0.1; policy in infra-local/host/tailscale/policy.hujson, route set by host/tailscale/setup.sh. Checked from a phone on mobile data: https://bao.core.home.cloudutsuk.com and https://auth.core.home.cloudutsuk.com load with valid certificates.

Considered options

  1. Cloudflare Tunnel and Access (ADR 0003): needs a payment method for the Zero Trust team.
  2. Cloudflare Tunnel without Access: would publish admin UIs (Argo CD, Grafana, OpenBao) on the internet with only the apps’ own login in front.
  3. Port forwarding on the home router plus a reverse proxy: exposes the home IP; inbound ports.
  4. Self-hosted WireGuard on the DGX: free, but needs an inbound UDP port on the router and manual key and DNS management per device.
  5. Headscale (self-hosted Tailscale control server): no vendor account, but the control server needs a public address, which brings back option 3’s problem.
  6. Tailscale free Personal plan (chosen): no payment method, no inbound port, clients for every platform, split DNS and per-identity policy built in.

Consequences

  • Good: Nothing is published on the internet; the attack surface is the owner’s devices.
  • Good: $0, no payment method, no inbound ports, no tunnel pods in platform.
  • Good: Internal names and certificates work unchanged from any device.
  • Good: kubectl with OIDC works from the laptop against every cluster.
  • Bad: Every device needs the Tailscale client; there’s no browser-only access from a borrowed machine.
  • Bad: Tailscale (the company) becomes a dependency for remote access; local access doesn’t depend on it.
  • Bad: The blog’s planned Dashboard tab (Phase 3) can link only to pages that work for tailnet devices; anonymous readers get nothing from it.
  • Bad: Plan terms can change (they did in April 2026); the subnet router counts as one tagged resource.

ADR-0038: Argo CD: installed by Terraform, everything else through an app-of-apps in homelab-gitops

Accepted 2026-10-11 · infra-local, homelab-gitops

Context

ADR 0007 puts one Argo CD in platform that manages every cluster, and ADR 0036 splits the work between the shared module’s Helm layer (bootstrap charts) and Argo CD (the rest). Step 13 had to settle how Argo CD itself is installed and seeded, how it gets its credentials, how clusters are registered, and which add-ons it owns, given that External Secrets, cert-manager and ExternalDNS already run in core from Terraform and External Secrets runs in every cluster from Terraform. The owner chose an app-of-apps in homelab-gitops over the Terraform Argo CD application layer.

Decision

  • Install: Argo CD (chart argo-cd, from Argo’s OCI registry) is a Helm release in infra-local (helm/platform/argocd). Terraform owns Argo CD itself; Argo CD doesn’t manage its own install. An upgrade is a chart version bump and a plan in infra-local.
  • Seed: a second release, helm/platform/argocd-root, creates the root Application (homelab-gitops/bootstrap, automated sync with prune and self-heal) once Argo CD’s CRDs exist. From there, bootstrap/ holds Applications for appsets/ and, later, clusters/<name>/.
  • Repository: homelab-gitops is read over SSH with a read-only deploy key. The private key is created by infra-local/host/argocd/deploy-key.sh straight into OpenBao (kv/platform/argocd-repo) and reaches Argo CD through an ExternalSecret.
  • Clusters: platform is Argo CD’s own (in-cluster). mon and ai each get an argocd-manager service account with cluster-admin and a long-lived token (helm/<cluster>/argocd-manager); host/argocd/register-cluster.sh copies the token, CA and API address (the homelab gateway) to kv/platform/argocd-clusters/<cluster>, and an ExternalSecret builds Argo CD’s cluster secret. Each cluster secret carries homelab/cluster and homelab/addons labels.
  • Add-ons: ApplicationSets with a cluster generator (homelab/addons=true) install cert-manager, ExternalDNS and a small per-cluster config chart (issuers, OpenBao-backed secrets) on platform, mon and ai. External Secrets stays in the Terraform bootstrap layer everywhere, because Argo CD itself depends on it for credentials. ExternalDNS in those clusters writes to PowerDNS in core through its HTTPS API (pdns-api.core.home.cloudutsuk.com), each with its own TXT owner id; the API key is copied per cluster (kv/<cluster>/external-dns), not into kv/shared, so future lab clusters can’t rewrite the zone.
  • core isn’t registered yet: everything in it is Terraform-owned (ADR 0036), and a cluster-admin token for the cluster that holds OpenBao and Keycloak is the largest blast radius on the platform. It’s registered when its first Argo CD app arrives.
  • Sign-in: Keycloak client argocd (public, PKCE, groups claim); platform-admins get role:admin, viewers role:readonly, anyone else nothing. The local admin account stays as break-glass (argocd-initial-admin-secret). Dex and notifications are off.
  • Polling: reconciliation every 180 s; no inbound webhook (ADR 0008).
  • As built (2026-10-11): everything tracks main. First sync: 11 Applications Synced and Healthy (root, appsets, and cert-manager, ExternalDNS and cluster-addons-config on platform, mon and ai). On the first sync the config app ran before ExternalDNS had created its namespace; the sync retry settled it within minutes, so no sync waves were added.

Considered options

  1. Terraform installs Argo CD; app-of-apps in git seeds the rest (chosen): one owner per object, and the bootstrap order (OpenBao, External Secrets, then Argo CD) stays in one tool.
  2. Argo CD manages itself after a one-off install: fewer tools touch it, but a bad self-sync can lock you out, and the credentials it needs come from releases Terraform owns anyway.
  3. The Terraform Argo CD application layer (the reference pattern’s argocd provider): the owner preferred keeping desired state in homelab-gitops, where CI writes image digests (ADR 0008).
  4. Short-lived cluster tokens from OpenBao’s Kubernetes secrets engine (ADR 0029): the target, but it needs a refresher; long-lived argocd-manager tokens are the v1 step.

Consequences

  • Good: everything after bootstrap is reviewed and changed in homelab-gitops; adding a cluster is a label plus a registration.
  • Good: no secret value lives in git or in Terraform state; deploy key, tokens and API keys sit in OpenBao.
  • Bad: the argocd-manager tokens don’t expire, and cluster-admin is broader than Argo CD needs. Rotate by recreating the secret and re-running register-cluster.sh; replace with OpenBao-issued tokens later (ADR 0029).
  • Bad: the PowerDNS API is now reachable from the homelab bridge (behind its API key and TLS).

ADR-0039: Backups with Velero: application dumps through file-system backup to Azure Blob

Accepted 2026-10-11 · infra-local, homelab-gitops, infra-azure

Context

ADR 0016 decided what to back up (every Postgres, the OpenBao raft snapshot, Zot’s config) and planned a nightly age-encrypted upload to Cloudflare R2. R2 asks for a payment method, which the owner declined (2026-10-11, the same reason as ADR 0037), and the owner asked to evaluate Velero (velero.io) instead.

What the platform actually holds (measured 2026-10-11):

  • No cluster has a PersistentVolume. Every stateful component mounts an inline hostPath under /srv/homelab (OpenBao raft, core-db, PowerDNS SQLite, Zot), so recreating a cluster keeps the data (ADR 0016).
  • A nightly set is under 1 MB today: OpenBao snapshot 57 KB, Keycloak dump about 250 KB compressed, PowerDNS under 200 KB. platform-db and ai-db will add tens to a few hundred MB per year.
  • Everything else is rebuilt from git: clusters (infra-local), apps (Argo CD), images (Zot mirror), Qdrant (re-ingest), metrics and logs.

What Velero offers and where it doesn’t fit as-is (Velero 1.18 docs):

  • It backs up Kubernetes objects to object storage and volume data through CSI snapshots or file-system backup (FSB: the node-agent DaemonSet reads volumes under /var/lib/kubelet/pods and uploads them with Kopia, deduplicated and encrypted with a repository password).
  • “hostPath volumes are not supported” by FSB, so it can’t copy any of the data above as mounted. local PVs and emptyDir are supported.
  • FSB copies live files, not a point-in-time image; a running Postgres or raft store copied that way is only crash-consistent. Velero’s answer is pre-backup hooks that run a command in a pod first.
  • Object backups of Secret objects would put plaintext secret values in the bucket.

Decision

  • Velero 1.18 (image v1.18.3, chart vmware-tanzu/velero 12.2.0, both at least 14 days old; images are multi-arch with arm64) in core, platform and ai, with the node-agent for FSB. mon holds only rebuildable telemetry and gets none.
  • It’s a bootstrap Helm release from infra-local (helm/<cluster>/velero, ADR 0036), not an Argo CD app: restoring core must not depend on platform.
  • Data goes through dumps, not raw volumes. Each cluster gets a small backup-dumps pod with an emptyDir. Velero pre-backup hooks on it run the application-consistent dumps into that emptyDir (pg_dump -Fc per database, bao operator raft snapshot save, Zot’s config and the PowerDNS zone export), and FSB uploads the emptyDir. The hostPath mounts stay as they are.
  • Objects too, minus secrets. The same nightly schedule backs up the cluster’s objects with --exclude-resources secrets (External Secrets recreates them from OpenBao). This gives a catalogue of what ran and a way to restore objects that aren’t in git.
  • Target: an Azure Blob container with the Azure plugin v1.14.3 (the release made alongside Velero 1.18.3), in a storage account of its own (not the Terraform state account), created by infra-azure with soft delete and versioning on, and a lifecycle rule to the cool tier. Credentials: a storage account key in OpenBao (kv/<cluster>/velero), delivered by External Secrets. Cost: cents a month.
  • Schedules: nightly, kept 30 days; a weekly one kept 12 weeks.
  • Recovery kit, kept offline with the OpenBao unseal keys: the storage account key and the Kopia repository password. Restoring core needs them before OpenBao exists. This replaces ADR 0016’s age key: Kopia encrypts the dumps, and OpenBao’s snapshot is already encrypted by its barrier.
  • Watching it: Velero’s Prometheus metrics go to mon (Phase 2). Alerts fire when the last successful backup per cluster is older than 26 hours, or a backup ends in error or with failed hooks. A Velero dashboard goes in the Platform folder (ADR 0027).

  • Restore test every quarter (ADR 0016) into a throwaway k3d cluster: restore the dumps with Velero, then load them (pg_restore, bao operator raft snapshot restore) and check Keycloak and a secret.

  • As built (2026-10-11), where it differs from the above:
    • Storage: infra-azure stack development/homelab-velero, account dvsthomelabbkp01 in eastus (the policy’s allowed regions), StorageV2 LRS with Cool as the account’s default tier (the shared module has no lifecycle-policy unit), versioning and 14-day soft delete, TLS 1.2, anonymous blob access off. Public network access is on: the DGX reaches it over the internet, and every request needs the key. Three soft-mandatory policy findings (private endpoint, tags on the container, a corporate app-* tag) are documented in the stack and were applied with --opa-override by the owner.
    • Releases per cluster: velero (server, node agent, CRDs) and velero-config (backup locations and schedules as raw resources in kube-system). The module’s Helm provider validates every rendered object against the API first, so Velero’s own kinds can only be applied once the CRDs exist.
    • Charts come from Zot (charts/velero, charts/velero-ui): the checked .tgz files were pushed there because Helm’s Go client timed out on GitHub’s release-asset host (ADR 0034 note).
    • Dumps only in core (backup-dumps: Keycloak’s database through a read-only pg_read_all_data role, and the OpenBao snapshot through the pod’s own Kubernetes login with a snapshot-only policy). PowerDNS’s zone and Zot’s config are left out: ExternalDNS rewrites the records and the config is in git. platform and ai hold no database yet, so they back up objects only.
    • UI: otwld/velero-ui 0.10.3 in platform (namespace velero-ui) at https://velero.platform.home.cloudutsuk.com, Keycloak sign-in only (client velero-ui; group platform-admins manages, viewers read), the chart’s own Roles instead of cluster-admin. platform’s Velero has read-only locations for core/ and ai/, so the one UI lists every cluster. The chart’s Role can read the velero namespace’s secrets, which include the storage key.
    • First run: core 714 objects plus 328 KB of dumps (both hooks OK), platform 732 objects, ai backed up; Kopia uses our password, not Velero’s default.

Considered options

  1. ADR 0016 as written (CronJob, age, R2): simplest, but R2 needs a payment method.
  2. Velero FSB on the raw volumes: not possible with hostPath. Moving to local PVs would make it possible, but copies of a running database or raft store aren’t trustworthy.
  3. Velero with dump hooks into an emptyDir (chosen): application-consistent data, Velero’s scheduling, retention, encryption, catalogue and restore workflow, and its metrics for alerting.
  4. A plain CronJob with restic or rclone to the same bucket: less machinery, but every piece (retention, encryption, status, alerts) is ours to write.

Consequences

  • Good: One tool and one restore workflow for every cluster, and later for labs with real PVCs.
  • Good: Encrypted, deduplicated, retained and monitored without custom scripts.
  • Good: No plaintext secrets in the bucket; $0 platform cost and no new payment method.
  • Bad: More moving parts than a CronJob: a node-agent per cluster, hooks, a Kopia repository.
  • Bad: Restoring data is two steps: Velero brings the dumps back, then each is loaded by hand (or by a restore hook later).
  • Bad: Hook commands run inside the dump pod, so it needs the client tools (pg_dump, bao) and short-lived credentials (a read-only Postgres role; an OpenBao token allowed only to take snapshots).
  • Bad, accepted: mon isn’t backed up. Losing the DGX disk loses up to 15 days of metrics, logs and traces; dashboards and alert rules come back from git.
  • Bad: FSB on k3d isn’t documented; the spike must prove the node-agent works there (k3d nodes do mount their root as shared, which mount propagation needs).

ADR-0040: A second, repo-scoped runner scale set for infra-local’s IaC jobs

Accepted 2026-10-11 · homelab-gitops, infra-local

Context

infra-local’s Terragrunt workflow (provisioned by the shared module, runner name from config.yaml) creates k3d clusters with the k3d CLI and installs Helm releases into every cluster. A job doing that needs the host’s Docker daemon and API access to core, platform, mon and ai. ADR 0019 keeps the build runners ephemeral and without host access, and ADR 0008 keeps kubeconfigs away from GitHub. Giving the org-wide homelab scale set Docker and cluster access would hand both to every private repo’s workflows.

The org is on GitHub’s free plan: it has only the Default runner group, so a runner group can’t limit a scale set to selected repos. ARC can, though, register a scale set with a single repository.

Decision

  • ARC in platform runs two runner scale sets from the same controller (chart 0.14.2):
    • homelab: registered with the org, used by every private repo’s build and test jobs. Ephemeral pods, no host mounts, no kubeconfig. ADR 0019 applies unchanged.
    • homelab-iac: registered with the infra-local repository only, at most one runner at a time. It’s the only runner with Docker access and with the cluster kubeconfigs (read from OpenBao). Still ephemeral, one job per pod.
  • infra-local’s config.yaml names homelab-iac as its runner (iac_runner).
  • How homelab-iac gets its access (Phase 1 step 1b), all from OpenBao kv/platform/ci-iac through one ExternalSecret that only its pods mount:
    • Docker: the pod has no Docker socket. A ghostunnel container on the homelab bridge (172.30.0.10:2376) forwards to the host’s socket only for a client certificate with CN homelab-iac (mutual TLS, its own CA). Lab clusters share the bridge, so the certificate is what opens the socket, not the network. The platform cluster isn’t recreated to mount the socket.
    • Clusters: a kubeconfig with the k3d-core, -platform, -mon and -ai admin contexts.
    • Terraform state: BACKEND_ARM_* for a service principal (homelab-iac-state) whose only role is Storage Account Contributor on the state storage account (backend_auth_alias: management).
    • Tools: k3d and kubectl, downloaded by an init container and checked against their release checksums.
  • No Azure, Docker or Kubernetes credential is stored in GitHub.
  • The Default runner group keeps “allow public repositories” off, so no public repo can use either set.

Considered options

  1. One scale set with host access for everything: every private repo’s workflow could reach the Docker daemon, which is root on the DGX.
  2. Plan-only IaC in CI, apply by hand: safe, but drift and applies stay manual.
  3. No IaC in CI: the provisioned workflow would never run.
  4. A second scale set registered with infra-local alone (chosen).

Consequences

  • Good: host-level access is limited to one repo, which only the owner pushes to.
  • Good: IaC plans and applies run from Git like everything else.
  • Bad: a compromised infra-local workflow is root on the DGX. Branch protection isn’t available for private repos on GitHub’s free plan, so the guard is who can push: only the owner has write access to infra-local, and the GitHub App never gets Contents write on it.
  • Bad: the platform cluster’s openbao ClusterSecretStore can read all of kv/platform/*, so any ExternalSecret in platform could read these credentials. Only Argo CD (from this repo) creates them today; a per-namespace store is the fix if that changes.
  • Bad: the client certificate (one year), the k3s admin certificates (one year) and the principal’s secret (one year) need rotating: re-run the infra-local host scripts.

ADR-0041: Monitoring stack on mon: monolithic Mimir, Loki and Tempo, one OTLP gateway, rules from git

Accepted 2026-10-12 · homelab-gitops, infra-local

Context

ADR 0009 puts Mimir, Loki, Tempo, Grafana and Alertmanager in mon behind an OTel gateway, with about 15 days of retention on a host path and a 6-8 GB memory budget (mon is capped at 10 GB). It doesn’t say how each piece is deployed, where the data lives, how rules reach the ruler, or how other clusters reach the gateway. The distributed charts run a dozen pods per store and need object storage; this is one host. Phase 2 also runs alongside Phase 1 (roadmap change, 2026-10-11): it uses only upstream charts through the Argo CD from Phase 0, no CI.

Decision

  • Argo CD from clusters/mon/ (child cluster-mon of the root app), namespace monitoring. Charts come through Zot’s ghcr.io mirror (ADR 0034), versions at least 14 days old (as ADR 0036 asks).
  • Monolithic everything, one replica: Mimir -target=all,alertmanager (bjw-s app-template, because mimir-distributed has no monolithic mode), Loki Monolithic mode, Tempo single binary, Grafana, and one gateway collector. Filesystem storage, 15 days retention in each store.
  • Data on the host folders /srv/homelab/mon/{mimir,loki,tempo,grafana} that infra-local mounts into the node, so it outlives the cluster. Static PVs pre-bound to each chart’s claim (class mon-hostpath, no provisioner); Mimir mounts its folder as a hostPath. The folders are owned by uid 1000, so every pod runs as 1000:1000. No CSI driver or extra storage tool: k3s’s local-path provisioner would keep data inside the node container, which a cluster rebuild deletes.
  • Rules from git: alerts/ becomes one ConfigMap mounted as the ruler’s local storage (tenant anonymous, multi-tenancy off). The ruler re-reads it every minute (ruler.poll_interval: 1m; Mimir’s default is 10 minutes); there’s no rules API to drift from.
  • Alertmanager is Mimir’s. Its fallback config is the mimir-alertmanager Secret, which External Secrets renders from kv/mon/alertmanager (infra-local host/monitoring/alerting.sh): warnings and critical alerts to a Slack channel, Watchdog every minute to a healthchecks.io check that emails when the pings stop (a dead man’s switch), severity none and info nowhere. Mimir reads it at startup. The alertmanager-sync Job (monitoring-base; after each sync and hourly) also loads it through Mimir’s config API, because Grafana reads contact points and policies only from that API, never from the fallback file; a loaded config also applies without a restart. Git and OpenBao stay the source: edits in Grafana’s UI are overwritten.
  • One OTLP entry point: every cluster sends OTLP/HTTP to https://otlp.mon.home.cloudutsuk.com (Traefik, Let’s Encrypt), inside mon also OTLP/gRPC. The gateway sets cluster from k8s.cluster.name (ADR 0026) and forwards OTLP to Mimir (/otlp, metric suffixes on), Loki (/otlp) and Tempo. A local smoke test (2026-10-12) showed vllm:* names, _total counters and job arriving unchanged, so the vendored dashboards and rules match without translation. One exception: a counter exposed without _total gains it (certmanager_controller_sync_call_count_total), so the dashboard import renames those few.
  • Grafana signs in through Keycloak (confidential client grafana, ADR 0028): platform-admins are Admin, viewers Viewer, others refused; local admin is break-glass (kv/mon/grafana). Data sources and dashboards are provisioned; dashboards are read-only in the UI.
  • The stack monitors itself (Phase 3 step 8a): ServiceMonitors in monitoring-base scrape Mimir, Loki, Tempo and Grafana as job monitoring/<service> (about 12,000 series; Grafana’s kept to what its dashboard reads), into the same Mimir. The Monitoring folder holds their upstream mixin dashboards and recording rules, pointed at the single binaries, plus an Alerts overview and an LGTM health dashboard. The mixins’ alerts aren’t used: they assume microservices. A Mimir that is down can’t report itself; the Watchdog’s dead man’s switch at healthchecks.io covers that.

Considered options

  1. lgtm-distributed / mimir-distributed with MinIO: production shape, but ~20 pods and object storage for one host; over the memory budget.
  2. grafana/otel-lgtm (all-in-one image): simplest, but one container for every store, Prometheus instead of Mimir (no Mimir ruler/Alertmanager), and not configurable enough for retention or rules from git.
  3. Monolithic per store, data on host folders, rules from git (chosen).
  4. local-path or Longhorn for volumes: dynamic, but local-path data lives in the node container and Longhorn needs several nodes to be worth it.

Consequences

  • Good: about 5 GB of limits for the whole stack; data survives k3d cluster delete mon.
  • Good: dashboards and rules change by commit only; the smoke-tested OTLP path needs no name mapping.
  • Bad: no redundancy; a mon restart is a monitoring gap (acceptable, ADR 0015).
  • Bad: the OTLP endpoint has no sender authentication yet; it’s reachable only on the homelab bridge and Tailscale. Authentication comes with the monitoring pack (Phase 2, step 3, ADR 0029).
  • Bad: filesystem storage can’t be shared; moving to object storage later means a data migration.

ADR-0042: Monitoring pack: OTel Operator collectors per cluster, Prometheus job names for the mixins, a token per cluster

Accepted 2026-10-12 · homelab-gitops, infra-local

Context

ADR 0026 decides what each cluster runs (the OpenTelemetry Operator, the Prometheus CRDs only, an agent DaemonSet and a cluster Deployment with the Target Allocator) and that every series carries cluster. It leaves open how the collectors are configured, which names the vendored mixins need (ADR 0027 selects by job), and how the gateway on mon (ADR 0041) knows who is sending. The gateway’s Ingress is reachable from every k3d cluster, labs included, so anything on the homelab bridge could write into Mimir and Loki.

Decision

  • ApplicationSets (appsets/monitoring-pack.yaml, clusters labelled homelab/addons=true) install four apps per cluster in namespace monitoring: prometheus-operator-crds, opentelemetry-operator (namespace opentelemetry-operator, webhook certificates from cert-manager), kube-state-metrics, and the local chart apps/monitoring-pack with two OpenTelemetryCollectors, both on the contrib image the gateway runs.
  • agent (DaemonSet): scrapes its own node’s kubelet and cAdvisor in Prometheus format as jobs kubelet and cadvisor, tails /var/log/pods (as root, read-only; the files are root 0640), and receives app OTLP at agent-collector.monitoring:4317/4318 with pod metadata added.
  • cluster (StatefulSet + Target Allocator; the operator refuses the allocator in deployment mode): jobs kube-state-metrics and kube-apiserver, every ServiceMonitor and PodMonitor in the cluster, and Kubernetes events as logs.
  • Names are the contract: jobs match the selectors in dashboards/mixins/*.libsonnet; every signal gets k8s.cluster.name=<cluster>, which the gateway turns into cluster. Histograms no dashboard or rule reads are dropped at scrape time.
  • A bearer token per cluster: infra-local host/monitoring/configure.sh seeds kv/mon/otlp-gateway (one key per cluster) and copies each into kv/<cluster>/otlp, the only path that cluster’s store can read. The gateway accepts that list of tokens on OTLP/gRPC and OTLP/HTTP and refuses anything else (401). A lab gets its own token when it registers; revoking one cluster is deleting its key.

  • mon sends to the gateway’s Service; the others to https://otlp.mon.home.cloudutsuk.com.

  • Labs as built (2026-10-14, lab contract v2): labs share one labs token instead of one each. They have no External Secrets, and they come and go too often for a token per lab to be worth managing; infra-local host/labs/lab.sh register copies it into the lab, and rotating it means re-registering the running labs. Labs are selected by homelab/addons=monitoring and get only cert-manager and the monitoring pack (no issuers, policies or ExternalDNS), with the operator’s NetworkPolicies off.

Considered options

  1. The opentelemetry-collector chart twice, no operator: fewer moving parts, but no Target Allocator, so ServiceMonitors that charts ship would be ignored (ADR 0026 wants them).
  2. kube-prometheus-stack in each cluster with remote write: the mixins’ native home, but a Prometheus per cluster (≈1 GB each) and a second telemetry path next to OTLP.
  3. Operator collectors with Prometheus job names, per-cluster tokens (chosen).
  4. mTLS from each cluster: stronger, but needs a CA and certificate rotation per cluster; a token in OpenBao is enough on a private bridge.

Consequences

  • Good: the vendored mixin dashboards and rules work unchanged; the drop-down lists every cluster.
  • Good: an unknown sender gets 401; a leaked token is one key to rotate.
  • Bad: the agent runs as root to read pod logs.
  • Bad: about 0.5 GB of requests per cluster (operator, kube-state-metrics, two collectors, allocator).
  • Bad: tokens are static until rotated by re-seeding (no expiry).

ADR-0043: CI build path: shared rootless BuildKit, OpenBao roles bound to the reusable workflow, digest commits by a GitHub App

Accepted 2026-10-12 · homelab-gitops, infra-local, homelab-services

Context

ADR 0008 says main builds push to Zot with rootless BuildKit and commit the new digest to homelab-gitops with a short-lived GitHub App token. ADR 0019 and 0034 say CI gets its Zot key from OpenBao through GitHub OIDC, bound to main, and that pull requests never push. Phase 1 steps 1c, 2 and 2a had to decide where BuildKit runs, what exactly an OpenBao role trusts, and how the digest reaches homelab-gitops. Every private repo in the org can run jobs on the homelab scale set, so “bound to main” alone would let any workflow on any repo’s main read the push key.

Decision

  • BuildKit. One rootless daemon, moby/buildkit:v0.33.0-rootless, in platform (apps/buildkit, namespace buildkit). Jobs reach it with docker buildx and the remote driver; the runner image already ships the Docker CLI and buildx, and no Docker daemon is involved. Its gRPC API has no authentication, so a NetworkPolicy admits only pods in arc-runners. Registry credentials stay in the job: buildx forwards them from the runner’s Docker config for each build. The cache is an emptyDir (25 GiB, garbage-collected at 20 GB).
  • Rootless needs two exceptions. Seccomp and AppArmor are Unconfined for the daemon’s user namespace, and --oci-worker-no-process-sandbox runs build steps without their own PID namespace, because k3d’s nested containers can’t mount a new /proc. That’s the buildkitd-rootless exception to Kyverno’s pod-security-restricted (ADR 0033). The root filesystem is read-only (2026-10-13): rootlesskit’s state, /run/user/1000 and /tmp are emptyDirs.
  • OpenBao sign-in. JWT auth at github-actions/ trusts GitHub’s OIDC issuer. Every role requires audience https://bao.core.home.cloudutsuk.com, the org’s numeric repository_owner_id (a deleted org’s name can be taken again) and runner_environment: self-hosted.
    • main (policy ci-main: kv/ci/zot-push, kv/ci/zot-pull, kv/ci/gitops-writer) also needs ref: refs/heads/main and job_workflow_ref equal to cloudutsukpilot/homelab-gitops/.github/workflows/build-image.yaml@refs/heads/main. Only that workflow’s own steps get the push key and the writer App, whichever repo calls it. A caller pinned to a SHA or branch, a pull request, or a hand-written workflow is refused.
    • pr (policy ci-pr: kv/ci/zot-pull) is any self-hosted job in the org.
    • Tokens last 15 minutes and each job revokes its token as soon as it has read the secret.
  • kv/ci/ is a new top-level area that no cluster’s External Secrets store can read. The Zot passwords are copied there from kv/core/zot (infra-local/host/openbao/configure-ci.sh) so the CI policies never see the admin password; rotating a Zot password means re-running that script.

  • Digest commit. A second GitHub App, cloudutsuk-gitops-writer, with Contents read/write and installed on homelab-gitops only. Its key lives in kv/ci/gitops-writer. The workflow’s promote job signs the App JWT with openssl, asks for an installation token limited to homelab-gitops and contents: write, and runs kustomize edit set image in the caller’s gitops-path. It pushes to main with up to five rebase retries, then revokes the token. Images are referenced by digest; the :<git-sha> tag is only for people.

  • Build path update (2026-10-12, Phase 1 step 4): BuildKit no longer pushes. buildx exports an OCI layout to the runner (with BuildKit’s provenance attestation, mode=min, until step 5 dropped it: a provenance index makes Kyverno look for a linux/amd64 image; ADR 0033). The gates scan it there, and on main crane pushes that layout. The job checks that the pushed digest is BuildKit’s. The tools come from the runner image (ADR 0044).

Considered options

  1. BuildKit as a sidecar in every runner pod: no shared cache, and the Unconfined exceptions would cover every runner instead of one daemon.
  2. Kaniko: archived by Google in 2025; only a third-party fork is maintained.
  3. mTLS between buildx and BuildKit: the NetworkPolicy gives the same boundary without a certificate to rotate; worth adding if anything else ever runs in arc-runners.
  4. Bind the push role to main of selected repos: every new service repo would need an OpenBao change, and any workflow on those repos’ main could still read the key.
  5. actions/create-github-app-token: needs the private key as an action input. The job’s openssl call keeps the key in a temporary file that is deleted right away.
  6. Shared rootless BuildKit, roles bound to the reusable workflow, writer App (chosen).

Consequences

  • Good: No secrets in GitHub. Pull requests can’t push, and other workflows can’t push either.
  • Good: Every repo gets the same build, push and deploy path from a five-line caller.
  • Good: The shared cache makes rebuilds fast.
  • Bad: One BuildKit daemon is a shared component. A build step could in principle reach the daemon’s processes (no process sandbox), and every org repo’s builds run there.
  • Bad: Callers must use @main. A breaking change to build-image.yaml reaches every repo at once.
  • Bad: Two copies of the Zot CI passwords, kept in step by a script.

ADR-0044: Own runner image with the CI tools baked in, rolled out by hand

Accepted 2026-10-12 · homelab-gitops, homelab-services

Context

Phase 1 step 4 put ADR 0032’s gates into build-image.yaml: Gitleaks, Semgrep, Bandit, Trivy, Checkov, Conftest and Hadolint, plus Syft, crane and Cosign for the image. The stock actions-runner image has none of them and no pip. Installing them in each job meant about 400 MB from GitHub releases and PyPI. GitHub’s release CDN served some connections at about 100 KB/s, so the install step took 2 to 7.5 minutes while every scan together took about 30 seconds. Runner pods are ephemeral, so nothing carries over between jobs.

Decision

  • The tool set lives in homelab-gitops/ci/tools. tools.txt lists each binary’s URL and sha256; requirements/*.txt are hash-locked Python requirements, one per scanner; install.sh installs both onto <prefix>/bin and writes a digest of the set to <prefix>/MANIFEST. Every version is at least two weeks old.
  • The homelab scale set runs our own image, platform/actions-runner: the stock actions-runner image (pinned by digest) with install.sh run at build time into /opt/ci-tools (ci/runner-image). runner-image.yml builds it with build-image.yaml like any service, so it passes the same gates and is signed with an SBOM attestation. The pods pull it from Zot with a zot-pull secret from kv/shared/zot-pull.
  • The security-tools action keeps working without it. It compares /opt/ci-tools/MANIFEST with the digest of the ci/tools it was loaded with. On a match it uses the baked tools; otherwise it runs install.sh in the job. A change to ci/tools works at once and gets fast again after the next image rollout.
  • Promotion is by hand. No bot commit puts the new digest in arc-runner-homelab.yaml, because a broken runner image would stop the CI that could fix it. Someone checks the signature, sets the digest and runs one build on it (ci/runner-image/README.md).
  • The image gate only reports for this image (image-gate: report). The stock runner bundles Node.js packages, runc and the Docker CLI, and the newest Gitleaks, Syft, Cosign and other tool releases are built with Go versions that have known CVEs. On 2026-10-12 that was 255 HIGH or CRITICAL findings with a fix that only upstream releases can apply. They stay in the run’s summary and reports. Every other image keeps image-gate: enforce.
  • homelab-iac keeps the stock image: its jobs don’t run the gates.

Considered options

  1. Install in every job (the first version): minutes per job, and a slow CDN connection makes builds fail or crawl.
  2. actions/cache: still downloads from GitHub’s cache service each job, and a cache miss is the slow path again.
  3. A tools artifact in Zot, pulled per job: fast, but a second artifact to build, push and pin next to the image the runner already pulls.
  4. Own runner image with the tools baked in, manual rollout (chosen).

Consequences

  • Good: Gate jobs start in seconds: the tools step went from 2 to 7.5 minutes to under a second, and a full hello-world build on main, gates, push, signing and deploy commit included, takes about 70 s.
  • Good: The runner image goes through the same build, scan and signing path as everything else.
  • Bad: The runner image is about 1 GB bigger (2.4 GB), and each new runner pod on a fresh node pulls it.
  • Bad: A new GitHub runner release or tool bump means a rebuild and a manual rollout. GitHub stops accepting old runner versions after a while, so the base image needs regular bumps.
  • Bad: For this image, fixable CVEs in upstream components don’t block the build.

ADR-0045: Lab UIs by name through one labs gateway on the host

Accepted 2026-10-15 · infra-local, sakharams.github.io

Context

Labs (lab contract v2) publish their UIs on 127.0.0.1 host ports, mapped to NodePorts. That keeps default logins off the network and needs no DNS or certificates, but the links only work on the DGX itself: from a laptop on the tailnet, the portal’s and the link page’s lab links are dead, while every long-lived cluster’s UI works by name. Labs come and go many times a day, often with a different node address each time, and labs that replace Traefik (the eBPF lab runs Cilium without it) have no ingress to hang a name on.

Decision

  • One gateway for all labs: Traefik on the host (infra-local host/labs-gateway), on the homelab Docker network and on its own bridge address, 172.30.0.5, which Tailscale routes like the clusters’ .2-.4 (ADR 0030, 0037).
  • One DNS record: *.labs.home.cloudutsuk.com -> 172.30.0.5 in PowerDNS, created once. A DNS wildcard answers for every name below it, so registering a lab never touches DNS.
  • Names from the cluster: infra-local host/labs/lab.sh register writes one route file per lab: <service>.<lab>.labs.home.cloudutsuk.com for each Service the lab publishes as a NodePort (to its server node on the homelab network), and <lab>.labs.home.cloudutsuk.com for the lab’s own ingress when its load balancer proxies port 80. <lab> is the cluster name without lab-. unregister removes the file; Traefik watches the folder.
  • Certificates: Let’s Encrypt through a Cloudflare DNS-01 challenge with the token cert-manager already uses, one certificate per lab for <lab>.labs… and *.<lab>.labs… (a wildcard covers one label), kept in the gateway’s state and reused when the lab comes back.
  • Labs keep their 127.0.0.1 ports for use on the DGX; nothing in a lab’s config changes.

Considered options

  1. A bridge address, DNS records and a certificate per lab: the long-lived clusters’ pattern. Labs would need ExternalDNS and an issuer (the monitoring-only add-on set gives them neither), and a host address per lab, which needs root each time.
  2. Publish lab ports on a bridge address instead of 127.0.0.1: reachable, but by address and port, without names or TLS.
  3. One host gateway with a wildcard record (chosen): no per-lab DNS, root or cluster add-ons; names follow Services, so a lab gets them without declaring anything.

Consequences

  • Good: every lab UI opens by name with a valid certificate from any tailnet device, and the portal and link page can link to it.
  • Good: labs need no new add-ons, records or root access; a rebuilt lab gets its routes back from make up.
  • Bad: lab UIs with default logins (the Falcosidekick UI) become reachable from the tailnet, not only the DGX; they need real logins or a sign-in in front.
  • Bad: one more host container to run and monitor; if it’s down, labs still work on localhost.
  • Bad: each new lab name is a Let’s Encrypt order (a few seconds, well inside the rate limits).

ADR-0046: First chat model: gpt-oss-20b (Apache 2.0) in a 0.17 slice of the GB10

Superseded by ADR-0049 2026-10-15 · homelab-gitops

Context

ADR 0010 left the first chat model and its licence to Phase 5, from two candidates: gpt-oss-20b and Qwen3-30B-A3B, served by vLLM 0.24 (nvcr.io/nvidia/vllm:26.07-py3) in the ai cluster, next to the embeddings model (0.1 slice, about 12 GB). Every model’s memory comes out of the DGX’s one 121 GB unified pool, which the desktop, the other clusters and the labs share. ADR 0010 planned for about 81 GB free; on 2026-10-15 the host had 39 GB available (labs running, a long-lived Jekyll server holding 22 GB), so the memory a model needs decides more than its benchmark scores.

The callers are the RAG chat (answers grounded in retrieved posts, with citations) and, in Phase 6, the SRE agent, which needs tool calls.

Decision

  • gpt-oss-20b (OpenAI, Apache 2.0, 21B parameters with 3.6B active per token, MoE). It ships in MXFP4, about 13 GB of weights, and vLLM 0.24 runs it on the GB10 (compute capability 12.1) with its Marlin MXFP4 MoE kernel and Triton attention. A restart with the weights cached is ready in about 2.5 minutes.
  • Its tokenizer (Harmony) needs two vocab files, which the image would download on the first request into a cache under /root that the non-root pod can’t use; an init container puts them on the models claim (lesson L069).
  • --gpu-memory-utilization 0.17 (about 21 GB), not ADR 0010’s 0.25 start. Measured on 2026-10-15 with --max-model-len 32768: weights 13.8 GiB, CUDA graphs about 1 GiB, and a KV cache of 2.78 GiB (113,900 tokens), which is 3.5 requests at the full 32k context and about 15 RAG-sized (8k) ones at a time. At 0.15 the cache was 1.53 GiB (62,581 tokens, fewer than two full requests). At 0.2 vLLM reported 15.36 GiB, far more than the platform needs. Raise the slice only when the AI / vLLM dashboard shows the KV cache full or requests preempted under real traffic, and the host has the room.
  • Served as gpt-oss-20b behind LiteLLM (ADR 0022), with OpenAI-format tool calls (--tool-call-parser openai) and the reasoning split off into reasoning_content (--reasoning-parser openai_gptoss), so answers carry only the answer.
  • Licence: Apache 2.0 allows use, change and redistribution, including commercially; the only duty that applies here is keeping the licence and notices if the weights are passed on, which this platform doesn’t.

Considered options

  1. Qwen3-30B-A3B in FP8 (Apache 2.0): stronger on some reasoning benchmarks, but about 31 GB of weights alone, so a 0.3 to 0.35 slice before any KV cache: more than the host had free.
  2. Qwen3-30B-A3B in a 4-bit quantisation (AWQ or GPTQ): fits, but a community quantisation, untested on this vLLM build and GPU.
  3. gpt-oss-20b (chosen): the smallest memory need of the candidates, an official quantisation from the model’s authors, native tool calling, and it runs on this vLLM build and GPU (tested 2026-10-15).

Consequences

  • Good: about 21 GB for the chat model, leaving room for the rest of Phase 5 at today’s free memory.
  • Good: tool calls are ready for the SRE agent without a second model.
  • Bad: it reasons before it answers, which costs tokens and time on simple questions; reasoning_effort (low, medium, high) per request is the control, and rag-api should send low by default.
  • Bad: 32k tokens of context per request, less than the model’s 128k, to keep the slice small. Raised to 64k on 2026-10-16 for Hermes Agent (ADR 0048) at the same 0.17 slice: that start’s KV cache held 95,834 tokens, 1.46 full-length requests (it varies by start; 173,602 had been seen).
  • Bad: its knowledge is general; everything platform-specific has to come from retrieval.

ADR-0047: Argo CD application groups: one AppProject per area, mirrored as the label homelab/group

Accepted 2026-10-15 · homelab-gitops, sakharams.github.io

Context

Argo CD manages about 60 Applications across platform, mon, ai and every registered lab, all in the default project and with no grouping label. Its UI lists them as one flat set: finding “everything that makes up monitoring” or “what a lab got” means reading names. The default project also lets any Application deploy from any repository to any cluster.

Upstream Argo CD (v3.5.3 here) has no grouped dashboard view. Codefresh’s GitOps dashboard has one (Application Groups, the codefresh.io/app-group annotation), but only in Codefresh’s hosted UI, with a Codefresh account, a runtime that keeps a tunnel open to the SaaS, a non-expiring Argo CD admin token and a paid plan after a trial. That conflicts with ADR 0023 (short-lived credentials from OpenBao), ADR 0037 (nothing reachable from outside) and the $0 hosting goal.

Decision

  • Five groups, each an AppProject in bootstrap/projects.yaml, synced by the root Application in wave -1: platform (portal, its database, CI, hello-world, ExternalDNS), monitoring (the mon stack and the monitoring pack), security (Kyverno and its policies, cert-manager, issuers and add-on secrets), ai (the ai cluster’s apps) and labs (everything Argo CD installs in a lab cluster).
  • Each project is also a guard rail: it allows only the sources its apps use (this repo, Zot’s mirror and the few public chart repos still used directly) and only the clusters they deploy to. ai reaches only the ai cluster; labs reaches only lab API servers (https://172.30.0.*:6443, with mon’s 172.30.0.1:6443 denied). Cluster-scoped resources stay allowed: the add-ons ship CRDs and RBAC.
  • The same name as a label, homelab/group, on every Application, next to the existing homelab/addon. The cert-manager and monitoring-pack ApplicationSets compute both per cluster: labs for clusters registered with homelab/addons=monitoring, otherwise security or monitoring.
  • The app-of-apps stay in default (root, appsets, cluster-platform, cluster-mon, cluster-ai), labelled platform: the root comes from infra-local, and they only create Applications.
  • Grafana: the vendored Argo CD dashboard gets a Group variable and a Groups row (health and out-of-sync apps per group on one timeline) through argocd_fixups in scripts/dashboards.py, keyed on the metrics’ project label, so no Argo CD metrics setting changes. Sync and reconcile activity default to per group.
  • The portal: the Argo CD view in Backstage (Phase 3 step 6) shows the groups as tabs.

Considered options

  1. Codefresh GitOps Runtime on our Argo CD (Application Groups): the grouped view exists only in Codefresh’s SaaS UI, and the runtime needs an account, an outbound tunnel, a non-expiring admin token and a paid plan. Rejected.
  2. Labels only: the UI’s label filter works, but every app keeps default’s unlimited sources and destinations, and the dashboard would need applicationLabels metrics enabled in infra-local.
  3. AppProjects only: the UI’s project filter and the project metric label are enough for grouping, but a label is what the portal and kubectl -l select on.
  4. AppProjects mirrored as a label (chosen): one name in both places; the project adds the guard rail.

Consequences

  • Good: the Argo CD UI filters by project or label (?proj=monitoring, ?labels=homelab/group=labs) and each view can be bookmarked; Grafana shows sync and health per group; a mistake in an ai app can’t deploy to platform or mon.
  • Good: no new component, credential or outbound connection.
  • Bad: a new app must name its project and label, and a new chart repository must be added to its project’s sourceRepos, or Argo CD refuses to sync it (the app shows an InvalidSpecError condition).
  • Bad: lab destinations are matched by address; a lab whose API isn’t on 172.30.0.*:6443 needs a rule.
  • ADR 0026, ADR 0027, ADR 0038, ADR 0042

ADR-0048: Hermes Agent on platform: non-root without s6, the local model at 64k, dashboard for platform-admins only

Accepted 2026-10-16 · homelab-gitops, infra-local, sakharams.github.io

Context

Hermes Agent (Nous Research, MIT) is a self-improving agent: it learns skills from use, keeps memories across sessions, runs cron jobs and can reach messaging platforms. It ships one Docker image (nousresearch/hermes-agent, amd64 and arm64, about 900 MB) with a gateway (the agent, the scheduler and an OpenAI-compatible API on 8642) and a web dashboard (9119). All state lives in /opt/data, including a SQLite database.

Running it on platform meets these constraints:

  • Fast releases: a release comes out almost daily. The newest at least two weeks old on 2026-10-16 is v0.21.5 (tag v2026.9.24, 2026-09-24).
  • Startup vs. Pod Security: the image starts as root under s6-overlay (/init as PID 1), remaps its user, chowns the volume, then drops to uid 10000. platform enforces restricted Pod Security and a read-only root (ADR 0033). As uid 10000, s6 refuses to start because /run isn’t its own.
  • Context window: Hermes refuses models with less than 64,000 tokens of context. gpt-oss-20b was served at 32k (ADR 0046).
  • Dashboard access: the dashboard reads and writes API keys and runs commands (a terminal-backed chat and maintenance actions). Its OIDC sign-in needs a public PKCE client and admits whoever the issuer signs in, with no group check.
  • Storage: a host folder for /opt/data would need a new k3d node mount, which only re-creating the cluster adds.

Decision

  • Image: docker.io/nousresearch/hermes-agent:v2026.9.24, pinned by digest and pulled through Zot’s docker.io mirror.
  • Pod: one replica (single writer on SQLite), Recreate, in namespace hermes (apps/hermes, project platform).
    • Security: uid 10000 from the start, read-only root, all capabilities dropped.
    • PID 1: shareProcessNamespace makes the pause container PID 1 (it reaps zombies), so the image’s entrypoint takes its documented non-PID-1 path, the stage2 setup without s6.
    • Containers: an init container runs that setup, then writes the settings git owns into config.yaml. Two containers then run what s6 would supervise: hermes gateway run and hermes dashboard, sharing /opt/data. /run, /tmp and /dev/shm are emptyDirs.
  • Model: gpt-oss-20b (Qwen3.5-35B-A3B at 128k since 2026-10-16, ADR 0049) through LiteLLM (https://llm.ai.home.cloudutsuk.com/v1) with Hermes’ own virtual key. vllm-chat goes to --max-model-len 65536 at the same 0.17 slice. Its KV cache varies by start: the first 64k start had 95,834 tokens, 1.46 full-length requests.
  • Dashboard: https://hermes.platform.home.cloudutsuk.com, tailnet only, with a Let’s Encrypt certificate.
    • Sign-in: Keycloak client hermes (public, PKCE). Its browser flow, browser platform-admins only (infra-local host/keycloak/configure-realm.sh), denies users without realm role platform-admin, which the group platform-admins carries.
    • Network: a NetworkPolicy lets only Traefik reach the dashboard. The dashboard trusts X-Forwarded-* only from the pod network (10.20.0.0/24).
  • API: the OpenAI-compatible API stays in the cluster (hermes.hermes.svc:8642, bearer key), reachable only from pods in hermes until a caller needs it.
  • Storage: /opt/data is a 10Gi local-path claim, backed up nightly by Velero’s file-system backup.
  • Secrets: both keys come from OpenBao kv/platform/hermes.

Considered options

  1. The image’s own init as root, with a Kyverno exception: the supported path. It would need root, a writable root filesystem and chown, the exact exceptions ADR 0033 exists to avoid, for a process that runs model-written code.
  2. Non-root without s6, the pause container as PID 1 (chosen): every policy holds. The cost is that nothing supervises inside the pod: Kubernetes restarts a failed container instead.
  3. A paid model (Anthropic, OpenRouter): better agent quality, but costs per token and sends prompts out of the homelab. The local model is free and private, and enough to start.
  4. Password sign-in, or OIDC for every realm user: simpler, but the dashboard is a command console. A password isn’t SSO, and “any realm user” includes everyone Entra can sign in.

Consequences

  • Good: Hermes runs under the same policies as every other app, with its UI behind the platform’s SSO.
  • Good: the model stays on the DGX at no cost; other callers also get 64k context.
  • Bad: at the 0.17 slice, only about one full 64k request fits in the KV cache at a time; long Hermes sessions and other callers then queue or are preempted. Raise the slice if the AI / vLLM dashboard shows it.
  • Bad: the dashboard changes settings in config.yaml. The keys the init container sets (model, proxy, public URL) revert on restart, so they change in git.
  • Bad: local-path keeps the data inside the node container. Re-creating platform needs a Velero restore of the namespace.
  • Bad (resolved by ADR 0049): gpt-oss-20b at 64k is a small model for an agent. Long sessions compress often, and a stronger model is a later decision.
  • Bad: the release cadence is fast, and the two-week rule keeps Hermes behind upstream on purpose.
  • ADR 0010, ADR 0022, ADR 0046 (the model and its gateway)
  • ADR 0028 (Keycloak), ADR 0033 (Kyverno policies), ADR 0039 (Velero)

ADR-0049: Chat model: Qwen3.5-35B-A3B (FP8, Apache 2.0) replaces gpt-oss-20b

Accepted 2026-10-16 · homelab-gitops, sakharams.github.io

Context

ADR 0046 chose gpt-oss-20b as the first chat model, in a 0.17 slice (about 21 GB) because the DGX had little free memory that day. Its first real agent workload showed its limits:

  • Broken tool-call output: Hermes Agent’s task t_8d6df5e7 crashed twice. In the middle of a tool call, the model wrote reasoning text where a function name belongs (to=functions? very? we need to). vLLM’s parser for gpt-oss’s output format (Harmony) rejected it, in streaming and non-streaming alike, so no client-side setting avoids it.
  • No way to turn thinking off: Harmony refuses reasoning_effort: none, so every Hermes session title cost a 400 and a retry.
  • Context: 64k at that slice held about 1.5 full-length requests.
  • Who depends on it: agents are the main users of the chat model (Hermes now, the SRE agent in Phase 6), so tool calling and context matter more than the smallest memory footprint.

On 2026-10-16 the host had about 56 GB available with no chat model loaded. ADR 0010’s other candidate family, Qwen, has a newer generation: Qwen3.5 (February 2026).

Decision

  • Model: Qwen/Qwen3.5-35B-A3B-FP8.
    • Apache 2.0, pinned to revision 9d1823d2 (2026-04-24, well over two weeks old).
    • A mixture of experts: 35B parameters, 3B active per token, so it decodes about as fast as gpt-oss (3.6B active) on the GB10’s memory bandwidth.
    • Linear attention in three of every four layers, so the KV cache is about 20 KB per token.
  • Same app: it runs in apps/vllm-chat, which keeps its role name. The Service, alerts, dashboards and LiteLLM’s api_base don’t change.
    • vLLM 0.24 (nvcr.io/nvidia/vllm:26.07), --language-model-only (no vision encoder).
    • --gpu-memory-utilization 0.36 (about 44 GB) and --max-model-len 131072, the model card’s floor for its thinking mode.
    • Tool calls with the qwen3_coder parser; thinking split into reasoning_content with qwen3.
  • Gateway: the model is published as qwen3.5-35b-a3b. The gpt-oss-20b model name is removed, and callers’ virtual keys move to the new name.
  • Hermes Agent: it follows at 128k context (apps/hermes).
  • Fallback: gpt-oss-20b’s weights stay on the models claim, so going back is a revert of the model commits.

Considered options

  1. Keep gpt-oss-20b and work around it: a bigger slice wouldn’t fix malformed Harmony output, and its context is 128k at most.
  2. Qwen3-8B (dense, FP8): small and cheap. A first attempt crash-looped: its native context is 40,960 tokens, under Hermes’ 64k minimum, so it needs YaRN scaling. It’s also weaker at agent work than a 35B MoE.
  3. Qwen3-30B-A3B-Instruct-2507: the generation ADR 0010 named; similar size, older, 256k context.
  4. Qwen3.5-35B-A3B in FP8 (chosen): the newest small MoE in the family, 262k context, built for tool calling, and a small KV cache.
  5. Two models, switched on demand (KEDA or a model-swapping proxy): both don’t fit in memory at once, and KEDA can’t keep two deployments mutually exclusive. The first request after a switch would wait for a 2.5 to 5 minute start.
  6. Larger models from the 2026 field (Llama 4 Scout, DeepSeek V3.2): 55 GB to 350 GB or more, far over the memory available.

Consequences

  • Good: tool calls in Qwen’s format, a thinking switch (chat_template_kwargs.enable_thinking), 128k context at a small KV cost, and the same Apache 2.0 licence.
  • Good: the role-named app made the swap a change of model inside vllm-chat, not a new app.
  • Bad: about 44 GB instead of 21 GB of unified memory. Host headroom drops by about 23 GB, which affects how many labs fit (the lab runner’s admission limit).
  • Bad: a 37.5 GB first download (about 10 MB/s here, so about an hour). The startup probe allows 90 minutes.
  • Bad: callers must change the model name and their keys’ model lists. That was a one-time cost with only Hermes as a caller.
  • ADR 0010 (vLLM on the GB10), ADR 0022 (the gateway), ADR 0046 (superseded), ADR 0048 (Hermes Agent)
projects, cloudutsuk-platform
Edit this post
Projects Cloudutsuk Platform Architecture ADR
This post is licensed under CC BY 4.0 by the author.