DGX Spark to Azure Private Endpoint: Deploy
Bring up the Azure side, the local probe and the DGX's WireGuard config, check each piece, and fix the failures I hit along the way.
Three commands deploy the lab. One builds the Azure side, one starts the local probe and Grafana, and one installs WireGuard on the DGX without starting it. By the end of this part you’ll have checked each piece on its own and confirmed the starting point for the experiments: a tunnel ready to go, and a Foundry account that answers 403 to the DGX.
Bring It Up
1. Azure. From the lab repo, with az login done and the right subscription selected:
1
make azure-up
Output from my run (warnings trimmed):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
bash infra/up.sh
location=eastus rg=rg-pe-lab account=pelab-3d07e659 your-ip=103.197.74.28
==> 1/4 Resource group and VNet
==> 2/4 Foundry (AIServices) account, public network access disabled
==> 3/4 Private endpoint and the cognitiveservices private DNS zone
==> 4/4 NSG, public IP and the WireGuard VM
account: pelab-3d07e659
private endpoint: 10.60.1.4,10.60.1.5,10.60.1.6
vm public ip: 20.102.50.119
written: .../private-connectivity-labs/azure-private-endpoint/.state/lab.env
vm: cloud-init done
Most of the time goes on the Foundry account, which sat in Accepted for several minutes, and the VM’s first boot. azure-wait polls the VM over SSH until cloud-init has touched /var/lib/pe-lab-ready, so when the target returns, WireGuard and dnsmasq are already running. up.sh is safe to re-run. Every resource is checked with show before it’s created, which is also how you update the NSG when your home IP changes.
2. The probe and Grafana, locally on the DGX:
1
make probe-up
The first run pulls grafana/otel-lgtm:0.35.0 and builds pe-probe:dev. Compose waits for LGTM’s health check before starting the probe, so its first exports aren’t lost. Grafana is at http://localhost:3012/d/azure-pe-lab/.
3. WireGuard on the DGX, which needs sudo:
1
make dgx-install
1
2
3
4
dgx-install: ## Install wireguard-tools and the tunnel-only /etc/wireguard/wg0.conf
sudo apt-get install -y wireguard-tools
bash dgx/render.sh
sudo install -m 600 -D .state/wg0.conf /etc/wireguard/wg0.conf
This installs wireguard-tools (1.0.20210914 on Ubuntu 24.04; the WireGuard module itself is already in the DGX’s 6.17 kernel) and the tunnel-only config. It does not start the tunnel. That’s Experiment 2.
Verify the Rollout
make status is the preflight the screenshot run uses. It fails unless lab.env exists, Grafana answers and the probe container is running:
1
2
3
4
$ make status
grafana: ok
probe: running
account: pelab-3d07e659 private endpoint: 10.60.1.4 vm: 20.102.50.119
Then check the VM end. make vm-ssh logs in with the lab’s own key, from your IP only:
1
2
3
4
5
6
7
8
9
10
11
$ make vm-ssh
azureuser@vm-wg-pe-lab:~$ sudo wg show wg0
interface: wg0
public key: OWcvAw0eW83wZR08sN8GT7F2ZAGdSqTnOqGApKFzimE=
private key: (hidden)
listening port: 51820
peer: 50LWN2QM2vNOusA7eqHVZU1Otuge7tZr9+hXt65mhRI=
allowed ips: 10.200.0.2/32
azureuser@vm-wg-pe-lab:~$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
No latest handshake line yet. The DGX hasn’t connected.
The most useful check is asking dnsmasq on the VM for each name, which shows what the DGX will get once split DNS is on. From my run, before experiment 4:
1
2
3
4
5
6
7
8
9
10
11
12
azureuser@vm-wg-pe-lab:~$ dig +short @10.200.0.1 pelab-3d07e659.cognitiveservices.azure.com
pelab-3d07e659.privatelink.cognitiveservices.azure.com.
10.60.1.4
azureuser@vm-wg-pe-lab:~$ dig +short @10.200.0.1 pelab-3d07e659.openai.azure.com
pelab-3d07e659.privatelink.openai.azure.com.
eastus.api.cognitive.microsoft.com.
cognitiveuseprod.trafficmanager.net.
cognitiveuseprod.azure-api.net.
apimgmttmsxrvtifzqdtqhxmfmfanqbnjviisx1axz0alfmmew.trafficmanager.net.
cognitiveuseprod-eastus-01.regional.azure-api.net.
apid69e4f0cd7984aa8a89d9d59fb140df1wjcqcmgrfkzhbxquwhytr.eastus.cloudapp.azure.com.
20.232.91.180
The cognitiveservices name stops at the private zone. The openai name goes through the same privatelink CNAME, finds no linked zone of that name, and carries on to a public address. That’s the configuration mistake the lab starts with.
Smoke Test
From inside the VNet, the private path already works. On the VM, with the API key from .state/lab.env:
1
2
3
4
5
6
$ curl -s -o /dev/null -w '%{http_code} %{remote_ip}\n' -H "api-key: $API_KEY" \
"https://pelab-3d07e659.cognitiveservices.azure.com/openai/models?api-version=2024-10-21"
200 10.60.1.4
$ curl ... "https://pelab-3d07e659.openai.azure.com/openai/models?api-version=2024-10-21"
403 20.232.91.180
{"error":{"code":"403","message": "Public access is disabled. Please configure private endpoint."}}
From the DGX, everything is public.
1
2
3
4
5
$ make dgx-status
wg0: not present
pelab-3d07e659.cognitiveservices.azure.com 20.232.91.180
pelab-3d07e659.openai.azure.com 20.232.91.180
pelab-3d07e659.services.ai.azure.com 20.232.91.180
All three names resolve to the same public front end, and all three calls return 403. The probe log says the same. That’s the baseline for Experiment 1.
Once the tunnel is up, one curl shows that routing works before any DNS change, by pinning the name to its private IP:
1
2
3
4
$ curl -s -o /dev/null -w '%{http_code} via %{remote_ip} in %{time_total}s\n' \
--resolve pelab-3d07e659.openai.azure.com:443:10.60.1.5 -H "api-key: $API_KEY" \
"https://pelab-3d07e659.openai.azure.com/openai/models?api-version=2024-10-21"
200 via 10.60.1.5 in 3.307212s
--resolve skips DNS, so this works even while the openai zone is missing. Your SDK has no such flag, and that’s what the rest of the lab is for. The 3.3 seconds is one fresh connection: about 211 ms round trip from my DGX to East US (ping 10.200.0.1), times a TCP handshake, a TLS handshake and the request, plus the server’s own time to list the models.
Common Failures
| Symptom | Cause | Fix |
|---|---|---|
az vm create dies with RuntimeError: The content for this response was already consumed | The CLI hides the real error. Mine was QuotaExceeded ... standardBpsv2Family ... Current Limit: 0 | Re-run with --validate --debug, find the inner error; pick a size family with quota (az vm list-usage -l eastus) |
PublicNetworkAccess is required for this resouce on a second make azure-up | az cognitiveservices account create on an account that already has public access disabled | Fixed in up.sh: it runs account show first and creates only if that fails |
The Zone privatelink... exists already and hence cannot be created again | private-dns zone create is not idempotent | ensure_zone checks first |
PE_IP points at the wrong IP | customDnsConfigs[0] was the openai IP (.5), not cognitiveservices | outputs.sh reads the NIC’s ipConfigurations and takes the default member |
wg show on the VM never shows a handshake | NSG source is an old home IP, or your ISP uses CGNAT on IPv4 | Re-run make azure-up to refresh the rule; compare curl -4 api.ipify.org with the NSG |
| Probe shows all names public even with split DNS on | Probe container not on host networking, or a resolver other than resolved | Check docker run --rm --network host busybox cat /etc/resolv.conf shows 127.0.0.53 |
| Zone added, Azure shows the record, the DGX still gets the public IP | dnsmasq and resolved cache the old public CNAME chain (900 s TTL); about 13 minutes in my run | make dns-flush |
| A name returns “no answer” (NODATA) instead of public or private | Its privatelink zone is linked but has no record, or was just deleted; there’s no fallback to public DNS | Add the record via the zone group, or wait out the deletion (about 12 minutes here) |
After make vm-start the tunnel never comes back; wg-quick@wg0 inactive with no log | systemd ordering cycle (dnsmasq After=wg-quick@wg0), visible as “Found ordering cycle” in journalctl -b | Don’t order dnsmasq after WireGuard; bind-dynamic is enough (fixed in cloud-init.yaml) |
VNet routes clash after wg-quick up | 10.60.0.0/16 overlaps a network the DGX already uses | Set VNET_CIDR (and the subnets) in .env before make azure-up |
Check Your Understanding
The VM gets 200 for the cognitiveservices name but 403 for the openai name. Both go to the same account. Why?
Each name is resolved separately. The cognitiveservices name hits a linked private zone and gets 10.60.1.4. The openai name’s privatelink.openai.azure.com zone doesn’t exist yet, so it resolves publicly, the request leaves Azure’s network edge as public traffic, and the account rejects it.
Why does curl --resolve succeed while the probe still reports PUBLIC for the same name?
--resolve hands curl the IP and skips DNS entirely. The probe asks systemd-resolved, which still sends the openai name to your home router and gets the public answer. Routing is fine; only DNS is wrong.
What does azure-wait protect you from?
az vm create returns once the VM is provisioned, before cloud-init has installed packages and started WireGuard. Bringing up the tunnel then would fail with no visible error: WireGuard is silent when its peer doesn’t answer. Waiting for the marker file means the VM end is ready.