DGX Spark to Azure Private Endpoint: Understand and Review

This part puts the six experiments back together into one path from the DGX to the Foundry account, lists what each one proved, and is honest about where a single VM stops being the right answer. It ends with the commands that delete everything, so the lab stops costing money.

The Full Signal Path

Ctrl + scroll (or pinch) to zoom

Laying out the diagram…

DGX Spark to Azure Private Endpoint

• You (on the DGX)
• DGX Spark (home network)
  • pe-probe (App / SDK)
  • Host networking (split DNS + tunnel): systemd-resolved, wg0
  • Grafana LGTM (Docker, grafana/otel-lgtm): OTLP receiver, Grafana
• Home router DNS (every other name)
• Azure (rg for the lab, East US)
  • Azure Resource Manager (az CLI)
  • VNet (10.60.0.0/16): snet-vpn, snet-pe
  • Azure DNS (168.63.129.16)
  • Private DNS zones (linked to the VNet): cognitiveservices zone, openai zone, services.ai zone
  • Foundry account (AIServices): Network rules, Data plane

Call over the internet: Experiment 1: no tunnel, no split DNS. The name resolves publicly and the account says 403.

1. Look up the name: The SDK needs an IP first. glibc asks systemd-resolved on 127.0.0.53.
2. Ask the home router: Without split DNS no link claims the name, so the router's DNS answers: a CNAME through privatelink.* to a public IP.
3. Call the public endpoint: DNS worked and the TCP and TLS handshakes succeed. Hover the arrow for both.
4. Refused: The account's network rules reject anything that doesn't arrive through its private endpoint.

Resolve the name privately: Experiment 3: split DNS sends only the Azure AI names through the tunnel to Azure DNS.

1. Look up the name: The same lookup as before; this time a routing domain on wg0 matches it.
2. Route it to wg0: Only the three Azure AI domains are routed to wg0. github.com still goes to the home router.
3. Through the tunnel: WireGuard wraps the query in UDP to the VM's public IP; the NSG accepts it from your home IP only.
4. dnsmasq picks it up: dnsmasq listens on the VM's tunnel address only.
5. Forward to Azure DNS: Azure DNS answers only from inside Azure, and only it sees the zones linked to the VNet.
6. The private zone answers: The public CNAME points into the privatelink zone; because it is linked, the answer is private.
7. The answer comes back: The answer retraces the path through the tunnel to the probe.

Call through the private endpoint: With 10.60.1.5 in hand, the HTTPS request takes the tunnel and comes back 200.

1. Into wg0: wg-quick routed 10.60.0.0/16 via wg0 (the AllowedIPs); nothing else uses the tunnel.
2. Encrypted to the VM: Same tunnel, same UDP port. The VM decrypts a TCP SYN from 10.200.0.2 to 10.60.1.5:443.
3. Forward and NAT: MASQUERADE rewrites the source, so the VNet needs no route back to 10.200.0.0/24.
4. Into the private endpoint: The private endpoint has one IP per public name: .4 cognitiveservices, .5 openai, .6 services.ai.
5. The account answers: Traffic through the private endpoint is allowed even with public network access disabled.
6. 200 back to the DGX: The reply goes to 10.60.2.4, the VM un-NATs it and sends it back through the tunnel.

VM deallocated: Experiment 5: with the VM off the Azure AI names fail instead of falling back to public.

1. Look up the name: wg0 is still up on the DGX, so the routing domain still matches.
2. Still sent to wg0: systemd-resolved doesn't fall back to the router for a name a link claims.
3. Nobody answers: The query goes into the tunnel to a VM that is off and nothing comes back: DNS TIMEOUT or SERVFAIL for all three names.

Watch it in Grafana: Every probe round becomes metrics, logs and traces on the local dashboard.

1. Probe exports: Each round's DNS answer, path, HTTP status and spans are exported to grafana/otel-lgtm.
2. Open the dashboard: No login: the container runs Grafana with anonymous admin, on the DGX only.
3. Query the signals: Name resolves to, HTTP status, the tunnel's health and the probe log.

Build the lab: make azure-up creates every Azure resource with the az CLI in about 10 minutes.

1. Run make azure-up: One resource group holds everything, so make down can delete it all.
2. Network and VM: cloud-init on the VM sets up WireGuard, IP forwarding, MASQUERADE and dnsmasq.
3. The account: From here on, only traffic through the private endpoint is let in.
4. Zones and links: A zone with no VNet link is never consulted by Azure DNS.
5. Records written: If the endpoint is ever recreated with new IPs, the records follow.
6. Done: About 10 minutes end to end. make probe-up then starts the local Grafana and the probe.

Every SDK call from the DGX now takes two trips through the tunnel: first the DNS question, then the HTTPS request.

1. Lookup. systemd-resolved matches <account>.openai.azure.com against the routing domain ~openai.azure.com on wg0 and asks 10.200.0.1.
2. Forward. dnsmasq on the VM asks Azure DNS (168.63.129.16), the only resolver that sees the VNet’s linked private zones. The public CNAME points into privatelink.openai.azure.com, and the linked zone answers 10.60.1.5.
3. Connect. The DGX routes 10.60.1.5 into wg0, because 10.60.0.0/16 is in AllowedIPs. The VM forwards the packet and MASQUERADE makes the source 10.60.2.4.
4. Allow. The packet reaches the account through its private endpoint, so the account’s publicNetworkAccess: Disabled doesn’t apply, and the call returns 200.

The probe watched the same path from the outside: DNS answer, TCP connect, TLS handshake and HTTP status, every 10 seconds, as metrics, logs and traces.

What the Experiments Showed

Experiment	What changed	What it proved
1. Over the internet	Nothing; the starting point	A private-only account answers 403 Public access is disabled at the HTTP level. DNS and TCP both work. The public name already CNAMEs into privatelink
2. Tunnel only	wg-quick up wg0	Routing to the endpoint’s IP works, and changes nothing for the SDK, because the name still resolves public
3. Split DNS	resolvectl routing domains on wg0	The cognitiveservices name went private (200). Names whose privatelink zone was missing didn’t. A private call cost 3 TLS round trips at about 211 ms each
4. Missing zones	privatelink.openai and privatelink.services.ai zones	A Foundry endpoint needs three zones. Caches held the old public answer for about 13 minutes after the fix
5. VM off	az vm deallocate	The names fail closed: DNS timeouts, never a fallback to public
6. VM on	az vm start	Full recovery about 75 seconds after the start command, with nothing touched on the DGX (once a systemd ordering cycle was fixed)

Three things surprised me while building it, and each is now part of the lab:

• An AIServices private endpoint has three private IPs, one per public name. Scripts that read customDnsConfigs[0] get the wrong one for cognitiveservices.
• DNS caches outlive your fix. Azure DNS was right within a minute, and the DGX was wrong for 13 more minutes.
• The first boot can hide boot-order bugs. cloud-init started the services by hand, so the ordering cycle appeared only on the second boot.

Limits of This Setup

• One VM, one zone, no redundancy. If vm-wg-pe-lab or its zone goes down, the DGX loses the private path (experiment 5). That’s fine for one developer machine, not for anything others rely on.
• You patch the VM. It’s a public-facing Ubuntu host. The NSG limits it to your IP, but OS updates, WireGuard key rotation and SSH hardening are on you. A VPN Gateway hands that to Azure.
• Your home IP is in the NSG. If your ISP changes it, the tunnel stops until you re-run make azure-up. Under CGNAT you may not have a stable IPv4 address at all.
• API key authentication. The probe uses the account key for simplicity. Real code should use Microsoft Entra ID (DefaultAzureCredential) and the account should ideally have local auth disabled. Entra token requests go to login.microsoftonline.com over the internet, which is fine: only the data plane is private.
• One client. AllowedIPs = 10.200.0.2/32 on the VM means exactly one peer. More machines need a peer block each, with its own key.
• The Grafana stack is a lab tool. Anonymous admin on port 3012 is fine on the DGX and nowhere else.

Taking It to Production

Recommendations, roughly in the order I’d do them:

1. Cap the DNS caches you own. Add max-cache-ttl=30 and max-ttl=30 to the VM’s dnsmasq config. The first limits dnsmasq’s own cache. The second limits the TTL it gives the DGX, so resolved re-asks within 30 seconds too. A zone change then reaches clients in under a minute instead of 15, for a few more lookups through the tunnel.
2. Link every zone the service needs from the start. For Foundry that’s privatelink.cognitiveservices.azure.com, privatelink.openai.azure.com and privatelink.services.ai.azure.com, all in the endpoint’s zone group. In a hub-and-spoke network, keep the zones central and link them to the hub, where the DNS forwarder lives.
3. Use Entra ID and disable local auth on the account.
4. Keep the VM patched with unattended upgrades, and remove the SSH rule from the NSG when you’re not using it (or use Azure Bastion).
5. Move to managed pieces when several people depend on it: a VPN Gateway (point-to-site, VpnGw1AZ at about $157 a month) for the tunnel, and a DNS Private Resolver inbound endpoint ($180 a month) in place of dnsmasq. That’s about $340 a month against about $14, so it’s worth it for a team, not for one DGX.
6. Or use Tailscale as a subnet router on the same VM: no inbound port and no NSG rule for your home IP, and its split DNS can send the three domains to 168.63.129.16 from the VM. The VM still needs outbound internet access (its public IP or a NAT Gateway).

Tear Down

On the DGX first, because it needs sudo and the Azure resources are what cost money:

make dgx-down                      # wg0 down and disabled at boot; the split DNS goes with the link
sudo rm /etc/wireguard/wg0.conf
make down                          # stop Grafana and the probe, delete rg-pe-lab, purge the account

make down runs docker compose down -v (which deletes the telemetry), then az group delete -n rg-pe-lab, which removes the VM, disk, public IP, NSG, VNet, private endpoint, all three zones and the Foundry account. A deleted AIServices account stays soft-deleted for 48 hours and keeps its name, so down.sh purges it as well. The WireGuard and SSH keys stay in .state/. Delete that folder too if you’re done for good.

If you only want to stop paying for compute, make vm-stop deallocates the VM and leaves the rest: about $6 a month for the IP and the disk, and make vm-start brings it back as in experiment 6.

Go Further

• Same pattern, other private endpoints. Add the domain to the resolvectl domain list and the zone to the VNet: Key Vault (~vaultcore.azure.net), Storage (~blob.core.windows.net), Azure AI Search (~search.windows.net). Each has its own privatelink zone.
• Call a model. Deploy a model in the account and point the OpenAI SDK at https://<account>.openai.azure.com. Watch the probe’s latency panel next to your own request times.
• Try the fallback policy. Set --resolution-policy NxDomainRedirect on a zone’s VNet link and repeat the zone reset from experiment 4. Names missing from the zone should then fall back to public answers instead of failing.
• Compare with the clouds next door. The same two halves (a route in, a private answer) apply to AWS PrivateLink with Route 53 private hosted zones and to GCP Private Service Connect.
• Background reading on this blog: Azure - Private Endpoints, Azure - Virtual Private Network (VPN).

Check Your Understanding

A colleague says "the private endpoint works, I tested it from a VM in the VNet". Your SDK on the DGX still gets 403. Name two reasons.

First, the DGX may not be sending that domain to the VNet’s DNS at all (no routing domain, or the tunnel is down), so it resolves the public chain. Second, the colleague may have tested the cognitiveservices name while your SDK calls <account>.openai.azure.com, whose privatelink zone is missing. A third possibility is a cache still holding the old public answer.

What does this setup cost per month if you deallocate the VM every night for 12 hours?

Compute is billed only while the VM runs: about 365 hours × $0.0104 ≈ $3.80. The static IP ($3.65) and the disk ($2.40) are billed all month, so the total is about $9.85 for the connectivity, plus the private endpoint and zones if they’re yours to pay for.

Why is the VM's NSG rule for UDP 51820 tied to your home IP, and what does that cost you?

It keeps the WireGuard port invisible to the rest of the internet, which is cheap defence in depth on top of WireGuard’s own key checks. The cost is that a change of home IP silently stops the tunnel until you update the rule (make azure-up).

When would you replace dnsmasq with a DNS Private Resolver?

When more than one network or several people depend on resolving private zones from outside Azure, and you want a managed, zone-redundant service instead of a process on one VM. At $180 a month for the inbound endpoint, it’s a team decision, not a single-developer one.