vLLM

Search for vLLM in the Azure Marketplace, select the SecureAnyCloud publisher and click Create. Choose an NC or ND-series GPU VM size.

Configure the network security group (NSG) rules from your client networks: TCP 22 (admin), TCP 443 (inference API over HTTPS) and TCP 80 (HTTP to HTTPS redirect and health probe).

Run these commands to create the network resources, accept the Marketplace terms and create the GPU VM:

# Deploy on a GPU VM size (subscription <sub-id>). Replace the gallery image version. RG="vllm-prod"; LOCATION="eastus"; VM_NAME="vllm-01" IMAGE_URN="sac1647283583153:vllm-ubuntu-24-04:default:latest" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create -g "$RG" --name vllm-vnet --address-prefix 10.100.0.0/16 --subnet-name vllm-subnet --subnet-prefix 10.100.1.0/24 az network nsg create -g "$RG" --name vllm-nsg az network nsg rule create -g "$RG" --nsg-name vllm-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name vllm-nsg --name allow-https --priority 110 \
  --destination-port-ranges 443 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name vllm-nsg --name allow-http --priority 120 \
  --destination-port-ranges 80 --access Allow --protocol Tcp
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:vllm-ubuntu-24-04:default:latest
az vm create -g "$RG" --name "$VM_NAME" --image "$IMAGE_URN" \
  --size Standard_NC24ads_A100_v4 --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name vllm-vnet --subnet vllm-subnet --nsg vllm-nsg --public-ip-sku Standard

vLLM needs the NVIDIA driver on the VM. On an NC/ND-series GPU VM, add the NVIDIA GPU Driver Extension, which installs the datacenter driver:

# Add the NVIDIA driver to the GPU VM you created in Step 2 (subscription <sub-id>).
az vm extension set -g "$RG" --vm-name "$VM_NAME" \
  --name NvidiaGpuDriverLinux --publisher Microsoft.HpcCompute --version 1.9 \
  --settings '{}'

If the extension does not install on your VM, install the driver from Canonical's prebuilt packages instead. Then reboot:

sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common
sudo ubuntu-drivers install sudo reboot

Once the driver is installed, vllm.service starts automatically, because it is enabled, and serves the model on the GPU. Confirm that the GPU is visible with nvidia-smi and that the service is active with sudo systemctl status vllm.service.

SSH in as azureuser. vLLM is installed in the /opt/vllm venv. nginx fronts it on :443, and the vLLM server binds loopback :8000 only. Run this command to confirm the version and service state:

/opt/vllm/bin/python -c 'import vllm; print("vLLM", vllm.__version__)' systemctl is-active nginx.service systemctl is-enabled vllm.service ss -tln | grep -E ':443 |:80 '

Then retrieve the per-VM bearer API key from the root-only file:

sudo grep '^VLLM_API_KEY=' /root/vllm-credentials.txt

vLLM ships with no authentication, so this image never exposes it directly. nginx enforces the per-VM bearer key. The open health endpoint answers without auth, an anonymous inference request is rejected with 401, and only the per-VM key is accepted.

Verify this on the VM. The -k flag skips the self-signed certificate check. In production, use --cacert /etc/nginx/tls/vllm.crt or your own CA certificate:

KEY="$(sudo grep '^VLLM_API_KEY=' /root/vllm-credentials.txt | cut -d= -f2-)" echo -n 'open /health -> '; curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/health echo -n 'anon /v1/models -> '; curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/v1/models echo -n 'keyed /v1/models -> '; curl -sk -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $KEY" https://127.0.0.1/v1/models

The open endpoint returns 200 and the anonymous inference request returns 401. nginx accepts the keyed request. It returns the model list as 200 once the GPU-backed vLLM upstream is serving. Until then, nginx forwards the accepted request and reports the upstream as unavailable.

vLLM requires a GPU. On a VM with no NVIDIA GPU, vllm.service does not start and logs a clear diagnostic, so the state is never ambiguous. Once the NVIDIA driver is present (Step 3), the service starts and serves on the GPU.

Issue the certificate for your public address

First boot puts the address that Azure reports into the TLS certificate, the credentials file and the login banner. With a Standard SKU public IP, which is the Azure default and the --public-ip-sku Standard setting in Step 2, the current image records the VM's private address, for example 10.100.1.4.

A client that trusts vllm.crt with --cacert, or an SDK that verifies TLS, then refuses the certificate at the public IP with curl: (60) SSL: no alternative certificate subject name matches. Check which address was recorded:

sudo grep '^VLLM_URL=' /root/vllm-credentials.txt

If the file shows a private 10. address, issue a new certificate for the public IP on the VM. Azure's instance metadata service reports the public IP to the VM itself, and nothing leaves the VM:

PUBLIC_IP=$(curl -s -H Metadata:true 'http://169.254.169.254/metadata/loadbalancer?api-version=2020-10-01' | grep -o '"frontendIpAddress":"[^"]*"' | head -1 | cut -d'"' -f4)
PRIV=$(hostname -I | awk '{print $1}'); echo "public $PUBLIC_IP, private $PRIV"
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout /etc/nginx/tls/vllm.key -out /etc/nginx/tls/vllm.crt \
  -subj "/C=GB/O=sac/CN=$PUBLIC_IP" \
  -addext "subjectAltName=IP:127.0.0.1,DNS:localhost,DNS:$(hostname -s),IP:$PRIV,IP:$PUBLIC_IP" sudo chmod 0600 /etc/nginx/tls/vllm.key
sudo sed -i "s/\b$PRIV\b/$PUBLIC_IP/g" /root/vllm-credentials.txt /etc/update-motd.d/90-vllm
sudo systemctl restart nginx

To use a DNS name, set PUBLIC_IP to the name and write DNS:$PUBLIC_IP in place of IP:$PUBLIC_IP. Your API key does not change. Copy the new /etc/nginx/tls/vllm.crt to your clients and use --cacert vllm.crt in place of -k in the steps below.

From a client that can reach the VM, list the models and run a chat completion. Replace <vm-ip> with your VM public IP and <key> with the value from Step 4. Add --cacert with the VM certificate, or use -k for the self-signed certificate during testing:

# List the served models curl -k -H "Authorization: Bearer <key>" https://<vm-ip>/v1/models
# Chat completion curl -k -H "Authorization: Bearer <key>" https://<vm-ip>/v1/chat/completions \
  -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Say hello in one sentence."}]}'

A successful /v1/chat/completions call returns an OpenAI-style JSON response with a choices[0].message.content field containing the model's reply.

The service reads its model name and API key from /etc/vllm/vllm.env. The starter model is pre-downloaded into the HuggingFace cache. Run these commands to view the environment file, with the key redacted, and the cached model files:

sudo sed 's/^VLLM_API_KEY=.*/VLLM_API_KEY=REDACTED/' /etc/vllm/vllm.env find -L /var/lib/vllm/hf -type f -name '*.safetensors' | sed 's#.*/hub/##'

To serve a different model, edit MODEL in /etc/vllm/vllm.env and restart the service. Larger models need more GPU memory.

# On the VM, as root — set MODEL to any HuggingFace model id (subscription <sub-id> example):
sudo sed -i 's#^MODEL=.*#MODEL=meta-llama/Llama-3.1-8B-Instruct#' /etc/vllm/vllm.env
sudo systemctl restart vllm.service

Point any OpenAI SDK, LangChain or LlamaIndex client at the VM by setting the base URL and the per-VM key:

from openai import OpenAI client = OpenAI(base_url="https://<vm-ip>/v1", api_key="<key>") resp = client.chat.completions.create( model="Qwen/Qwen2.5-1.5B-Instruct", messages=[{"role": "user", "content": "Explain PagedAttention in one sentence."}], ) print(resp.choices[0].message.content)

For production, replace the self-signed certificate at /etc/nginx/tls/vllm.crt and /etc/nginx/tls/vllm.key with your own CA-signed certificate, keeping the same paths. Then reload nginx, so clients can verify the endpoint without -k.

The model cache lives at /var/lib/vllm/hf on the OS disk. For large model libraries, attach a separate Azure managed data disk, then format and mount it. Point HF_HOME at the mount in /etc/vllm/vllm.env, so model weights sit on an independently sized volume that you can snapshot.

Snapshot the disk with the Azure CLI on your own schedule.