llama.cpp Server

About the bundled demo model: license and attribution

This image ships Qwen2.5-1.5B-Instruct (quantized to GGUF Q4_K_M), so the chat UI and API work immediately after first boot with no model download or internet access required. The Qwen team (Alibaba Cloud) publishes this model and, per its Hugging Face model card metadata, licenses it under the Apache License 2.0. This license is separate from, and in addition to, llama.cpp's own MIT license.

Apache 2.0 permits commercial use and redistribution. For this reason the image bundles this model rather than the 3B or 72B Qwen2.5 variants, which use a different, more restrictive Qwen License. If you replace it with a different model, check that model's own license before you redistribute it or rely on it in production.

Prerequisites

• An active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region.
• Standard_D4s_v3 (4 vCPU / 16 GiB RAM, dedicated, non-burstable) is the recommended starting point. CPU inference is compute-bound, so a burstable B-series size throttles under sustained chat or API use once its credits are exhausted. Size up for a larger model or higher concurrent request volume.
• NSG inbound: allow 22/tcp from your management network, 80/tcp for the chat UI and API, and 443/tcp if you add TLS.

llama-server serves plain HTTP. For production use, terminate TLS in front of it with your own domain.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for llama.cpp by SecureAnyCloud, and select Create.

On Basics, pick your subscription, resource group, region and a 4+ vCPU size (Standard_D4s_v3 recommended). Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:llama-cpp-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name llama-cpp \
  --image sac1647283583153:llama-cpp-ubuntu-24-04:default:latest \
  --size Standard_D4s_v3 \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name llama-cpp --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active llama-cpp.service nginx.service

Both report active. llama-server serves the chat UI and the full OpenAI-compatible API on the loopback address 127.0.0.1:8080 only. nginx fronts port 80 with the per-VM HTTP Basic Auth gate, which is the only public path in. The bundled demo model lives on the dedicated data disk mounted at /var/lib/llama-cpp.

Step 5 - Retrieve your password

nginx protects the chat UI and API with HTTP Basic Auth. The username is admin. Your VM generates a unique password on first boot and writes it to a root-only file:

sudo cat /root/llama-cpp-credentials.txt

The file contains LLAMA_CPP_USERNAME, LLAMA_CPP_PASSWORD, the LLAMA_CPP_URL to open in a browser, an example curl command against the API, and a note on the bundled model's license. The password is stored on disk only as a bcrypt hash in /etc/nginx/.llama-cpp.htpasswd, so no plaintext password ships in the image. Store the password somewhere safe.

Step 6 - Confirm the health endpoint and authentication

nginx serves an unauthenticated health endpoint for load balancers and probes, while the chat UI and API require the per-VM password. Test all four cases:

PW=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
echo "healthz : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/healthz)"
echo "unauth : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/)"
echo "wrongpw : $(curl -s -o /dev/null -w '%{http_code}' -u admin:wrong-pw http://127.0.0.1/)"
echo "authed : $(curl -s -o /dev/null -w '%{http_code}' --compressed -u admin:$PW http://127.0.0.1/)"

It prints healthz : 200, unauth : 401, wrongpw : 401, authed : 200. Only the per-VM password reaches the chat UI and API, because llama-server itself is bound to loopback and nginx port 80 is the only way in.

The --compressed flag is needed for the chat UI specifically, because its static assets are served gzip-only.

Step 7 - Call the OpenAI-compatible chat completions API

PW=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PW -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is the capital of France? Answer in one short sentence."}],"max_tokens":60}' \
  http://127.0.0.1/v1/chat/completions | python3 -m json.tool

The response is a standard OpenAI-shaped chat completion. choices[0].message.content holds the generated answer, model reports the loaded model's alias (qwen2.5-1.5b-instruct), and timings reports live generation speed in tokens per second. Use it to judge whether your VM size fits your workload.

To integrate llama.cpp Server into your own applications, point any OpenAI-API-compatible client or SDK at http://<vm-ip>/v1/ with HTTP Basic Auth credentials.

Step 8 - Sign in and chat in the browser

Browse to http://<vm-public-ip>/. When your browser prompts for a username and password, enter admin and the password from Step 5. llama.cpp's built-in chat UI opens, showing the loaded model (qwen2.5 1.5B instruct) in the message composer.

Type a question and press Enter. The UI streams the response back token by token and reports generation stats (tokens, time, tokens per second) under each message.

Step 9 - Explore the model and sampling settings

Click the gear icon to open Settings, then choose Sampling. Here you can view and adjust the inference parameters that the UI sends with every request, including temperature, top-K and top-P. Each has an inline explanation of what it controls.

These settings are stored in your browser and applied to the next request. They do not change the systemd service's own defaults.

Step 10 - List the loaded model via the API

GET /v1/models returns the OpenAI-compatible model list. You can also open it directly in a browser once you are authenticated. The list confirms the alias, context size and quantization of the model that llama-server currently has loaded.

Swapping in a different or larger model

The bundled Qwen2.5-1.5B-Instruct model is intentionally small, so first boot is fast and inference stays responsive on a modest VM. For more capability, download a different GGUF model (for example a larger Qwen2.5, Llama or Mistral instruct model in GGUF format) onto the data disk and point llama-server at it:

sudo curl -fL -o /var/lib/llama-cpp/<your-model>.gguf "<model-download-url>"
sudo chown llama-cpp:llama-cpp /var/lib/llama-cpp/<your-model>.gguf
sudo sed -i 's|--model [^ ]*|--model /var/lib/llama-cpp/<your-model>.gguf|' /etc/systemd/system/llama-cpp.service
sudo systemctl daemon-reload
sudo systemctl restart llama-cpp

A larger model needs more vCPU and RAM than this VM's current size. Resize the VM (or redeploy onto a larger size) before you switch to a 7B+ parameter model. Always check the license of any model you download before you use it commercially.

Maintenance

• Password: The UI and API password is set on first boot and stored as a bcrypt entry in /etc/nginx/.llama-cpp.htpasswd. To change it, run sudo htpasswd -B /etc/nginx/.llama-cpp.htpasswd admin and then sudo systemctl reload nginx.
• Thread count: llama-server's --threads flag is set from nproc at first boot, so it matches your VM's actual vCPU count. If you resize the VM after deployment, re-run first boot (or edit /etc/systemd/system/llama-cpp.service by hand) to keep it correctly tuned.
• Context size: The default --ctx-size 4096 balances memory use against conversation length. For longer conversations and documents, increase it in /etc/systemd/system/llama-cpp.service (this needs more RAM), then run sudo systemctl daemon-reload && sudo systemctl restart llama-cpp.
• Restrict access: llama-server serves plain HTTP on port 80. For production, restrict access to trusted IP ranges in your Network Security Group, and front it with TLS (for example certbot with your own domain) terminating on :443.
• Loopback binding: llama-server is bound to 127.0.0.1:8080 in /etc/systemd/system/llama-cpp.service (--host 127.0.0.1), so nginx is the only path in. Keep it that way, and do not change the bind address to a public interface.
• Data disk: The model file(s) live on the dedicated data disk at /var/lib/llama-cpp, which you can resize independently of the OS disk.
• Logs: journalctl -u llama-cpp shows recent model load and inference activity. journalctl -u nginx shows the HTTP access and error logs.
• Security patches: unattended-upgrades remains enabled, so the OS continues to receive security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.

Deploy on Azure

Launch llama.cpp Server on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com