llama.cpp Server b9937

Option A: AWS Marketplace console

• Open the SecureAnyCloud Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace. Choose View purchase options, then Subscribe. One subscription covers every version in the listing.
• Once the subscription is active, choose Launch your software.
• Choose One-click launch from AWS Marketplace, or Launch from EC2 Console for the full launch wizard. In Version, choose llama.cpp Server b9937 on Ubuntu 24.04, then choose your Region.
• Pick an instance type (m5.large or larger; the software charge is $0.04 per vCPU per hour) and your key pair.
• Set the security group. The listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks your users and applications use.
• Choose your VPC and subnet, and leave the metadata defaults, because the image already requires IMDSv2. Then choose Launch. The 20 GiB model volume is created and attached automatically.

Option B: AWS CLI

Run the following commands on your own workstation. They create a security group that allows SSH from your management address and HTTPS from your own network, then launch the instance. Replace the AMI ID with the one shown for your Region on the Marketplace launch page. Replace the VPC, subnet, key pair and CIDR ranges with your own.

SG=$(aws ec2 create-security-group --group-name llama-cpp-server \
  --description "llama.cpp Server" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=llama-cpp-server}]'

You do not need --metadata-options, because the AMI already makes IMDSv2 required.

Connect as ubuntu with the key pair you chose.

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes a few minutes after launch. A new instance restores the model volume from its snapshot as it is first read, so first boot reads the model through once before llama-server finishes loading it.

First boot then sets the hostname that llama.cpp Server is served as, which is the instance's public DNS name when it has one. It creates the TLS certificate for that name and generates the password. It proves that the gate refuses anonymous and wrong credentials, starts nginx and writes /root/llama-cpp-credentials.txt (mode 0600, root).

The following commands list the entries in that file without showing the secret values.

sudo grep -E '^LLAMA_CPP_(URL|USERNAME|API_URL)=' /root/llama-cpp-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/llama-cpp-credentials.txt
sudo stat -c '%a %U:%G' /root/llama-cpp-credentials.txt

LLAMA_CPP_URL=https://ec2-54-166-74-253.compute-1.amazonaws.com/ LLAMA_CPP_USERNAME=admin LLAMA_CPP_API_URL=https://ec2-54-166-74-253.compute-1.amazonaws.com/v1/chat/completions LLAMA_CPP_HOST= LLAMA_CPP_URL= LLAMA_CPP_USERNAME= LLAMA_CPP_PASSWORD= <set-at-first-boot>= LLAMA_CPP_LOOPBACK_API_KEY= 600 root:root

The entries are:

• LLAMA_CPP_HOST is the hostname the chat UI and API are served as.
• LLAMA_CPP_URL is the chat UI address of this instance.
• LLAMA_CPP_USERNAME is always admin.
• LLAMA_CPP_PASSWORD is the password for the chat UI and the API, generated on this instance.
• LLAMA_CPP_API_URL is the OpenAI-compatible chat completions endpoint.
• LLAMA_CPP_LOOPBACK_API_KEY is llama-server's own key. Use it only in scripts on this instance that call http://127.0.0.1:8080 directly. nginx adds it for you on HTTPS.

To read the password, run sudo cat /root/llama-cpp-credentials.txt in your own session and store it in your secrets manager.

Check that the key unit, llama-server, nginx and the first-boot unit are active, and that only sshd and nginx listen beyond loopback. llama-server itself is on 127.0.0.1:8080, so nginx is the only way in from the network.

systemctl is-active llama-cpp-apikey.service llama-cpp.service nginx.service llama-cpp-firstboot.service
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u LD_LIBRARY_PATH=/opt/llama-cpp/llama-b9937 /opt/llama-cpp/llama-b9937/llama-server --version 2>&1 | head -2

active active active active 0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:8080 users:(("llama-server")) 127.0.0.53%lo:53 users:(("systemd-resolve")) 127.0.0.54:53 users:(("systemd-resolve")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx")) version: 9937 (2021515a1) built with GNU 11.4.0 for Linux x86_64

nginx answers /healthz itself, without a password, and it reveals nothing about your configuration. It is safe to use as a load balancer health check.

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt "https://$H/healthz"

ok

Without the password, the chat UI and every API path are refused. A password that is wrong by one character is refused too. This instance's password is accepted.

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-) for R in / /v1/models /v1/chat/completions /props; do printf '%s -> %s\n' "$R" "$(curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w '%{http_code}' "https://$H$R")" done curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w 'wrong password -> %{http_code}\n' -K <(printf 'user = "admin:%sx"\n' "$P") "https://$H/v1/models" curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w 'this instance password -> %{http_code}\n' -K <(printf 'user = "admin:%s"\n' "$P") "https://$H/v1/models"

/ -> 401 /v1/models -> 401 /v1/chat/completions -> 401 /props -> 401 wrong password -> 401 this instance password -> 200

The second gate is llama-server's own API key. A call straight to the loopback port, without going through nginx, is refused by llama-server itself.

curl -s -w ' HTTP_%{http_code}\n' -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}]}' http://127.0.0.1:8080/v1/chat/completions

{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}} HTTP_401

The commands above pass the password to curl through a configuration read from a pipe (-K). The password never appears in the process list or your shell history.

Browse to LLAMA_CPP_URL (https://<public-dns-name>/). The certificate is self-signed, so your browser warns you once. Accept it for this host, or install your own certificate (see Step 8). The browser then asks for a username and password. Use admin and LLAMA_CPP_PASSWORD.

The chat UI opens with the bundled model selected.

Type a message and press Enter. The reply streams in token by token, with the prompt and generation speed shown under each message.

The browser stores conversations, not the server. Under Settings, you can set a system message and the sampling parameters. Leave API Key empty, because nginx already supplies llama-server's key once you have signed in.

The first time you send a message, the UI may offer to add MCP servers. That feature runs in your browser and connects from there. The server-side MCP proxy is turned off on this image.

The model is listed under the alias qwen2.5-1.5b-instruct. List the models first.

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -K <(printf 'user = "admin:%s"\n' "$P") "https://$H/v1/models" | jq -r '.data[] | [.id, .owned_by] | @tsv'

qwen2.5-1.5b-instruct llamacpp

Then request a chat completion, which the instance's CPU generates.

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -K <(printf 'user = "admin:%s"\n' "$P") \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is the capital of France? Answer in one word."}],"max_tokens":16,"temperature":0}' \
  "https://$H/v1/chat/completions" | jq -r '.choices[0].message.content, .usage.total_tokens'

Paris 43

From your own workstation, copy the instance's certificate (/etc/ssl/llama-cpp/llama-cpp.crt) and trust it. Alternatively, put a certificate your clients already trust in front of the instance (Step 8).

With the OpenAI Python SDK, pass the password as HTTP Basic credentials through the HTTP client. The gate does not use the api_key value.

import httpx from openai import OpenAI client = OpenAI( base_url="https://<LLAMA_CPP_HOST>/v1", api_key="unused", http_client=httpx.Client(auth=("admin", "<LLAMA_CPP_PASSWORD>"), verify="llama-cpp.crt"), ) reply = client.chat.completions.create( model="qwen2.5-1.5b-instruct", messages=[{"role": "user", "content": "Hello!"}], ) print(reply.choices[0].message.content)

To receive the reply token by token, add "stream": true to the request body, or stream=True in the SDK.

The model lives on its own gp3 volume at /var/lib/llama-cpp, so you can grow it independently of the root volume. To grow it, modify the volume in the EC2 console, then run sudo resize2fs on its device.

Check the volume and the running service.

df -h /var/lib/llama-cpp | tail -1 sudo ls -l /var/lib/llama-cpp/*.gguf systemctl show -p ExecStart --value llama-cpp.service | grep -oE -- '--(model|alias|ctx-size|threads) [^ ]+'

/dev/nvme1n1 20G 1.1G 18G 6% /var/lib/llama-cpp -rw-r----- 1 llama-cpp llama-cpp 1117320736 Oct 20 01:47 /var/lib/llama-cpp/qwen2.5-1.5b-instruct-q4_k_m.gguf --model /var/lib/llama-cpp/qwen2.5-1.5b-instruct-q4_k_m.gguf --alias qwen2.5-1.5b-instruct --ctx-size 4096 --threads 2

To serve another GGUF model, download it onto the volume from a source you trust and check its licence. Then point the unit at it and restart. For example:

sudo -u llama-cpp curl -fL -o /var/lib/llama-cpp/<model>.gguf https://huggingface.co/<org>/<repo>/resolve/main/<model>.gguf
sudo sed -i 's#--model [^ ]*#--model /var/lib/llama-cpp/<model>.gguf#; s#--alias [^ ]*#--alias <model-alias>#' /etc/systemd/system/llama-cpp.service
sudo systemctl daemon-reload && sudo systemctl restart llama-cpp

A model needs roughly its file size in memory plus room for the context, so size the instance to match. Keep the --api-key-file and --no-slots options in the unit.

Port 80 redirects to HTTPS on the configured hostname. The certificate covers that name, the instance's public and private addresses, localhost and 127.0.0.1. Check the redirect and the certificate.

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://$H/v1/models" openssl x509 -in /etc/ssl/llama-cpp/llama-cpp.crt -noout -subject -enddate -ext subjectAltName

301 https://ec2-54-166-74-253.compute-1.amazonaws.com/v1/models subject=CN = ec2-54-166-74-253.compute-1.amazonaws.com, O = sac notAfter=Jan 22 01:48:08 2029 GMT X509v3 Subject Alternative Name: DNS:ec2-54-166-74-253.compute-1.amazonaws.com, IP Address:54.166.74.253, IP Address:172.31.87.137, DNS:ip-172-31-87-137.ec2.internal, IP Address:127.0.0.1, DNS:localhost

To use your own name, point a DNS record at the instance. An Elastic IP keeps the address stable across stops. Then run the helper. It records the name in /etc/sac/llama-cpp-hostname, issues a new self-signed certificate for it, updates the credentials file and reloads nginx.

sudo llama-cpp-set-hostname <your-dns-name>

For a publicly trusted certificate, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate. Alternatively, replace /etc/ssl/llama-cpp/llama-cpp.crt and llama-cpp.key with your own and run sudo systemctl reload nginx.

llama-server carries the metadata block and runs with the slot monitor and property changes off. nginx refuses cross-origin writes and requests for any hostname it was not configured with. Run the following checks.

systemctl show -p IPAddressDeny llama-cpp.service
K=$(sudo grep '^LLAMA_CPP_LOOPBACK_API_KEY=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") -w ' HTTP_%{http_code}\n' http://127.0.0.1:8080/slots curl -s -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") -w ' HTTP_%{http_code}\n' -H 'Content-Type: application/json' -d '{}' http://127.0.0.1:8080/props
H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w 'cross-origin POST: %{http_code}\n' -K <(printf 'user = "admin:%s"\n' "$P") \
  -X POST -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{}' "https://$H/v1/chat/completions" code=$(curl -s -o /dev/null -w '%{http_code}' -H 'Host: evil.example' http://127.0.0.1/ || true); echo "unknown Host: ${code} (connection closed by nginx)"

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128 {"error":{"code":501,"message":"This server does not support slots endpoint. Start it with `--slots`","type":"not_supported_error"}} HTTP_501 {"error":{"code":501,"message":"This server does not support changing global properties. Start it with `--props`","type":"not_supported_error"}} HTTP_501 cross-origin POST: 403 unknown Host: 000 (connection closed by nginx)

The SecureAnyCloud build also proves, before the image is captured, that:

• A process inside llama-cpp.service cannot reach the metadata service, while the same process still reaches loopback.
• A forged X-Forwarded-For never reaches llama-server.
• No password or key appears in the journal, /var/log or any process command line.