GPT-Load 1.4.12

Option A: AWS Marketplace console

• Open the SecureAnyCloud Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace. Choose View purchase options, then Subscribe. One subscription covers every version in the listing.
• Once the subscription is active, choose Launch your software.
• Choose One-click launch from AWS Marketplace (or Launch from EC2 Console for the full launch wizard). In Version, choose GPT-Load 1.4.12 on Ubuntu 24.04, then choose your Region.
• Pick an instance type (m5.large recommended; the software charge is $0.04 per vCPU per hour) and your key pair.
• Set the security group. The listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks your applications and administrators use. This relay spends your provider credit.
• Choose your VPC and subnet and leave the metadata defaults, because the image already requires IMDSv2. Then choose Launch.

Option B: AWS CLI

Run the following commands on your own workstation. They create a security group that allows SSH from your management address and HTTPS from your application network, then launch the instance. Replace the AMI ID with the one shown for your Region on the Marketplace launch page. Replace the VPC, subnet, key pair and CIDR ranges with your own:

SG=$(aws ec2 create-security-group --group-name gpt-load-relay \
  --description "GPT-Load relay" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=gpt-load-relay}]'

You do not need --metadata-options, because the AMI already makes IMDSv2 required.

Connect as ubuntu with the key pair you chose:

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes within a minute of launch. It does the following:

• Sets the hostname GPT-Load is served as (the instance's public DNS name when it has one) and creates the TLS certificate for it.
• Generates the gate password, the auth key, the encryption key and the gate token.
• Proves that the gate refuses anonymous and wrong credentials.
• Starts GPT-Load and nginx.
• Writes /root/gpt-load-credentials.txt (mode 0600, root).

These are the entries in that file. The secret values are not shown here:

sudo grep -E '^GPTLOAD_(URL|GATE_USERNAME)=' /root/gpt-load-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/gpt-load-credentials.txt
sudo stat -c '%a %U:%G' /root/gpt-load-credentials.txt

GPTLOAD_URL=https://ec2-44-204-33-219.compute-1.amazonaws.com/ GPTLOAD_GATE_USERNAME=gptload GPTLOAD_URL= GPTLOAD_GATE_USERNAME= GPTLOAD_GATE_PASSWORD= <set-at-first-boot>= GPTLOAD_ENCRYPTION_KEY= 600 root:root

Value, and what it is for:

GPTLOAD_URL: the dashboard and relay address of this instance.

GPTLOAD_GATE_USERNAME: the SecureAnyCloud gate's username for a browser (always gptload).

GPTLOAD_GATE_PASSWORD: the SecureAnyCloud gate's password for a browser, generated on this instance.

GPTLOAD_AUTH_KEY: the one credential your applications use, and the dashboard's Auth Key.

GPTLOAD_ENCRYPTION_KEY: encrypts your pooled provider keys at rest. Keep it.

To read the values, run sudo cat /root/gpt-load-credentials.txt in your own session and store them in your secrets manager. Keep GPTLOAD_ENCRYPTION_KEY: your pooled provider keys are encrypted with it, and changing it without upstream's key migration makes the stored pool unreadable.

Confirm that the first-boot unit, GPT-Load and nginx are active, and that only sshd and nginx listen beyond loopback. GPT-Load itself is on 127.0.0.1:3001, so the nginx gate is the only way in.

systemctl is-active gpt-load-firstboot.service gpt-load.service nginx.service
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u curl -sk https://127.0.0.1/nginx-health

active active active 0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx")) 127.0.0.1:3001 users:(("gpt-load")) 127.0.0.53%lo:53 users:(("systemd-resolve")) 127.0.0.54:53 users:(("systemd-resolve")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx")) OK

nginx itself answers /nginx-health and reveals nothing about your configuration, so it is safe to use as a load balancer health check.

Run the checks below. The gate refuses anonymous requests to the dashboard, GPT-Load's health route, the admin API and the relay paths. It also refuses an auth key that is wrong by one character. It accepts this instance's own key in every form an SDK sends it:

for R in / /health /api/settings /api/groups /api/logs /proxy/default/v1/models; do printf '%s -> %s\n' "$R" "$(curl -sk -o /dev/null -w '%{http_code}' https://127.0.0.1$R)" done curl -sk -o /dev/null -w 'anonymous POST /proxy/default/v1/chat/completions -> %{http_code}\n' \
  -X POST -H 'Content-Type: application/json' -d '{}' https://127.0.0.1/proxy/default/v1/chat/completions
K=$(sudo grep '^GPTLOAD_AUTH_KEY=' /root/gpt-load-credentials.txt | cut -d= -f2-)
curl -sk -o /dev/null -w 'wrong key -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %sX"\n' "$K") https://127.0.0.1/api/settings for H in "Authorization: Bearer" "X-Api-Key:" "X-Goog-Api-Key:"; do
curl -sk -o /dev/null -w "$H <key> -> %{http_code}\n" -K <(printf 'header = "%s %s"\n' "$H" "$K") https://127.0.0.1/api/settings done
curl -sk -o /dev/null -w '?key=<key> -> %{http_code}\n' -K <(printf 'url = "https://127.0.0.1/api/settings?key=%s"\n' "$K")

/ -> 401 /health -> 401 /api/settings -> 401 /api/groups -> 401 /api/logs -> 401 /proxy/default/v1/models -> 401 anonymous POST /proxy/default/v1/chat/completions -> 401 wrong key -> 401 Authorization: Bearer <key> -> 200 X-Api-Key: <key> -> 200 X-Goog-Api-Key: <key> -> 200 ?key=<key> -> 200

The gate accepts six equivalent proofs. Each is compared exactly and is unique to this instance. Proof, and who uses it:

HTTP Basic with GPTLOAD_GATE_USERNAME / GPTLOAD_GATE_PASSWORD: your browser, and curl -u.

A gate cookie issued after the browser signs in (Secure, HttpOnly, SameSite=Lax): your browser, on every dashboard call.

Authorization: Bearer <auth key>: the OpenAI SDK.

x-api-key: <auth key>: the Anthropic SDK.

x-goog-api-key: <auth key>: the Google GenAI SDK.

?key=<auth key>: GPT-Load's native query form.

The last four are exactly the idioms GPT-Load's own authentication reads. Your application therefore sends one credential that satisfies both the SecureAnyCloud gate and GPT-Load's own check.

Browse to GPTLOAD_URL (https://<public-dns-name>/). The certificate is self-signed, so your browser warns once. Accept it for this host, or install your own certificate (see Step 8).

The browser then asks for a username and password. Enter GPTLOAD_GATE_USERNAME and GPTLOAD_GATE_PASSWORD. GPT-Load's own login page then asks for its Auth Key. Paste GPTLOAD_AUTH_KEY.

The dashboard opens with nothing configured: no keys and no requests. This image never ships a provider key, so that is exactly the state you receive:

The footer may offer an update to a GPT-Load 2.0 release candidate. The 1.4 line is the current stable release and the one this version ships.

The image ships with no group configured, and the relay's key store is encrypted with this instance's own key. Confirm both from the shell:

K=$(sudo grep '^GPTLOAD_AUTH_KEY=' /root/gpt-load-credentials.txt | cut -d= -f2-)
curl -sk -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") https://127.0.0.1/api/groups | jq -c .data
curl -sk -H 'Accept-Language: en' -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") https://127.0.0.1/api/dashboard/encryption-status | jq -c .data

[] {"has_mismatch":false,"message":"","scenario_type":"","suggestion":""}

has_mismatch: false means that the key GPT-Load is running with is the key its stored data is encrypted with. Then, in the dashboard:

• Open Keys and choose Create Group.
• Give the group a short, URL-safe name, for example gemini. The name becomes part of your endpoint URL.
• Choose the Channel Type that matches the provider: openai, gemini or anthropic.
• Set the Upstream Address to the provider's own API base, for example https://generativelanguage.googleapis.com for Gemini or https://api.openai.com for OpenAI.
• Save the group, then use Add Key to paste your API keys, one per line. These keys are the pool.

A group can also be an aggregate group of sub-groups with weights, which blends several providers behind one endpoint.

Your endpoint is https://<GPTLOAD_HOST>/proxy/<your-group-name> and your API key is GPTLOAD_AUTH_KEY. Either copy the instance's certificate (/etc/ssl/gpt-load/gpt-load.crt) to your client and trust it, or put a certificate your clients already trust in front (Step 8).

With the OpenAI Python SDK, against a group of channel type openai:

from openai import OpenAI client = OpenAI( base_url="https://<GPTLOAD_HOST>/proxy/<your-group-name>/v1", api_key="<GPTLOAD_AUTH_KEY>", ) resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello"}], ) print(resp.choices[0].message.content)

With the Anthropic Python SDK, against a group of channel type anthropic:

from anthropic import Anthropic client = Anthropic( base_url="https://<GPTLOAD_HOST>/proxy/<your-group-name>", api_key="<GPTLOAD_AUTH_KEY>", )

With curl, against a group of channel type gemini:

curl --cacert gpt-load.crt "https://<GPTLOAD_HOST>/proxy/<your-group-name>/v1beta/models/gemini-2.0-flash:generateContent" \
  -H "X-Goog-Api-Key: <GPTLOAD_AUTH_KEY>" -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"Hello"}]}]}'

GPT-Load passes everything after the group name through to the provider unchanged. For example, /proxy/gemini/v1beta/models/... becomes https://generativelanguage.googleapis.com/v1beta/models/... with one of your pooled keys attached.

Key health, rotation and blacklisting

• Rotation. Each request takes the next key from the pool, so load is spread across every key you own.
• Validation. Choose Validate on a group and GPT-Load tests every key in the pool against the upstream.
• Blacklisting. GPT-Load automatically moves a key that fails repeatedly out of rotation, so a dead or rate-limited key stops failing your traffic. Restore all invalid puts the keys back, for example after a quota window resets.

The Logs page records every relayed request with its group, the pooled key used, the status, the latency and the real client address:

Port 80 redirects to HTTPS on the configured hostname. The certificate covers that name, the instance's public and private addresses, localhost and 127.0.0.1. Check both:

H=$(sudo grep '^GPTLOAD_URL=' /root/gpt-load-credentials.txt | cut -d= -f2- | sed -E 's#https://([^/]+)/#\1#')
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://$H/api/groups" openssl x509 -in /etc/ssl/gpt-load/gpt-load.crt -noout -subject -enddate -ext subjectAltName

301 https://ec2-44-204-33-219.compute-1.amazonaws.com/api/groups subject=CN = ec2-44-204-33-219.compute-1.amazonaws.com, O = sac notAfter=Jan 22 00:15:18 2029 GMT X509v3 Subject Alternative Name: DNS:ec2-44-204-33-219.compute-1.amazonaws.com, IP Address:44.204.33.219, IP Address:172.31.89.206, IP Address:127.0.0.1, DNS:localhost

To use your own name, point a DNS record at the instance. An Elastic IP keeps the address stable across stops. Then run the helper. It records the name in /etc/sac/gpt-load-hostname, issues a new self-signed certificate for it, updates the allowed origins and the credentials file, and reloads nginx. Afterwards, update Settings > Project address in the dashboard to match:

sudo gpt-load-set-hostname <your-dns-name>

For a publicly trusted certificate, either put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate, or replace /etc/ssl/gpt-load/gpt-load.crt and gpt-load.key with your own and run sudo systemctl reload nginx.

Run the checks below. Both services carry the metadata block. nginx refuses cross-origin writes and requests for any hostname it was not configured with. GPT-Load records the name of each setting it initializes but never its value:

systemctl show -p IPAddressDeny gpt-load.service nginx.service
H=$(sudo grep '^GPTLOAD_URL=' /root/gpt-load-credentials.txt | cut -d= -f2- | sed -E 's#https://([^/]+)/#\1#')
K=$(sudo grep '^GPTLOAD_AUTH_KEY=' /root/gpt-load-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'cross-origin POST: %{http_code}\n' --cacert /etc/ssl/gpt-load/gpt-load.crt -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -X POST -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{}' "https://$H/api/groups" code=$(curl -s -o /dev/null -w '%{http_code}' -H 'Host: evil.example' http://127.0.0.1/ || true); echo "unknown Host: ${code} (connection closed by nginx)"
sudo journalctl -u gpt-load.service --no-pager -o cat | grep 'Initialized system setting: proxy_keys' | tail -1 | sed -E 's/^time="[^"]*" //'

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128 IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128 cross-origin POST: 403 unknown Host: 000 (connection closed by nginx) level=info msg="Initialized system setting: proxy_keys"

The SecureAnyCloud build also proves, before the image is captured, that a group whose upstream is http://169.254.169.254 fails at the connection while a normal upstream answers, and that a request carrying a forged X-Forwarded-For is logged with the caller's real address by both nginx and GPT-Load.

The file /var/lib/gpt-load/data/gpt-load.db holds the whole application state: your groups, your encrypted key pool, the settings and the request log. Only the gpt-load user can read it:

sudo stat -c '%a %U:%G %n' /var/lib/gpt-load/data
sudo ls /var/lib/gpt-load/data

700 gpt-load:gpt-load /var/lib/gpt-load/data gpt-load.db

There is no database server and no cache server on this instance, so nothing else listens and nothing else needs patching.

To back up, stop the relay and copy the directory together with the credentials file, because the database is unreadable without the encryption key. Alternatively, take an EBS snapshot of the instance:

sudo systemctl stop gpt-load.service
sudo tar czf /root/gpt-load-backup-$(date +%F).tar.gz /var/lib/gpt-load/data /root/gpt-load-credentials.txt
sudo systemctl start gpt-load.service

GPT-Load is MIT licensed, and the upstream license ships with the image. The binary carries the upstream commit it was compiled from, and the build provenance records the exact dependency changes SecureAnyCloud made and the vulnerability scan of the shipped binary:

head -3 /usr/share/gpt-load/LICENSE.upstream LC_ALL=C grep -a -o 'vcs\.revision=[0-9a-f]\{40\}' /usr/local/bin/gpt-load | head -1 cat /usr/share/gpt-load/build-provenance/go.mod.diff grep -E '^Vulnerability #|affected' /usr/share/gpt-load/build-provenance/govulncheck.txt

MIT License Copyright (c) 2024 GPT-Load Contributors vcs.revision=18e8b7893d6b5da0ba43178d21fccba82a4674c0 -github.com/klauspost/compress v1.18.1 -go 1.25.0 -golang.org/x/crypto v0.52.0 -golang.org/x/net v0.55.0 // indirect -golang.org/x/sync v0.21.0 // indirect -golang.org/x/sys v0.45.0 // indirect -golang.org/x/text v0.39.0 +github.com/klauspost/compress v1.18.7 +go 1.26.0 +golang.org/x/crypto v0.57.0 +golang.org/x/net v0.60.0 // indirect +golang.org/x/sync v0.23.0 // indirect +golang.org/x/sys v0.48.0 // indirect +golang.org/x/text v0.42.0 Vulnerability #1: GO-2026-5932 Your code is affected by 1 vulnerability from 1 module.

Those seven lines are the only change to upstream's go.mod. The golang.org/x/net and klauspost/compress modules were raised to their fixed releases, and Go's version selection requires the other golang.org/x modules and the go directive to follow. /usr/share/gpt-load/build-provenance/BUILD-PROVENANCE.txt lists the toolchain and every pinned download. GO-2026-5932 is in golang.org/x/crypto/openpgp, which GPT-Load does not use and for which no fixed version exists.

Every dependency compiled into the binary, the Go modules and the dashboard's npm packages, was license-checked at build time. They are all permissive. There is no AGPL, SSPL, Business Source, Elastic License, Commons Clause or non-commercial component:

awk -F'\t' '$1=="go"||$1=="npm"{print $4}' /usr/share/gpt-load/dependency-licences.txt | sort | uniq -c | sort -rn | head -6

290 MIT 20 Apache-2.0 15 BSD 13 ISC 10 BSD-2-Clause 3 BSD-3-Clause