LiteLLM 1.88.0

Option A: AWS Marketplace console

• Open the SecureAnyCloud Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace. Choose View purchase options, then Subscribe. One subscription covers every version in the listing.
• When the subscription is active, choose Launch your software.
• Choose One-click launch from AWS Marketplace, or Launch from EC2 Console for the full launch wizard. In Version, choose LiteLLM 1.88.0 on Ubuntu 24.04, then choose your Region.
• Pick an instance type and your key pair. m5.large is recommended, and the software charge is $0.04 per vCPU per hour.
• Set the security group. The listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks that your applications call from.
• Choose your VPC and subnet, leave the metadata defaults (the image already requires IMDSv2), then choose Launch.

Option B: AWS CLI

Run these commands on your own workstation. They create a security group that allows SSH from your management address and HTTPS from your application network, then launch the instance. Replace the AMI ID with the one shown for your Region on the Marketplace launch page. Replace the VPC, subnet, key pair and CIDR ranges with your own:

SG=$(aws ec2 create-security-group --group-name litellm-gateway \
  --description "LiteLLM gateway" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=litellm-gateway}]'

You do not need --metadata-options, because the AMI already requires IMDSv2.

Connect as ubuntu with the key pair you chose:

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes within a minute of launch. It sets the hostname that LiteLLM is served as (the instance's public DNS name when it has one) and creates the TLS certificate for it. It also generates the keys, starts LiteLLM and nginx, and writes /root/litellm-credentials.txt. This command lists the entries in that file without showing the values:

sudo grep -E '^LITELLM_(HOST|URL|UI_USERNAME)=' /root/litellm-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/litellm-credentials.txt

LITELLM_HOST=ec2-3-84-115-65.compute-1.amazonaws.com LITELLM_URL=https://ec2-3-84-115-65.compute-1.amazonaws.com/ LITELLM_UI_USERNAME=admin LITELLM_HOST= LITELLM_URL= LITELLM_MASTER_KEY= LITELLM_UI_USERNAME= LITELLM_UI_PASSWORD=

To read your master key, run sudo cat /root/litellm-credentials.txt in your own session. LITELLM_MASTER_KEY (an sk- string) is the admin key for the whole gateway. Keep it secret and store it in your secrets manager.

Run these checks to confirm that the first-boot unit, LiteLLM and nginx are active. They also confirm that the installed version is 1.88.0 under the MIT licence, and that only sshd and nginx listen beyond loopback. LiteLLM itself listens on 127.0.0.1:4000.

systemctl is-active litellm-firstboot.service litellm.service nginx.service /opt/litellm/venv/bin/pip show litellm | grep -E '^(Name|Version|License)'
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u

active active active Name: litellm Version: 1.88.0 License-Expression: MIT 0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:4000 users:(("litellm")) 127.0.0.53%lo:53 users:(("systemd-resolve")) 127.0.0.54:53 users:(("systemd-resolve")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

LiteLLM answers on https://<LITELLM_HOST>/. LiteLLM's own auth layer refuses an anonymous call, and the master key lists the configured models (none yet):

H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
KEY=$(sudo grep '^LITELLM_MASTER_KEY=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/health/liveliness"; echo
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/v1/models"; echo
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/v1/models" -H "Authorization: Bearer $KEY"; echo

"I'm alive!" {"error":{"message":"Authentication Error, No api key passed in.","type":"auth_error","param":"None","code":"401"}} {"data":[],"object":"list"}

From your own machine, copy the instance's certificate (/etc/ssl/litellm/litellm.crt) and pass it to your client, or add it to your trust store. Any OpenAI-compatible SDK works. Set its base URL to https://<LITELLM_HOST>/v1 and its API key to the master key (or, once you add a database, a virtual key).

Define models in /etc/litellm/config.yaml. Keep provider keys in /etc/litellm/litellm-secrets.env (root-only), never in the YAML file. The shipped configuration is:

sudo grep -vE '^\s*#' /etc/litellm/config.yaml | sed '/^$/d'

model_list: [] general_settings: master_key: os.environ/LITELLM_MASTER_KEY litellm_settings: drop_params: true

For example, to offer OpenAI's gpt-4o and Anthropic's Claude Sonnet through the gateway, edit the file with sudo nano /etc/litellm/config.yaml. Replace model_list: [] with:

model_list: - model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY - model_name: claude-sonnet litellm_params: model: anthropic/claude-sonnet-4-5 api_key: os.environ/ANTHROPIC_API_KEY

Then add the keys to the secrets file and restart LiteLLM:

sudo nano /etc/litellm/litellm-secrets.env
# add one line each for OPENAI_API_KEY and ANTHROPIC_API_KEY
sudo systemctl restart litellm.service

A chat completion now goes through the gateway like any OpenAI call:

curl --cacert litellm.crt https://<LITELLM_HOST>/v1/chat/completions \
  -H "Authorization: Bearer <master key>" -H 'Content-Type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello"}]}'

Self-hosted servers work the same way. An Ollama instance in your VPC is model: ollama/llama3.1 with api_base: http://<ollama-private-ip>:11434. The LiteLLM documentation lists the providers and the parameters for each one.

Virtual keys, spend tracking and the admin UI at https://<LITELLM_HOST>/ui/ store their state in a PostgreSQL database. Virtual keys are scoped keys per application, with their own budgets, rate limits and model lists. This image does not configure a database, so out of the box the master key is the only key and the UI login reports that no database is connected. Asking for a virtual key shows the same:

H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
KEY=$(sudo grep '^LITELLM_MASTER_KEY=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/key/generate" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"models":[]}'; echo

{"error":{"message":"{'error': 'DB not connected. See https://docs.litellm.ai/docs/proxy/virtual_keys'}","type":"internal_server_error","param":"None","code":"500"}}

To turn these features on, point LiteLLM at a PostgreSQL database that you control, such as an Amazon RDS for PostgreSQL instance in the same VPC. Keep the database private to the VPC. Add the connection string to the secrets file, and follow the LiteLLM virtual keys documentation for the database client setup:

sudo nano /etc/litellm/litellm-secrets.env
# add: DATABASE_URL=postgresql://litellm:<db-password>@<rds-endpoint>:5432/litellm
sudo systemctl restart litellm.service

The UI then signs in as admin with LITELLM_UI_PASSWORD from your credentials file. Do not change LITELLM_SALT_KEY once the database holds provider credentials, because LiteLLM encrypts them with it.

Port 80 redirects to HTTPS on the configured hostname, and the certificate covers that name, the instance's public and private addresses, localhost and 127.0.0.1:

H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://$H/v1/models" openssl x509 -in /etc/ssl/litellm/litellm.crt -noout -subject -enddate -ext subjectAltName

301 https://ec2-3-84-115-65.compute-1.amazonaws.com/v1/models subject=CN = ec2-3-84-115-65.compute-1.amazonaws.com, O = sac notAfter=Jan 21 11:37:14 2029 GMT X509v3 Subject Alternative Name: DNS:ec2-3-84-115-65.compute-1.amazonaws.com, IP Address:3.84.115.65, IP Address:172.31.85.155, DNS:ip-172-31-85-155.ec2.internal, IP Address:127.0.0.1, DNS:localhost

To use your own name, point a DNS record at the instance. An Elastic IP keeps the address stable across stops. Then run the helper. It records the name in /etc/sac/litellm-hostname, issues a new self-signed certificate for it, updates the credentials file and reloads nginx:

sudo litellm-set-hostname <your-dns-name>

For a publicly trusted certificate, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate, or replace /etc/ssl/litellm/litellm.crt and litellm.key with your own and run sudo systemctl reload nginx.

These checks confirm that the LiteLLM service carries the metadata block, and that nginx refuses cross-origin writes and requests for any hostname it was not configured with:

systemctl show -p IPAddressDeny litellm.service
H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'cross-origin POST: %{http_code}\n' --cacert /etc/ssl/litellm/litellm.crt \
  -X POST -H 'Origin: https://evil.example' "https://$H/key/generate" code=$(curl -s -o /dev/null -w '%{http_code}' -H 'Host: evil.example' http://127.0.0.1/ || true); echo "unknown Host: ${code} (connection closed by nginx)"

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128 cross-origin POST: 403 unknown Host: 000 (connection closed by nginx)