SecureAnyCloud
Deployment guide

LiteLLM

Videos

Overview: LiteLLM on Ubuntu 24.04 on Azure: overview (watch on YouTube)

Launch and log in: LiteLLM on Ubuntu 24.04 on Azure: launch and log in (watch on YouTube)

Deep Dive: LiteLLM on Ubuntu 24.04 on Azure: deep dive (watch on YouTube)

Overview

LiteLLM is an open-source proxy and gateway that puts a single OpenAI-compatible API in front of 100+ LLM providers, including OpenAI, Anthropic, Azure OpenAI, Bedrock, Vertex AI, Mistral and Ollama. You point your applications at one endpoint, and you can switch or load-balance providers without changing client code. LiteLLM adds virtual keys, per-key budgets, rate limits and request logging.

This image installs LiteLLM 1.88 in a Python virtual environment at /opt/litellm and fronts it with an nginx reverse proxy on TCP 80. First boot generates a unique master key for every VM. The image ships no model weights and is CPU-only.

What you get:

  • LiteLLM 1.88 proxy in a Python venv at /opt/litellm
  • nginx reverse proxy on :80 in front of the proxy (bound to loopback :4000)
  • Per-VM master key sk-… (LITELLM_MASTER_KEY) generated at first boot, in a root-only file
  • litellm.service and nginx.service as systemd units, enabled and active
  • An empty model list, ready for you to add your providers
  • 24/7 SecureAnyCloud support

Prerequisites

You need an active Azure subscription, an SSH key pair, and a virtual network with a subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is a good starting point, because LiteLLM is a lightweight gateway and not a model server.

In the network security group (NSG), allow 22/tcp inbound from your management network and 80/tcp from the clients that call the API. For public exposure, front port 80 with TLS (see Enabling HTTPS).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, and search the Marketplace for LiteLLM by SecureAnyCloud. Select Create.

  • On Basics, pick your subscription, resource group, region and size.
  • Under Administrator account, choose SSH public key and paste your key.
  • Under Inbound port rules, allow SSH (22) and HTTP (80).
  • Choose Review + create, then Create.

Step 2 - Deploy from the Azure CLI

Accept the Marketplace terms once per subscription, then create the VM.

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:litellm-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name litellm \
  --image sac1647283583153:litellm-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name litellm --port 80 --priority 1010

Step 3 - Connect to your VM

Connect over SSH as azureuser.

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Check the services, the listening ports and the liveliness endpoint.

systemctl is-active litellm.service nginx.service ss -tln | grep -E ':80 |:4000 ' curl -s http://127.0.0.1/health/liveliness

Both services report active. nginx listens on port 80, the LiteLLM proxy listens on loopback 4000, and the liveliness endpoint responds.

Step 5 - Retrieve your master key

First boot generates a master key that is unique to your VM and writes it to a root-only file. Callers pass it as a Bearer token. Print the file.

sudo cat /root/litellm-credentials.txt

The LITELLM_MASTER_KEY value (an sk-… string) is your admin and master key. Keep it secret. Use it to create scoped virtual keys for your applications (Step 8).

Step 6 - Add a provider

LiteLLM ships with an empty model list. Add your providers and credentials to /etc/litellm/config.yaml, then restart the service. This example adds an OpenAI model.

sudo tee -a /etc/litellm/config.yaml >/dev/null <<'YAML' model_list: - model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY YAML echo 'OPENAI_API_KEY=<set-at-first-boot>' | sudo tee -a /etc/litellm/litellm.env >/dev/null
sudo systemctl restart litellm

LiteLLM supports 100+ providers. See the upstream documentation for the parameters of each.

Step 7 - Call the OpenAI-compatible API

Any OpenAI-compatible client works. Point base_url at the VM and use your master key or a virtual key. List the configured models.

KEY=$(sudo grep '^LITELLM_MASTER_KEY=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s http://127.0.0.1/models -H "Authorization: Bearer $KEY"

Chat completions, embeddings and the other OpenAI routes are served at the same base URL.

Step 8 - Virtual keys, budgets and the admin UI

Use the master key to create scoped virtual keys for each application through the /key/generate API. Each virtual key has its own budget and rate limits, so you never share the master key.

LiteLLM also serves an admin UI for keys, spend and logs.

Step 9 - Confirm the runtime

Check the installed LiteLLM version.

/opt/litellm/venv/bin/pip show litellm | grep ^Version

Enabling HTTPS

For production, terminate TLS at nginx with a real domain pointed at the VM's public IP. Install certbot and request a certificate, replacing the domain with your own.

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

Backup and maintenance

Your configuration lives in /etc/litellm/. Keep the OS patched with sudo apt update && sudo apt upgrade. Restart the proxy after configuration changes with sudo systemctl restart litellm. The proxy holds no model weights, so the image stays small and starts fast.

Support

SecureAnyCloud provides 24/7 support for this image. Contact us by email or chat for help with provider integration, virtual keys and budgets, load balancing, TLS and scaling.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on Azure

Launch LiteLLM on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure