Infinity 0.0.77

Option A: AWS Marketplace console

• Open the SecureAnyCloud Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace. Choose View purchase options, then Subscribe. One subscription covers every version in the listing.
• Once the subscription is active, choose Launch your software.
• Choose One-click launch from AWS Marketplace, or Launch from EC2 Console for the full launch wizard. In Version, choose Infinity 0.0.77 on Ubuntu 24.04, then choose your Region.
• Pick an instance type and your key pair. We recommend m5.large. The software charge is $0.04 per vCPU per hour.
• Review the security group. The listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks your applications call from.
• Choose your VPC and subnet. Leave the metadata defaults, because the image already requires IMDSv2. Then choose Launch.

Option B: AWS CLI

Run this on your own workstation. It creates a security group that allows SSH from your management address and HTTPS from your application network, then launches the instance.

Replace the AMI ID with the one shown for your Region on the Marketplace launch page. Replace the VPC, subnet, key pair and CIDR ranges with your own:

SG=$(aws ec2 create-security-group --group-name infinity \
  --description "Infinity embeddings API" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=infinity}]'

You do not need --metadata-options, because the AMI already requires IMDSv2.

An embeddings API usually serves applications inside your VPC. Keep 443 closed to the internet unless clients outside the VPC need it.

Connect as ubuntu with the key pair you chose:

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes within a minute of launch. It sets the hostname that Infinity is served as (the instance's public DNS name when it has one, else its IP address) and creates the TLS certificate for it. It generates the API key and the login password, starts Infinity on loopback and loads the model. It then starts nginx and checks the credential gate.

First boot writes /root/infinity-credentials.txt (mode 0600, root). These commands show the entries in that file without the secrets:

sudo grep -E '^INFINITY_(HOST|URL|MODEL|USERNAME)=' /root/infinity-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/infinity-credentials.txt
sudo stat -c '%a %U:%G' /root/infinity-credentials.txt

INFINITY_HOST=ec2-54-210-8-252.compute-1.amazonaws.com INFINITY_URL=https://ec2-54-210-8-252.compute-1.amazonaws.com/ INFINITY_MODEL=BAAI/bge-small-en-v1.5 INFINITY_USERNAME=admin INFINITY_HOST= INFINITY_URL= INFINITY_MODEL= INFINITY_API_KEY= <set-at-first-boot>= INFINITY_PASSWORD= 600 root:root

The entries are used as follows:

• INFINITY_URL: the base URL of this instance's API
• INFINITY_MODEL: the model ID to put in each request
• INFINITY_API_KEY: the API key, sent as Authorization: Bearer <key> (the OpenAI SDK api_key)
• INFINITY_USERNAME / INFINITY_PASSWORD: the HTTP Basic login, for your browser on the /docs page

To read the secrets, run sudo cat /root/infinity-credentials.txt in your own session and store them in your secrets manager.

The first-boot unit, Infinity and nginx should be active. Only sshd and nginx listen beyond loopback. Infinity itself is on 127.0.0.1:7997, so nginx is the only way in. The open health endpoint returns a timestamp:

systemctl is-active infinity-firstboot.service infinity.service nginx.service
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u
H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" "https://$H/health"; echo

active active active 0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:7997 users:(("infinity_emb")) 127.0.0.53%lo:53 users:(("systemd-resolve")) 127.0.0.54:53 users:(("systemd-resolve")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx")) {"unix":1792456780.069389}

/health and nginx's /healthz reveal nothing sensitive, so you can safely use them as load balancer health checks.

Infinity refuses requests without a credential, and it refuses an API key or a password that is wrong by one character. It accepts this instance's own API key and password.

A cross-origin POST is refused even with the right key, an unknown Host header gets no answer, and plain HTTP redirects to the configured HTTPS name:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-) C=(-s -o /dev/null --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1") curl "${C[@]}" -w 'no credential -> %{http_code}\n' "https://$H/models" curl "${C[@]}" -w 'wrong API key -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %sX"\n' "$K") "https://$H/models" curl "${C[@]}" -w 'wrong password -> %{http_code}\n' -K <(printf 'user = "admin:%sX"\n' "$P") "https://$H/models" curl "${C[@]}" -w 'this instance API key -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") "https://$H/models" curl "${C[@]}" -w 'this instance password -> %{http_code}\n' -K <(printf 'user = "admin:%s"\n' "$P") "https://$H/models" curl "${C[@]}" -w 'cross-origin POST -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -X POST -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{}' "https://$H/embeddings" curl -sk -o /dev/null -w 'unknown Host -> %{http_code}\n' -H 'Host: evil.example' https://127.0.0.1/health || true curl -s -o /dev/null -w 'http -> %{http_code} %{redirect_url}\n' -H "Host: $H" http://127.0.0.1/models

no credential -> 401 wrong API key -> 401 wrong password -> 401 this instance API key -> 200 this instance password -> 200 cross-origin POST -> 403 unknown Host -> 000 http -> 301 https://ec2-54-210-8-252.compute-1.amazonaws.com/models

The commands pass the secrets to curl through a config on a file descriptor (-K), so they never appear on a command line, in the process list or in your shell history.

Send text to /embeddings with the API key. The response uses the OpenAI format, with one vector per input plus token usage:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
M=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" \
  -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -H 'Content-Type: application/json' "https://$H/embeddings" \
  -d "{\"model\":\"$M\",\"input\":[\"sac makes self-hosted AI easy\"]}" \
  | jq '{object, model, vectors: (.data | length), dimensions: (.data[0].embedding | length), first3: .data[0].embedding[0:3], usage}'

{ "object": "list", "model": "BAAI/bge-small-en-v1.5", "vectors": 1, "dimensions": 384, "first3": [ -0.08709850162267685, -0.047451201826334, 0.014566515572369099 ], "usage": { "prompt_tokens": 34, "total_tokens": 34 } }

Infinity embeds a whole batch in one call. Embed a query together with candidate documents, then compare the vectors with cosine similarity. The document that answers the query scores highest, although it shares few words with the query:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
M=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" \
  -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -H 'Content-Type: application/json' "https://$H/embeddings" \
  -d "{\"model\":\"$M\",\"input\":[\"What is the capital of France?\",\"Paris is the capital of France.\",\"Berlin is a city in Germany.\"]}" \
  | python3 -c ' import json, math, sys v = [d["embedding"] for d in json.load(sys.stdin)["data"]] cos = lambda a, b: sum(x * y for x, y in zip(a, b)) / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))) print(f"{len(v)} vectors of {len(v[0])} dimensions") print(f"query vs Paris sentence: {cos(v[0], v[1]):.3f}") print(f"query vs Berlin sentence: {cos(v[0], v[2]):.3f}")'

3 vectors of 384 dimensions query vs Paris sentence: 0.812 query vs Berlin sentence: 0.578

In a RAG application, you store the document vectors in a vector database (pgvector, Qdrant, Weaviate, Chroma and so on) and embed each incoming query the same way.

/models lists what Infinity is serving. It is the same call that the OpenAI SDK makes for client.models.list():

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" \
  -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") "https://$H/models" \
  | jq '{object, data: [.data[] | {id, backend, capabilities}]}'

{ "object": "list", "data": [ { "id": "BAAI/bge-small-en-v1.5", "backend": "torch", "capabilities": [ "embed" ] } ] }

The bundled model can embed. /rerank and /classify need a reranker or classifier model (Step 10).

Infinity serves the OpenAI paths at the root, with no /v1 prefix. Point the OpenAI Python SDK at the instance and pass the API key.

The client needs to trust the self-signed certificate. Copy /etc/ssl/infinity/infinity.crt from the instance to your client, or install your own certificate (Step 9):

pip install openai httpx

import httpx from openai import OpenAI client = OpenAI( base_url="https://<your-infinity-host>",
# INFINITY_URL without the trailing slash api_key="<your INFINITY_API_KEY>", http_client=httpx.Client(verify="infinity.crt"),
# the instance's certificate ) resp = client.embeddings.create(model="BAAI/bge-small-en-v1.5", input=["first document", "second document"]) print(len(resp.data), len(resp.data[0].embedding)) # 2 384

LangChain's OpenAIEmbeddings and LlamaIndex's OpenAIEmbedding take the same base_url (or api_base) and api_key. Tools that speak only HTTP Basic can use admin and INFINITY_PASSWORD instead.

The interactive API page

Infinity's built-in OpenAPI page is at https://<your-infinity-host>/docs. Your browser asks for the login. Use admin and INFINITY_PASSWORD. The page lists every route.

Open POST /embeddings, choose Try it out, enter a request body and choose Execute to see the vectors that Infinity returns.

GET /models shows the served model with its batch statistics.

The page loads its scripts and styles (Swagger UI) from the jsDelivr CDN in your browser, so the browser needs internet access. The API itself does not.

Point a DNS record at the instance (an Elastic IP keeps the address stable), then tell Infinity the name.

The helper re-issues the self-signed certificate for the name and updates nginx's accepted names. It records the name in /etc/sac/infinity-hostname, which persists across reboots, and updates INFINITY_URL:

sudo infinity-set-hostname embeddings.example.com

infinity-set-hostname: Infinity is served at https://embeddings.example.com/

To use a certificate from your own CA or from Let's Encrypt instead, replace /etc/ssl/infinity/infinity.crt and /etc/ssl/infinity/infinity.key. Keep the key at mode 0600, owner root. Then run sudo systemctl reload nginx.

The served model is the MODEL line in /etc/infinity/infinity.env. To serve a reranker, a classifier or a larger embedding model, change it to the model's Hugging Face ID and restart Infinity.

The instance downloads the model on that restart, so it needs outbound HTTPS to huggingface.co for that one start:

sudo sed -i 's|^MODEL=.*|MODEL=BAAI/bge-reranker-base|' /etc/infinity/infinity.env
sudo systemctl restart infinity.service
sudo journalctl -u infinity.service -n 20 --no-pager

/rerank then answers with relevance scores.

Serve only models from publishers you trust, because loading a model runs its configuration and weights through transformers and PyTorch. Larger models need more memory, so check the model card and size the instance to match.

The first-boot gate uses the configured model. Set the model you want before you create an image of your own instance.

First boot made the certificate for this instance, and both services are denied the instance metadata service. The last line runs a metadata request from inside Infinity's own service control group, exactly as a request handled by Infinity would:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
openssl s_client -connect 127.0.0.1:443 -servername "$H" </dev/null 2>/dev/null | openssl x509 -noout -subject -enddate systemctl show -p IPAddressDeny --value infinity.service nginx.service
CG=$(systemctl show -p ControlGroup --value infinity.service)
sudo bash -c "echo \$\$ > /sys/fs/cgroup$CG/cgroup.procs; exec curl -s -m 5 -o /dev/null -w 'metadata from inside infinity.service -> %{http_code}\n' -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' http://169.254.169.254/latest/api/token" || true

subject=CN = ec2-54-210-8-252.compute-1.amazonaws.com, O = sac notAfter=Jan 22 00:37:14 2029 GMT 168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 metadata from inside infinity.service -> 000

Confirm that the installed release is the one in the version title, that PyTorch is the CPU build with no CUDA runtime and no NVIDIA packages, and that telemetry is off:

/opt/infinity/venv/bin/pip show infinity-emb | grep -E '^(Name|Version|License)' /opt/infinity/venv/bin/python -c 'import torch; print("torch", torch.__version__, "cuda:", torch.version.cuda)' /opt/infinity/venv/bin/pip list --format=freeze | grep -ciE '^(nvidia|triton)' || true
sudo grep -E '^(MODEL|INFINITY_ANONYMOUS_USAGE_STATS|DO_NOT_TRACK|HF_HUB_DISABLE_TELEMETRY)=' /etc/infinity/infinity.env

Name: infinity_emb Version: 0.0.77 License: MIT torch 2.14.1+cpu cuda: None 0 MODEL=BAAI/bge-small-en-v1.5 INFINITY_ANONYMOUS_USAGE_STATS=0 DO_NOT_TRACK=1 HF_HUB_DISABLE_TELEMETRY=1

To issue a new API key, write it into nginx's key map and reload nginx. The old key stops working at once.

The instance generates the key and hands it to sudo tee on standard input, so it never appears on a sudo command line, which sudo writes to the system logs. The map file keeps its mode 0600, owner root:

NEWKEY="inf-$(openssl rand -hex 24)" printf '"Bearer %s" 1;\n' "$NEWKEY" | sudo tee /etc/nginx/infinity-apikey.map >/dev/null
sudo systemctl reload nginx printf 'New API key: %s\n' "$NEWKEY"

To change the login password, use htpasswd. It reads the password from standard input, so the password never appears on a command line, in your shell history or in the system logs. Enter the new password when prompted, then reload nginx:

sudo htpasswd -i -B /etc/nginx/.infinity.htpasswd admin
sudo systemctl reload nginx

Then put the new values in /root/infinity-credentials.txt, or delete that file, so that it does not hold stale secrets.