Deployment guide

Infinity

Overview

Infinity is a high-throughput, low-latency REST server for text embeddings, reranking and CLIP models. It serves models such as BGE, GTE, E5, Sentence Transformers and mixedbread with dynamic batching and an OpenAI-compatible embeddings API (/embeddings), so existing OpenAI SDK code works unchanged.

This SecureAnyCloud image runs Infinity on Ubuntu 24.04 LTS on Azure. It installs the pinned Infinity release (infinity-emb) into a dedicated Python virtual environment at /opt/infinity/venv and runs it under systemd.

The Infinity server binds to loopback only (127.0.0.1:7997) and has no authentication of its own. nginx is the only network-facing surface. It terminates TLS on port 443 and enforces HTTP Basic authentication against a bcrypt htpasswd file.

No anonymous access. Infinity ships with no built-in authentication, so an open port is an open embeddings API for anyone who can reach it. On the first boot, this image generates a per-VM admin credential and writes it into the nginx htpasswd file. It also regenerates a fresh self-signed TLS certificate for this specific VM, with Subject Alternative Names that include the VM public IP and hostname.

Anonymous requests to the API return HTTP 401 with a WWW-Authenticate: Basic challenge, and only the per-VM credential is accepted. The open GET /health endpoint stays available for load balancer probes. The credential is written to /root/infinity-credentials.txt (mode 0600, root only).

CPU capable, GPU ready. The image generates embeddings on CPU on any VM size and ships PyTorch's bundled CUDA runtime. It can offload to the GPU when launched on an NC or ND-series GPU VM with the NVIDIA driver present (see Step 9). A small open-weights model (BAAI/bge-small-en-v1.5, 384 dimensions) is pre-downloaded, so the API responds immediately.

What you get:

Infinity installed into a Python virtual environment at /opt/infinity/venv, run under systemd as the unprivileged infinity system user, with the model cache at /var/lib/infinity/hf
The Infinity server bound to loopback only (127.0.0.1:7997), never directly exposed to the network
nginx terminating TLS on :443 and enforcing htpasswd Basic auth on the API. GET /health and /healthz stay open for health probes
Port :80 returns a 301 redirect to HTTPS for every path except an unauthenticated /healthz endpoint (HTTP 200)
A per-VM admin credential generated on first boot and stored in /root/infinity-credentials.txt (0600), with no anonymous access and no shared default password
A self-signed TLS certificate regenerated per VM on first boot (SAN includes the VM public IP and hostname). Replace it with your own CA-signed certificate for production
A pre-downloaded BAAI/bge-small-en-v1.5 model so the API answers on launch. Serve a different model by editing one line of configuration
A best-effort update to the latest Infinity release on first boot. The update is offline-safe, because the image ships a working pinned build regardless
Ubuntu 24.04 LTS base with the latest security patches applied at build time
Azure Linux Agent for cloud integration and SSH key injection
24/7 SecureAnyCloud support with a guaranteed 24-hour response SLA

Prerequisites

An active Azure subscription, an SSH public key, and a virtual network and subnet in the target region
A subscription to the Infinity listing on Azure Marketplace

Recommended virtual machine size: for high-throughput embedding workloads, launch on a GPU VM. Standard_NC4as_T4_v3 (4 vCPU, 28 GB RAM, one NVIDIA T4) is a good entry point. Larger NC and ND-series VMs serve bigger models and more concurrency.

The image also runs on CPU-only sizes such as Standard_B4ms or Standard_D4s_v5, for light use and testing, at lower throughput. See Step 9 to enable GPU acceleration.

Deploy from the Azure Portal

Search the Marketplace for Infinity, select the SecureAnyCloud publisher, and select Create.

Add these network security group (NSG) rules, allowing your client networks:

TCP 22 for administration
TCP 443 for the API over HTTPS
TCP 80 for the HTTP to HTTPS redirect and the health probe

Deploy from the Azure CLI

RG="infinity-prod"; LOCATION="eastus"; VM_NAME="infinity-01" IMAGE_URN="sac1647283583153:infinity-ubuntu-24-04:default:latest" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create -g "$RG" --name infinity-vnet --address-prefix 10.100.0.0/16 --subnet-name infinity-subnet --subnet-prefix 10.100.1.0/24 az network nsg create -g "$RG" --name infinity-nsg az network nsg rule create -g "$RG" --nsg-name infinity-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name infinity-nsg --name allow-https --priority 110 \
  --destination-port-ranges 443 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name infinity-nsg --name allow-http --priority 120 \
  --destination-port-ranges 80 --access Allow --protocol Tcp
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:infinity-ubuntu-24-04:default:latest
az vm create -g "$RG" --name "$VM_NAME" --image "$IMAGE_URN" \
  --size Standard_NC4as_T4_v3 --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name infinity-vnet --subnet infinity-subnet --nsg infinity-nsg --public-ip-sku Standard

Connect via SSH and retrieve your credential

First boot generates the per-VM admin credential and stores it in a root-only file. Connect over SSH and read it:

ssh azureuser@<public-ip>
sudo cat /root/infinity-credentials.txt

The file contains the API URL, the served model, the username (admin), the per-VM password, and quick-start commands. Store the password somewhere safe, because it is unique to this VM.

Verify the Infinity and nginx services

Confirm that both services are active and that Infinity is bound to loopback only.

In the listener output, 127.0.0.1:7997 (Infinity) is loopback only, while nginx listens publicly on :443 and :80:

systemctl status infinity nginx --no-pager ss -tln | grep -E ':7997|:443|:80 '

Check the health endpoint

nginx serves GET /health without authentication, which suits load balancer and Azure health probes:

curl -s -o /dev/null -w 'health: HTTP %{http_code}\n' --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/health
curl -k -o /dev/null -w 'healthz: HTTP %{http_code}\n' https://127.0.0.1/healthz

Both return HTTP 200.

Generate an embedding and confirm that anonymous access is rejected

Use the per-VM credential to generate an embedding through the API. Then confirm that an anonymous request (no credential) is rejected with HTTP 401.

The self-signed certificate lives at /etc/nginx/tls/infinity.crt, so pass it with --cacert:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
MODEL=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
echo "# Authenticated embedding (vector dimension):"
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/embeddings \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"sac makes self-hosted AI easy\"]}" | jq '.data[0].embedding | length' echo "# Anonymous request (no credential) must be rejected:" curl -s -o /dev/null -w 'anonymous /embeddings -> HTTP %{http_code}\n' --cacert /etc/nginx/tls/infinity.crt \
  -X POST https://127.0.0.1/embeddings -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"x\"]}"

The authenticated request returns the embedding length (384 for the default model). The anonymous request returns HTTP 401. This API has no anonymous access.

Batch embeddings for semantic search

Infinity embeds a whole batch in a single call with dynamic batching. You can vectorise a query and its candidate documents together, and then rank the documents by cosine similarity to the query. Send several inputs at once and confirm that you get one vector per input:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
MODEL=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-) sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/embeddings \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"What is the capital of France?\",\"Paris is the capital of France.\",\"Berlin is a city in Germany.\"]}" \
  | jq '{vectors: (.data | length), dim: (.data[0].embedding | length), tokens: .usage.total_tokens}'

The response contains one embedding per input (three vectors of 384 dimensions here). In your application, compute the cosine similarity between the query vector and each document vector to rank documents. The query matches the France sentence far more closely than the Germany one.

Reranking and classification. Infinity also serves /rerank (cross-encoder reranking) and /classify (sequence classification). Those endpoints require a model with the matching capability, and the shipped BAAI/bge-small-en-v1.5 is embedding-only.

To serve a reranker (for example mixedbread-ai/mxbai-rerank-base-v1 or BAAI/bge-reranker-base) or a classifier, edit MODEL in /etc/infinity/infinity.env and restart the service (Step 13). The corresponding endpoint then becomes available.

List the served models

The /models endpoint lists the models that Infinity is serving. It is the same call that the OpenAI SDK makes for client.models.list():

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-) sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/models | jq '.'

The response is an OpenAI-style {"object":"list","data":[...]} document. It lists the served model id, its backend (torch) and its capabilities (embed).

Enable GPU acceleration (NC/ND-series VMs)

The image generates embeddings on CPU by default. On a GPU VM, you can offload embedding to the GPU for much higher throughput.

Add the NVIDIA GPU Driver Extension to the VM. In the Portal, open the VM, go to Extensions, and add NVIDIA GPU Driver Extension. Alternatively, run az vm extension set --name NvidiaGpuDriverLinux --publisher Microsoft.HpcCompute .... Reboot when it completes.
Confirm the driver. nvidia-smi lists the GPU.
Switch Infinity to the GPU. Edit the ExecStart line in /etc/systemd/system/infinity.service, changing --device cpu to --device cuda. Then run sudo systemctl daemon-reload && sudo systemctl restart infinity.
Check the service logs with sudo journalctl -u infinity -n 30. They show the model loading on the CUDA backend.

The same image and model cache are used. Only the device flag changes.

Use the OpenAI-compatible endpoint from your apps

Infinity exposes an OpenAI-compatible embeddings API. Point any OpenAI SDK at https://<public-ip> and use HTTP Basic auth. Infinity mounts /embeddings and /models at the root, with no /v1 prefix. Example with curl:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
MODEL=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-) sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/embeddings \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"first document\",\"second document\"]}" | jq '.data | length'

From Python (LangChain, LlamaIndex, or the OpenAI SDK), set base_url="https://<public-ip>" and pass the admin / password basic-auth pair. Feed the returned vectors into a vector database such as Weaviate or Chroma for retrieval-augmented generation.

Use your own domain and a trusted certificate

The self-signed certificate is a convenience for getting started. For production, front Infinity with your own DNS name and a CA-signed certificate.

Point a DNS record (for example infinity.example.com) at the VM public IP.
Replace /etc/nginx/tls/infinity.crt and /etc/nginx/tls/infinity.key with your CA-signed certificate and key, for example from Let's Encrypt or your own certificate authority.
Run sudo nginx -t && sudo systemctl reload nginx.

Clients then call https://infinity.example.com/ with no certificate trust step.

Server components

Runtime: Infinity (infinity-emb), pinned at build and updated on a best-effort basis to the latest release on first boot
Server binding: 127.0.0.1:7997 (loopback only, no auth at this layer)
Front proxy: nginx (TLS on :443, htpasswd Basic auth, :80 redirects to HTTPS with a 301)
Model cache: /var/lib/infinity/hf on the OS disk (BAAI/bge-small-en-v1.5 pre-downloaded)
Credential: per-VM admin password, generated on first boot, in /root/infinity-credentials.txt (0600)
TLS certificate: self-signed, regenerated per VM on first boot (SAN = public IP and hostname), in /etc/nginx/tls/
API: REST and OpenAI-compatible (/embeddings, /rerank, /classify, /models)
Operating system: Ubuntu 24.04 LTS (patched at build)
License: MIT (Infinity)

Manage the service and change the model

Use these commands to manage the service:

sudo systemctl status infinity --no-pager sudo journalctl -u infinity --no-pager -n 20
sudo systemctl reload nginx

To serve a different embedding or reranking model, edit MODEL in /etc/infinity/infinity.env. Use any HuggingFace embedding or reranking model id. Then run sudo systemctl restart infinity. The model downloads into /var/lib/infinity/hf on first use.

The model cache lives under /var/lib/infinity/hf and persists across service restarts and VM reboots. For large model libraries, attach and mount an Azure data disk at /var/lib/infinity/hf. Stop infinity, move the data, update the mount, and restart.

Security notes

Restrict the NSG. Allow TCP 443 (and 22 for administration) only from the client networks that need the API. Do not expose the API to the public internet unless you intend to.

Rotate the credential. To change the admin password, run sudo htpasswd -B /etc/nginx/.infinity.htpasswd admin, then sudo systemctl reload nginx. Update /root/infinity-credentials.txt to match.

Add more users. Grant additional accounts with sudo htpasswd -B /etc/nginx/.infinity.htpasswd <username>, then reload nginx.

Use a trusted certificate (Step 11) for production, so clients validate the endpoint without a manual trust step.

Keep the OS patched. Unattended security upgrades remain enabled on the running VM.

Support and licensing

Infinity is distributed under the MIT License. This SecureAnyCloud image bundles the unmodified upstream Infinity release.

SecureAnyCloud provides the packaging, hardening, per-VM credential and TLS automation, and 24/7 support with a guaranteed 24-hour response SLA.

Deploy on Azure

Find Infinity on Ubuntu 24.04 on the Azure Marketplace, published by SecureAnyCloud. Deploy from the Portal or the Azure CLI as shown above.

Need Help?

Email sales@secureanycloud.com for deployment help, configuration questions or licensing enquiries.

Deploy on Azure

Launch Infinity on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

This image runs Infinity 0.0.77, the high-throughput, low-latency server for text embedding and reranking models (michaelfeil/infinity, MIT), on Ubuntu 24.04 LTS with NVIDIA GPU acceleration. The NVIDIA driver is preinstalled and was verified on real hardware during the build.

Infinity serves an OpenAI-compatible /embeddings endpoint, plus /rerank and /classify, so existing OpenAI SDK code works unchanged.

The server listens on the loopback address 127.0.0.1:7997 under an unprivileged infinity system account. nginx fronts it on port 80 with HTTP Basic Authentication. The public /health endpoint stays open, while /embeddings, /rerank and /classify require the password. The default security group opens only port 22 (SSH) and port 80 (HTTP), so port 7997 is not reachable externally.

On the first boot of every deployed instance, a one-shot service generates a fresh password that is unique to that instance. It writes the password to /root/infinity-credentials.txt (mode 0600, root only).

The embedding model lives under HF_HOME=/var/lib/infinity/hf on a dedicated, independently resizable EBS data volume. A small open-weights model (BAAI/bge-base-en-v1.5, MIT) is pre-downloaded and served by default. To serve a different model, edit MODEL in /etc/infinity/infinity.env.

Prerequisites

An AWS account subscribed to this product in AWS Marketplace.
An EC2 key pair in your target region for SSH access.
A security group that allows inbound TCP 22 (SSH) from your IP and TCP 80 (HTTP) from your users.
An NVIDIA GPU instance. The recommended type is g4dn.xlarge (NVIDIA T4). Embedding models are light and run comfortably on a T4. Use g5 or g6 for higher throughput or larger models.

Connecting to your instance

For the Ubuntu 24.04 variant, the login user is ubuntu. For example: ssh -i your-key.pem ubuntu@<instance-public-ip>

Step 1 - Launch from the AWS Marketplace console

Open the product page in AWS Marketplace and choose Continue to Subscribe, then Continue to Configuration.
Select the Infinity 0.0.77 on Ubuntu 24.04 delivery option and your region, then choose Continue to Launch.
Choose a GPU instance type (g4dn.xlarge or larger), your VPC and subnet, your key pair, and the security group described above. Then launch.

Step 2 - Launch from the AWS CLI

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type g4dn.xlarge \
  --key-name your-key \
  --security-group-ids sg-xxxxxxxx \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=infinity}]'

Step 3 - Connect to your instance

ssh -i your-key.pem ubuntu@<instance-public-ip>

Step 4 - Confirm the GPU and services

Check the GPU and the services:

nvidia-smi --query-gpu=name,memory.total --format=csv,noheader systemctl is-active infinity.service nginx.service curl -s http://127.0.0.1/health -w ' HTTP %{http_code}\n'

Expected output:

Tesla T4, 15360 MiB active active {"unix":1781039921.9092817} HTTP 200

Step 5 - Retrieve your password

Read the credentials file:

sudo cat /root/infinity-credentials.txt

# Infinity Embeddings Server - generated on first boot by infinity-firstboot.service INFINITY_URL=http://<instance-public-ip>/ INFINITY_MODEL=BAAI/bge-base-en-v1.5 INFINITY_USERNAME=admin INFINITY_PASSWORD=<your-unique-password>

Step 6 - List the served model

The /health endpoint is open. Everything else requires the password (user admin).

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PASS http://127.0.0.1/models | jq -c '.data[] | {id}'

{"id":"BAAI/bge-base-en-v1.5"}

Step 7 - Generate embeddings (OpenAI-compatible)

Request an embedding:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PASS http://127.0.0.1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"BAAI/bge-base-en-v1.5","input":["the quick brown fox","retrieval augmented generation"]}' \
  | jq -c '{model, count:(.data|length), dim:(.data[0].embedding|length)}'

{"model":"BAAI/bge-base-en-v1.5","count":2,"dim":768}

From your own machine, use the instance's public IP instead of 127.0.0.1.

Step 8 - Use the OpenAI SDK

Call the endpoint from the OpenAI SDK:

from openai import OpenAI import base64 auth = base64.b64encode(b"admin:<your-unique-password>").decode() client = OpenAI(base_url="http://<instance-public-ip>", api_key="unused", default_headers={"Authorization": "Basic " + auth}) r = client.embeddings.create(model="BAAI/bge-base-en-v1.5", input=["hello world"]) print(len(r.data[0].embedding))

Feed the vectors into a vector database such as Weaviate or Chroma to build a retrieval-augmented generation pipeline.

Step 9 - Confirm GPU use

Check the GPU:

nvidia-smi --query-compute-apps=process_name,used_memory --format=csv,noheader

A Python process holding GPU memory confirms the model is served on the GPU:

/opt/infinity/venv/bin/python3, 1664 MiB

Serve a different model

Infinity serves the model named in its environment file. Edit the file and restart the service:

sudo sed -i 's#^MODEL=.*#MODEL=mixedbread-ai/mxbai-embed-large-v1#' /etc/infinity/infinity.env
sudo systemctl restart infinity.service

The model downloads into /var/lib/infinity/hf on first use. Infinity also serves reranking models (query the /rerank endpoint) and CLIP models.

Enable HTTPS

Run certbot with the nginx plugin:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

certbot edits the nginx site at /etc/nginx/sites-available/sac-infinity to add the TLS listener, and arranges automatic renewal.

Backup and maintenance

Model weights live under /var/lib/infinity/hf on a dedicated EBS volume. Snapshot that volume to back up downloaded models, or download them again by model name.
The password is in the nginx htpasswd file /etc/nginx/.infinity.htpasswd. Rotate it with sudo htpasswd /etc/nginx/.infinity.htpasswd admin.
Restart the service with sudo systemctl restart infinity.service. Read the logs with sudo journalctl -u infinity.service.

Support

SecureAnyCloud provides 24/7 technical support for this image by email and chat. Support covers Infinity deployment, model selection, GPU sizing, batching and throughput tuning, the OpenAI-compatible API, TLS termination and scaling. Contact details are on the AWS Marketplace listing.

Deploy on AWS

Launch Infinity Embeddings Server on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS