Deployment guide

LocalAI

Overview

LocalAI is a free, open-source, self-hosted drop-in replacement for the OpenAI API. It exposes the same OpenAI-compatible REST API for chat completions, completions and embeddings. It runs entirely on your own infrastructure with no external API calls and no GPU required, using the llama.cpp GGUF backend for inference on commodity CPU.

The SecureAnyCloud image installs LocalAI 4.4.3 as the single local-ai binary at /usr/local/bin/local-ai. It runs as a dedicated localai system user, bound to loopback behind an nginx reverse proxy on TCP 80 with HTTP Basic auth. All models are stored on a dedicated Azure data disk. A small instruct model is pre-pulled, so the API and the web UI work the moment the VM boots. A unique web password is generated on the first boot of every VM.

What you get:

LocalAI 4.4.3 at /usr/local/bin/local-ai
The built-in web UI and the OpenAI-compatible REST API (/v1/chat/completions, /v1/embeddings, /v1/models and more), fronted by nginx on :80
nginx HTTP Basic auth (LocalAI has no built-in authentication) with a per-VM password in a root-only file
A small, CPU-friendly instruct model (smollm2-135m-instruct), pre-pulled so the chat API and the web UI work out of the box with no internet access
The CPU llama.cpp inference backend, pre-pulled so inference runs offline with no GPU
A dedicated Azure data disk at /var/lib/localai holding the models directory, separate from the OS disk and re-provisioned with every VM
localai.service and nginx.service as systemd units, enabled and active
24/7 SecureAnyCloud support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet and subnet in the target region
Standard_B4ms (4 vCPU / 16 GiB RAM) as a good starting point for CPU inference. Scale up for larger models or higher throughput
NSG inbound rules that allow 22/tcp from your management network and 80/tcp for the web UI and API. Front the VM with TLS for public exposure (see Enabling HTTPS)

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for LocalAI by SecureAnyCloud, and select Create.

On the Basics tab, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab. Then choose Review + create, followed by Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:localai-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name localai \
  --image sac1647283583153:localai-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name localai --port 80 --priority 1010

Step 3 - Connect to your VM

Connect over SSH as azureuser.

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Check that both services are active.

systemctl is-active localai.service nginx.service

Both services report active. LocalAI loads the pre-pulled model on first boot and is ready to serve the OpenAI-compatible API immediately.

Step 5 - Retrieve your web password

The admin password is generated uniquely on the first boot of your VM and written to a root-only file. Read the file.

sudo cat /root/localai-credentials.txt

The file contains localai.user (admin) and localai.password, plus the URL for the web UI. Store the password somewhere safe.

Step 6 - Check the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes. Request it.

curl -s http://localhost/health

The endpoint returns ok.

Step 7 - Open the web UI

Browse to http://<vm-public-ip>/ and sign in as admin with the password from Step 5. The home page shows system status, the pre-pulled model loaded and ready, and a chat box. Use the left nav to reach Chat, Install Models, Studio and Talk.

Open Chat in the left nav, pick the smollm2-135m-instruct model, type a message and send it. The model runs on the VM CPU and streams its reply.

Step 8 - Call the OpenAI compatible API

The OpenAI-compatible REST API is served on the same port 80, behind the same Basic auth. List the available models.

curl -s -u admin:<LOCALAI_ADMIN_PASSWORD> http://localhost/v1/models | jq .

You get a JSON response whose data array includes the pre-pulled smollm2-135m-instruct model. A request without credentials returns 401.

Run a chat completion against the pre-pulled model.

curl -s -u admin:<LOCALAI_ADMIN_PASSWORD> http://localhost/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"smollm2-135m-instruct","messages":[{"role":"user","content":"In one sentence, what is Azure?"}],"max_tokens":64,"temperature":0.2}' | jq -r '.choices[0].message.content'

The model running locally on the VM CPU creates the response. Point any OpenAI client library or tool at http://<vm-public-ip>/v1 with HTTP Basic credentials. Change only the base URL to run existing applications against models on your own hardware.

Step 9 - Generate embeddings

LocalAI also serves the OpenAI-compatible embeddings endpoint. Install an embedding model such as bert-embeddings from the gallery (see Step 10). Then request a vector.

curl -s -u admin:<LOCALAI_ADMIN_PASSWORD> http://localhost/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"bert-embeddings","input":"The quick brown fox"}' | jq '.data[0].embedding | length'

The response carries the embedding vector under data[0].embedding.

Step 10 - Install more models from the gallery

The image ships one small model so the API works out of the box. LocalAI's gallery offers hundreds more. In the web UI, open Install Models in the left nav, search the gallery, and select install on any model.

You can also install a model from the command line with the model apply API.

curl -s -u admin:<LOCALAI_ADMIN_PASSWORD> http://localhost/models/apply \
  -H 'Content-Type: application/json' \
  -d '{"id":"localai@qwen2.5-0.5b-instruct"}'

Installed models are downloaded into the models directory on the dedicated data disk and appear in /v1/models once ready. Larger models need more RAM and CPU, so choose a VM size to match.

Step 11 - Confirm models live on the dedicated disk

All models are stored on the dedicated Azure data disk, so they survive OS changes and can be resized independently. Check the mount.

findmnt /var/lib/localai

The mount is backed by a separate Azure data disk that is captured into the image and re-provisioned on every VM. The pre-pulled model and the CPU inference backend live under /var/lib/localai/models and /var/lib/localai/backends.

Enabling HTTPS

The nginx reverse proxy serves plain HTTP on port 80. For public exposure, put a certificate in front of it. The simplest approach is to add a DNS name for the VM and use the companion SecureAnyCloud nginx-ssl-certbot image as a TLS reverse proxy. Alternatively, install certbot and extend the existing nginx site with a listen 443 ssl; server block and your certificate paths.

Keep LocalAI itself bound to loopback, so the only public surface is the authenticated, TLS-terminated proxy.

Maintenance

Configuration: The runtime settings (models path, backends path, loopback address, context size, threads) are set in /etc/localai/localai.env. Edit the file, then run sudo systemctl restart localai to apply changes.
Backups: Snapshot the /var/lib/localai data disk to preserve your installed models.
Upgrades: Replace /usr/local/bin/local-ai with a newer release and restart the service.
Security patches: unattended-upgrades remains enabled, so the OS continues to receive security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.

LocalAI is a trademark of its respective owner. SecureAnyCloud produces this image and is not affiliated with or endorsed by the LocalAI project. LocalAI is distributed under the MIT License.

Deploy on Azure

Launch LocalAI on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

LocalAI is the free, open-source, self-hosted alternative to the OpenAI API. It exposes the same OpenAI-compatible REST API for chat completions, completions, embeddings and more. It runs entirely on your own infrastructure with no external API calls and no GPU required.

What you get:

The LocalAI binary under /opt/localai, running as a dedicated unprivileged localai system account under a systemd service that starts it on boot and restarts it on failure
The models directory and the CPU inference backend at /var/lib/localai, on a dedicated, independently resizable EBS data volume
A small, CPU-friendly instruct model named smollm2-135m-instruct, pre-pulled so the chat API and the web interface work the moment the instance boots
An nginx reverse proxy that publishes the web UI and the OpenAI-compatible API on port 80. The inference server binds to the loopback interface only and is never exposed directly
HTTP Basic authentication on the web UI, and a LocalAI API key, sent as a bearer token, on the API

The LocalAI web UI ships with no built-in authentication, which is why nginx provides it. Both the admin password and the API key are generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share credentials. They are written to /root/localai-aws-credentials.txt with mode 0600, so only the root user can read them.

Prerequisites

Before you deploy this image, you need:

An Amazon Web Services account where you can launch EC2 instances
IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
An EC2 key pair in the target Region for SSH access to the instance
A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the web UI and the API
The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for LocalAI. Select the SecureAnyCloud listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.large or larger. LocalAI runs CPU inference, which benefits from memory and vCPUs. Choose your EC2 key pair under Key pair (login).

Under Network settings, select your VPC and subnet. Create or select a security group that opens port 22 from your management network and port 80 for the web UI and the API. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialization takes a few seconds after the instance state becomes Running and the status checks pass.

Launch the Instance from the AWS CLI

Launch an instance from the SecureAnyCloud LocalAI Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=localai}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name. Find it in the EC2 console or with aws ec2 describe-instances.

Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant. For Ubuntu 24.04, the SSH login user is ubuntu.

ssh -i <key-name>.pem ubuntu@<public-ip>

Retrieve the Admin Password and API Key

The web UI admin password and the LocalAI API key are unique to your instance. First boot generated them. Read them as root.

sudo cat /root/localai-aws-credentials.txt

The file lists the web UI URL, the admin user and the generated password, and the LocalAI API key, along with example API calls. Keep these credentials somewhere safe.

Sign In to the Web UI

nginx serves the LocalAI web UI on port 80, behind HTTP Basic authentication. In a browser, go to this address.

http://<instance-public-ip>/

When prompted for credentials, sign in as admin with the password from the credentials file. The pre-pulled model is already loaded and selected, so you can start chatting immediately. nginx supplies the API key to the UI for you.

The Home dashboard shows live system status, the loaded model and quick links to install more models or open the chat assistant.

The Install Models page is the model gallery. Browse it and install hundreds of additional open-source models onto the instance with one click.

Confirm LocalAI Is Running

Over SSH, confirm that the inference server and the nginx proxy are active and that the ports are listening.

sudo systemctl is-active localai nginx
sudo ss -tlnp | grep -E ':(80|8080) '

Both services should report active. The inference server listens on 127.0.0.1:8080 (loopback only), and nginx listens on port 80.

Next, confirm that the server reports ready through the public liveness probe.

curl -s -o /dev/null -w 'readyz: %{http_code}\n' http://127.0.0.1/readyz

List the Available Models

The pre-pulled model appears on the OpenAI-compatible models endpoint. The LocalAI API key gates the API, so read the key from the credentials file and pass it as a bearer token. Run this on the instance.

KEY=$(sudo grep '^localai.api.key=' /root/localai-aws-credentials.txt | cut -d= -f2-)
curl -s -H "Authorization: Bearer ${KEY}" http://127.0.0.1/v1/models

You should see smollm2-135m-instruct in the returned list.

Call the Chat Completions API

LocalAI answers the OpenAI-compatible chat completions endpoint. On the instance, send a prompt to the pre-pulled model.

KEY=$(sudo grep '^localai.api.key=' /root/localai-aws-credentials.txt | cut -d= -f2-)
curl -s -H "Authorization: Bearer ${KEY}" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1/v1/chat/completions \
  -d '{"model":"smollm2-135m-instruct","messages":[{"role":"user","content":"Say hello in one short sentence."}],"max_tokens":32}'

The response is the standard OpenAI chat completion JSON, with the model's reply in choices[0].message.content.

From outside the instance, send the same request to http://<instance-public-ip>/v1/chat/completions with the same bearer token.

Point any OpenAI client library at http://<instance-public-ip>/v1 and set its API key to your LocalAI API key. Existing applications then work unchanged against the local model.

Verify the API Key Gates the API

Your per-instance API key protects the OpenAI-compatible API, and a request with a wrong key is rejected. On the instance, send a request with a wrong key.

curl -s -o /dev/null -w 'wrong key: %{http_code}\n' \
  -H 'Authorization: Bearer sk-localai-wrongkey000000' http://127.0.0.1/v1/models

The request returns 401, confirming the key is enforced. Keep the API key secret. To rotate it, edit LOCALAI_API_KEY in /etc/localai/localai.env (and the matching nginx files under /etc/nginx/conf.d/), then restart the services.

Install More Models from the Gallery

Open the Install Models page in the web UI to browse the gallery. Select Install on any model to download and configure it automatically. You can also install a model over the API. The model and its configuration are written to the models directory on the data volume.

The Data Volume

The models directory and the CPU inference backend live on a dedicated EBS volume mounted at /var/lib/localai. This keeps your models off the operating system disk and lets you resize or snapshot them independently. Confirm the mount.

df -h /var/lib/localai

To grow the models store, expand the EBS volume in the AWS console. Then grow the filesystem on the instance with sudo resize2fs on the underlying device. Larger models need more disk and memory, so size the instance and the volume to the models you intend to run.

Enable HTTPS

nginx serves the web UI and API over plain HTTP on port 80. For production use, place them behind TLS.

Obtain a certificate for your domain, for example with a managed certificate on an Application Load Balancer in front of the instance, or with Certbot installed on the instance. Then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:8080, exactly as the bundled site does for port 80. Keep the HTTP Basic authentication and the API key in place.

Restrict the security group so ports 80 and 443 are reachable only from the networks that use the service.

Backup and Maintenance

Back up your models and configuration by snapshotting the /var/lib/localai EBS volume.

Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade, and reboot when a new kernel is installed. LocalAI and nginx start automatically on boot.

Support

SecureAnyCloud publishes and supports this image. Support covers deployment, model installation and configuration, the OpenAI-compatible API, prompt templates, embeddings, the model gallery, TLS and performance tuning. Contact SecureAnyCloud through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch LocalAI on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS