Ollama AnythingLLM

• In the Azure Portal, search the Marketplace for Ollama + AnythingLLM on Ubuntu 24.04 LTS and select Create.
• Choose your subscription, resource group and region.
• Set the VM size to Standard_B4ms or larger.
• Under Administrator account, select SSH public key and supply your own key.
• Under Inbound port rules, allow HTTPS (443), HTTP (80) and SSH (22). Then restrict all three to your own address ranges on the networking blade.
• Select Review + create.

az vm image terms accept --urn sac1647283583153:ollama-anythingllm-ubuntu-24-04:default:latest
az vm create \
  --resource-group my-resource-group \
  --name my-private-ai \
  --image sac1647283583153:ollama-anythingllm-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --public-ip-sku Standard
# Open HTTPS (and the HTTP redirect) to YOUR network only. MY_CIDR="203.0.113.0/24"
# replace with your own range
az network nsg rule create \
  --resource-group my-resource-group \
  --nsg-name my-private-aiNSG \
  --name allow-https-from-office --priority 1010 \
  --source-address-prefixes "${MY_CIDR}" \
  --destination-port-ranges 80 443 --protocol Tcp --access Allow

ssh azureuser@<vm-ip>

First boot takes about a minute. It generates this VM's admin password, secrets and certificate, starts the services and proves the sign-in gate before it finishes. Check that every service is active:

# Every unit reports "active". The firstboot and firewall units are oneshots that stay active (exited).
systemctl is-active ollama-anythingllm-firstboot ollama anythingllm anythingllm-collector nginx ollama-anythingllm-firewall
# Ollama listens on loopback only; nginx is the only public listener.
sudo ss -lntp | grep -E ':(80|443|11434)\b'

Expected output:

active active active active active active LISTEN 0 4096 127.0.0.1:11434 0.0.0.0:* users:(("ollama",...)) LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",...)) LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",...)) LISTEN 0 511 [::]:80 [::]:* users:(("nginx",...)) LISTEN 0 511 [::]:443 [::]:* users:(("nginx",...))

First boot generated everything in this file on this VM. Read it with:

sudo cat /root/ollama-anythingllm-credentials.txt

# Ollama + AnythingLLM - per-VM credentials generated on first boot (...).
# Keep this file private. Change the password in AnythingLLM: Settings > Users (or your profile). ANYTHINGLLM_URL=https://<vm-ip>/ ANYTHINGLLM_USERNAME=admin ANYTHINGLLM_PASSWORD=<ANYTHINGLLM_PASSWORD>

• Open ANYTHINGLLM_URL from the credentials file in your browser. The certificate is self-signed and generated for this VM, so your browser warns you the first time. Accept it for this address, or follow Step 12 to use your own domain and a trusted certificate.
• Sign in as admin with ANYTHINGLLM_PASSWORD.
• The first time you sign in, AnythingLLM shows a set of recovery codes for the admin account. Download them and keep them somewhere safe. Together with the username, they let you reset the password if you lose it. AnythingLLM shows them only once.

After you sign in, you land on the AnythingLLM home page. Workspaces are listed in the sidebar.

The settings (the spanner icon at the bottom of the sidebar) hold the model, embedding, vector database, user and security configuration.

• Sign-in. AnythingLLM runs in multi-user mode. Every page and API route needs a signed-in user, except the sign-in page itself and a few public status endpoints. First boot created the only account, admin, with a password stored as a bcrypt hash in AnythingLLM's database. The plain text exists only in the root-only credentials file. Add users for your team under Settings > Users, with the default, manager or admin role.
• Network. nginx is the only public listener. It serves AnythingLLM on port 443 with a certificate generated for this VM, and port 80 only redirects to HTTPS. AnythingLLM's server (3001) and document collector (8888) cannot be told to bind to loopback, so an nftables table drops traffic to them, and to Ollama (11434), from anything but the VM itself. Ollama also binds to 127.0.0.1 only.
• Brute-force protection. nginx limits sign-in attempts to ten a minute per client.
• Privacy. AnythingLLM's anonymous telemetry is switched off (DISABLE_TELEMETRY="true") and Ollama runs with OLLAMA_NO_CLOUD=1. The models on this VM process your chats and documents.

You can check all of it from the VM itself:

code() { curl -sk -o /dev/null -w '%{http_code}' "$@"; } token() { curl -sk -H 'Content-Type: application/json' -d "{\"username\":\"admin\",\"password\":\"$1\"}" https://127.0.0.1/api/request-token; }
# Multi user mode is on, and anonymous callers are refused (401).
curl -sk https://127.0.0.1/api/system/multi-user-mode; echo echo "anonymous /api/workspaces : $(code https://127.0.0.1/api/workspaces)"
# A wrong password is refused, this VM's password is accepted.
echo "wrong password : $(token wrong-password-123 | python3 -c 'import sys,json;print("valid:",json.load(sys.stdin)["valid"])')"
echo "per-VM password : $(token '<ANYTHINGLLM_PASSWORD>' | python3 -c 'import sys,json;print("valid:",json.load(sys.stdin)["valid"])')"

Expected output:

{"multiUserMode":true} anonymous /api/workspaces : 401 wrong password : valid: False per-VM password : valid: True

AnythingLLM is configured to use the Ollama server on this VM for both chat and embeddings. You can see the models and how the app is wired from the shell. AnythingLLM rewrites its own .env when you change settings in the web app, so the order of the lines may differ:

ollama --version ollama list
sudo grep -E '^(LLM_PROVIDER|OLLAMA_BASE_PATH|OLLAMA_MODEL_PREF|EMBEDDING_ENGINE|EMBEDDING_MODEL_PREF|VECTOR_DB|DISABLE_TELEMETRY)=' /opt/anythingllm/server/.env

ollama version is 0.40.4 NAME ID SIZE MODIFIED nomic-embed-text:latest 0a109f422b47 274 MB ... granite4:micro 89962fcc7523 2.1 GB ... DISABLE_TELEMETRY="true" LLM_PROVIDER="ollama" OLLAMA_BASE_PATH="http://127.0.0.1:11434" OLLAMA_MODEL_PREF="granite4:micro" EMBEDDING_ENGINE="ollama" EMBEDDING_MODEL_PREF="nomic-embed-text:latest" VECTOR_DB="lancedb"

The same settings are visible in the web app under Settings > AI Providers > LLM, where the provider is Ollama and the model is granite4:micro.

Ollama's own API is available to anything running on the VM. For example, ask the chat model a question directly:

curl -s http://127.0.0.1:11434/api/chat -d '{"model":"granite4:micro","stream":false, "options":{"temperature":0,"seed":7,"num_predict":120}, "messages":[{"role":"user","content":"In two sentences, what is retrieval augmented generation?"}]}' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["message"]["content"])'

The models run on the CPU. On a Standard_B4ms, a short answer takes a few seconds to start and then streams at a readable pace.

The first request after the model has been idle for an hour also loads it into memory, which takes a few seconds more.

• Select New Workspace in the sidebar and give it a name. A workspace is a separate set of documents and chats.
• Open the workspace and select the upload icon next to its name. Upload files (PDF, Word, text, Markdown, CSV and more) or paste a link. Move them into the workspace and select Save and Embed. The document collector extracts the text, and the nomic-embed-text model turns it into vectors stored in LanceDB on this VM.
• Ask a question in the chat box. AnythingLLM retrieves the most relevant passages from your documents, and the granite4:micro model answers from them. The answer lists the documents it drew on under Citations.

Each workspace has a chat mode in its settings:

• Automatic (the default for new workspaces) and Chat blend the model's general knowledge with your documents.
• Query answers only from your documents, and says so when they do not contain the answer.

Agents. Type @agent at the start of a message to hand it to AnythingLLM's agent, which can use tools such as web scraping, document summaries and saving files.

Because the models run on the CPU, this image sets PROVIDER_DISABLE_NATIVE_TOOL_CALLING="ollama" in /opt/anythingllm/server/.env. Ordinary messages are then answered directly from your documents, instead of going through the agent's tool-calling loop on every message, which is much slower on a CPU. Agents still work when you ask for them with @agent.

On a large VM or a GPU VM, you can remove that line and run sudo systemctl restart anythingllm to let Automatic mode call tools natively.

AnythingLLM has a REST API for building your own tools on top of the workspace. Create an API key under Settings > Tools > Developer API, or from the shell as shown below. The interactive API reference is at https://<vm-ip>/api/docs.

This example runs entirely on this VM. It signs in as admin, creates an API key, creates a workspace, adds a document to it and asks a question about it:

B=https://127.0.0.1/api
TOKEN=$(curl -sk -H 'Content-Type: application/json' -d '{"username":"admin","password":"<ANYTHINGLLM_PASSWORD>"}' $B/request-token | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
KEY=$(curl -sk -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"name":"guide"}' $B/admin/generate-api-key | python3 -c 'import sys,json;print(json.load(sys.stdin)["apiKey"]["secret"])')
SLUG=$(curl -sk -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"name":"API demo"}' $B/v1/workspace/new | python3 -c 'import sys,json;print(json.load(sys.stdin)["workspace"]["slug"])')
curl -sk -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"textContent\":\"The staff car park opens at 7am and closes at 9pm. Visitors park on level 2.\",\"addToWorkspaces\":\"$SLUG\",\"metadata\":{\"title\":\"car-park.txt\"}}" \
  $B/v1/document/raw-text | python3 -c 'import sys,json;d=json.load(sys.stdin);print("document added:",d["success"])'
curl -sk -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"message":"Where do visitors park?","mode":"query"}' $B/v1/workspace/$SLUG/chat \
  | python3 -c 'import sys,json;d=json.load(sys.stdin);print("answer:",d["textResponse"]);print("sources:",[s["title"] for s in d["sources"]])'

document added: True answer: Visitors park on level 2. sources: ['car-park.txt']

Developer API keys have full access. Store them like passwords, and delete the ones you no longer need under Settings > Tools > Developer API.

Ollama can run any model from its library that fits in the VM's memory. Pull a model from the shell. Then select it in Settings > AI Providers > LLM (the model list is read from Ollama), or per workspace in the workspace's Chat Settings:

# Example: a larger general purpose model (about 5 GB; needs a VM with 16 GiB or more). ollama pull qwen3:8b ollama list

• Check each model's licence before using it commercially: ollama show <model> --license.
• Models are stored under /var/lib/ollama/models. The OS disk has room for a few more small models. If you plan to keep several large models, resize the OS disk in Azure, or attach a data disk and point OLLAMA_MODELS at it in a systemd drop-in.
• To remove a model you no longer use, run ollama rm <model>.
• If you change the embedding model, AnythingLLM has to re-embed your documents, because vectors from different models are not compatible.
• On a GPU VM size (NC or ND series) with the NVIDIA driver installed, for example through the Azure NVIDIA GPU Driver extension, Ollama uses the GPU automatically. Its CUDA runtime is already included.

The self-signed certificate is fine for a trial. For regular use, point a DNS name at the VM's public IP and install a trusted certificate.

nginx already serves the Let's Encrypt HTTP-01 challenge path on port 80.

sudo apt-get update && sudo apt-get install -y certbot
sudo certbot certonly --webroot -w /var/www/html -d ai.example.com
sudo sed -i \
  -e 's#/etc/ollama-anythingllm/tls/server.crt#/etc/letsencrypt/live/ai.example.com/fullchain.pem#' \
  -e 's#/etc/ollama-anythingllm/tls/server.key#/etc/letsencrypt/live/ai.example.com/privkey.pem#' \
  /etc/nginx/sites-available/ollama-anythingllm
sudo nginx -t && sudo systemctl reload nginx

Certbot renews the certificate automatically. Add --deploy-hook "systemctl reload nginx" to the certonly command so nginx picks up each renewal.

# Restart the app (the collector restarts with it if needed) or Ollama.
sudo systemctl restart anythingllm anythingllm-collector
sudo systemctl restart ollama
# Recent logs (add -f to follow them live).
sudo journalctl -u anythingllm -u anythingllm-collector -u ollama -n 20 --no-pager
# What is installed.
cat /usr/share/ollama-anythingllm/VERSIONS

• Data. AnythingLLM keeps its database (users, workspaces, chats, settings), uploaded documents and LanceDB vectors in /opt/anythingllm/server/storage, and its configuration in /opt/anythingllm/server/.env. Back up both, together with /var/lib/ollama/models if you pulled extra models, or take Azure disk snapshots of the VM.
• Users and passwords. Add, suspend and reset users under Settings > Users. Change your own password from your profile. If you lose the admin password, use the recovery codes from your first sign-in.
• Operating system updates. Unattended security upgrades are on. Reboot when /var/run/reboot-required exists.
• Upgrading AnythingLLM or Ollama. Both are installed from pinned upstream releases. Take a snapshot first, then follow the upstream release notes. SecureAnyCloud support can help plan an upgrade.