SecureAnyCloud
Deployment guide

whisper.cpp Server

Overview

whisper.cpp is a high-performance implementation of OpenAI's Whisper automatic speech recognition model, written in plain C and C++. It runs entirely on the CPU, with no GPU, no Python runtime and no external service dependencies. This image ships the project's bundled HTTP server, so the machine becomes a private speech-to-text service the moment it boots: post an audio file and receive a transcript back.

This guide covers deploying and using whisper.cpp Server on Microsoft Azure with SecureAnyCloud's preconfigured Ubuntu 24.04 LTS virtual machine image. Because everything runs locally, audio never leaves the virtual machine you control. That suits recordings that cannot be sent to a third-party transcription service.

What you get:

  • whisper.cpp 1.9.1, built from the pinned upstream source tag
  • The bundled whisper.cpp HTTP server, listening on the loopback interface only
  • The ggml-base.en speech recognition model, preinstalled and verified by checksum
  • nginx 1.24 as an authenticating reverse proxy on port 80
  • ffmpeg, for automatic conversion of common audio and video formats
  • A short test clip, so you can prove the service works before you use your own audio
  • 24/7 SecureAnyCloud support

Key facts:

  • Platform: Microsoft Azure
  • Operating system: Ubuntu 24.04 LTS
  • Default SSH user: azureuser
  • Web interface and API: port 80, HTTP Basic authentication
  • Inference engine: 127.0.0.1:8080, loopback only, not reachable from the network
  • Model file: /var/lib/whisper-cpp-server/models/ggml-base.en.bin
  • Credentials file: /root/whisper-cpp-server-credentials.txt
  • Licence notices: /opt/whisper/

About the bundled model

whisper.cpp is a program, not a model. It needs a separate model file to do anything, and that file carries its own licence. This image ships ggml-base.en, and its provenance was traced to its root rather than assumed from a repository badge.

This image includes the ggml-base.en speech recognition model, a format conversion of OpenAI's original Whisper base.en checkpoint into the GGML format used by whisper.cpp. It is not a fine-tune and not a distillation of any third-party model. Both OpenAI's Whisper weights and whisper.cpp itself are distributed under the MIT License. Copies of both licences ship in the image at /opt/whisper/, together with a written provenance record.

The model file is pinned by name and by SHA-256 checksum at build time, so the image cannot silently ship a different model.

Prerequisites

Before you deploy, make sure you have:

  • An active Microsoft Azure subscription
  • An active Azure Marketplace subscription for this offer
  • An SSH key pair for connecting to the virtual machine
  • Permission to create virtual machines and network security groups in your target region

Recommended VM size: Standard_B2s (2 vCPU, 4 GB RAM). See Performance for what that size delivers and when to choose a larger one.

Network security group configuration

Allow these inbound ports:

  • TCP 22: SSH access
  • TCP 80: whisper.cpp web interface and transcription API, authenticated

Port 8080 is deliberately not in this list. The inference engine binds to the loopback interface only and is never reachable from the network. See Security model.

1

Deploy the virtual machine

Option A: Azure Portal

  • In the Azure Portal, go to Marketplace and search for whisper.cpp Server on Ubuntu 24.04 LTS.
  • Select the offer and click Create.
  • Choose your subscription, resource group and region.
  • Set the VM size to Standard_B2s or larger.
  • Choose SSH public key as the authentication type and set the username to azureuser.
  • Under Inbound port rules, allow SSH (22) and HTTP (80).
  • Click Review + create, then Create.

Option B: Azure CLI

az vm image terms accept --urn sac1647283583153:whisper-cpp-server-ubuntu-24-04:default:latest
az vm create \
  --resource-group my-resource-group \
  --name my-whisper-vm \
  --image sac1647283583153:whisper-cpp-server-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard \
  --location eastus

Open the web port:

az vm open-port --resource-group my-resource-group --name my-whisper-vm --port 80
2

Connect over SSH

Find the public IP address of your virtual machine:

az vm show --resource-group my-resource-group --name my-whisper-vm \
  --show-details --query publicIps --output tsv

Connect with SSH:

ssh azureuser@<vm-ip>
3

Retrieve your credentials

On the first boot of every virtual machine, this image generates a password that is unique to that machine. Nothing is shared between deployments, and no password is baked into the image. Read the credentials file with:

sudo cat /root/whisper-cpp-server-credentials.txt

Only root can read the file. It contains your username, your password and the URL of your service:

WHISPER_USER=whisper WHISPER_PASSWORD=... WHISPER_URL=http://<vm-ip>/ WHISPER_API_URL=http://<vm-ip>/inference

Confirm that the file exists with the correct ownership and permissions. The command below shows that the file is present and root-only, without printing the password itself:

sudo ls -l /root/whisper-cpp-server-credentials.txt
4

Confirm the service is running

sudo systemctl status whisper-server.service --no-pager
sudo systemctl status nginx.service --no-pager

Both units should report active (running).

Check that the inference engine is answering on the loopback interface:

curl -s -o /dev/null -w 'health: HTTP %{http_code}\n' http://127.0.0.1:8080/health

Expected output:

health: HTTP 200
5

Transcribe the bundled test clip

The image ships a short test clip so you can prove the service works end to end before you use your own audio. It was synthesised on the build machine with eSpeak NG, so no third-party audio is bundled with this image.

Transcribe it, substituting the password from Step 3:

sudo curl -s -u whisper:<your-password> \
  -F file=@/usr/local/share/whisper-cpp-server/samples/sac-test.wav \
  -F response_format=json \
  http://127.0.0.1/inference

Expected output:

{"text":" A quick brown fox jumps over the lazy dog.\n"}

If you get a transcript back containing those words, the model loaded correctly and the whole path works. An empty text value with an HTTP 200 would mean the model failed to load, so check the transcript content rather than the status code.

6

Use the browser interface

Open http://<vm-ip>/ in a browser. The browser prompts you for the username and password from Step 3.

After you sign in, the interface documents the API and provides an upload form. Choose an audio file, pick a response format and click Submit. The browser shows the transcript directly.

Note that the interface page documents a /load endpoint. That endpoint is deliberately blocked on this image and returns HTTP 403. See Security model for why.

7

Transcribe your own audio

Copy an audio file to the virtual machine:

scp <your-audio-file> azureuser@<vm-ip>:/tmp/

Then transcribe it. ffmpeg is installed and the server converts input automatically, so it accepts MP3, M4A, FLAC, OGG, WAV and the audio track of most video files:

sudo curl -s -u whisper:<your-password> \
  -F file=@/tmp/<your-audio-file> \
  -F response_format=text \
  http://127.0.0.1/inference

Response formats

Pass response_format to choose the output shape:

  • text: plain transcript text
  • json: {"text": "..."}
  • verbose_json: transcript plus per-segment and per-word timings and probabilities
  • srt: SubRip subtitles
  • vtt: WebVTT subtitles

Generate subtitles for a recording:

sudo curl -s -u whisper:<your-password> \
  -F file=@/usr/local/share/whisper-cpp-server/samples/sac-test.wav \
  -F response_format=srt \
  http://127.0.0.1/inference

Expected output:

1 00:00:00,000 --> 00:00:03,600 A quick brown fox jumps over the lazy dog.

Calling the API from another machine

Replace the loopback address with your virtual machine's public IP address and supply the same credentials:

curl -s -u whisper:<your-password> \
  -F file=@recording.mp3 -F response_format=text \
  http://<vm-ip>/inference

Performance

whisper.cpp runs on the CPU, so transcription speed scales with the number of vCPUs. These figures were measured on this image running on Standard_B2s (2 vCPU) with the bundled ggml-base.en model:

  • 3.5-second clip: approximately 2.9 seconds, dominated by fixed startup cost
  • 90.7 seconds of speech: approximately 17.4 seconds, about 5 times faster than real time

In practice, on Standard_B2s, expect roughly 5 times faster than real time for sustained audio, so a 10-minute recording takes around 2 minutes. Short clips are dominated by a fixed per-request cost of a couple of seconds.

Use this guidance to choose a size:

  • Standard_B2s is a good fit for occasional transcription, evaluation and low-volume API use. B-series machines are burstable, so sustained batch work can exhaust CPU credits and slow down.
  • For sustained or batch transcription, choose a compute-optimised size such as Standard_F4s_v2 or Standard_F8s_v2. Throughput scales roughly with vCPU count.
  • Memory is not the constraint with this model. The ggml-base.en model uses a few hundred megabytes, and 4 GB of RAM is sufficient.

Requests are processed one at a time. If you need concurrency, place a queue in front of the API or run several virtual machines behind a load balancer.

Using a different model

ggml-base.en balances accuracy and speed, and is English only. Larger models are more accurate and slower; multilingual models drop the .en suffix. To use a different one, download it and point the service at it:

sudo curl -fL -o /var/lib/whisper-cpp-server/models/ggml-small.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en.bin
sudo chown whisper:whisper /var/lib/whisper-cpp-server/models/ggml-small.en.bin

Then edit the ExecStart line in /etc/systemd/system/whisper-server.service to reference the new file, and reload the service:

sudo systemctl daemon-reload
sudo systemctl restart whisper-server.service

Check the licence of any model you add. The models in the repository above are conversions of OpenAI's original Whisper checkpoints and are MIT licensed, but models elsewhere, including community fine-tunes, may carry more restrictive terms that prohibit commercial use.

Security model

whisper.cpp's bundled server has no authentication of any kind, and the upstream project's own documentation warns against exposing it. It accepts user file uploads and shells out to ffmpeg for conversion, and its /load endpoint allows a caller to load an arbitrary server-side file path as a model. Left open on a public IP address, that is an abuse and cost surface, and transcription endpoints handle audio that is often sensitive.

This image does not expose it. The protections below are applied in the image and verified on a freshly launched virtual machine before the image is published:

  • The inference engine binds to the loopback interface only. It listens on 127.0.0.1:8080 and is not reachable from the network, whatever your network security group allows.
  • Every path requires authentication. nginx fronts the engine on port 80 and demands HTTP Basic credentials. There is no anonymous route to any endpoint.
  • The /load endpoint is blocked outright. It returns HTTP 403 even with valid credentials, so the remote model-swap primitive is unreachable rather than merely protected.
  • No default credential exists. The password is generated on each machine's first boot. Until that happens the credential store is empty, which fails closed: every request is refused.
  • The service runs unprivileged as the whisper system user, under systemd hardening including NoNewPrivileges, ProtectSystem=strict and PrivateTmp.
  • Audio and transcripts are not persisted. Uploads are handled per request, and nothing but the model file remains on disk afterwards.

You can confirm the first three points yourself. The engine must refuse a connection on the machine's own routable address:

curl -s -o /dev/null -m 5 -w '%{http_code}\n' http://$(hostname -I | awk '{print $1}'):8080/health || echo "connection refused, as expected"

A status of 000 means curl could not connect at all, which is the desired result: nothing is listening on that address.

An anonymous request must be refused:

curl -s -o /dev/null -w 'anonymous: HTTP %{http_code}\n' http://127.0.0.1/inference

A wrong password must be refused:

curl -s -o /dev/null -w 'wrong password: HTTP %{http_code}\n' -u whisper:definitely-not-the-password http://127.0.0.1/

And the model-swap endpoint must be blocked even with the correct credentials:

sudo curl -s -o /dev/null -w 'load endpoint: HTTP %{http_code}\n' -u whisper:<your-password> http://127.0.0.1/load

Expected output from those four commands:

000 connection refused, as expected anonymous: HTTP 401 wrong password: HTTP 401 load endpoint: HTTP 403

Recommended additional hardening

  • Restrict port 80 by source address in your network security group so only your own networks can reach the service.
  • Add TLS. Basic authentication sends credentials in a reversible encoding, so on any untrusted network you should terminate TLS in front of it. The nginx configuration at /etc/nginx/sites-available/sac-whisper-cpp-server is ready for a certificate.
  • Change the password if you wish, using sudo htpasswd /etc/nginx/whisper-cpp-server.htpasswd whisper, and record the new value safely. nginx picks up the change immediately, with no reload.
  • Keep the engine on loopback. Do not change --host 127.0.0.1 in the service unit.

Verifying the bundled model

The model shipped with this image can be checked against the checksum recorded at build time:

sha256sum /var/lib/whisper-cpp-server/models/ggml-base.en.bin

Expected output:

a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002 /var/lib/whisper-cpp-server/models/ggml-base.en.bin

The licence notices and the written provenance record are here:

ls -l /opt/whisper/
cat /opt/whisper/MODEL-PROVENANCE.txt

Server components

The image includes these components, listed with version and install path:

  • whisper.cpp server: version 1.9.1, in /usr/local/bin/whisper-server
  • Speech recognition model: ggml-base.en, in /var/lib/whisper-cpp-server/models/ggml-base.en.bin
  • nginx: version 1.24, in /usr/sbin/nginx
  • ffmpeg: version 6.1.1, in /usr/bin/ffmpeg
  • eSpeak NG: version 1.51, in /usr/bin/espeak-ng

Filesystem layout

The virtual machine uses these mount points:

  • / is the root filesystem.
  • /boot holds the operating system kernel files.
  • /boot/efi is the UEFI boot partition, Generation 2 Hyper-V.
  • /mnt is the Azure temporary resource disk.

The key directories are:

  • /var/lib/whisper-cpp-server/models/: speech recognition model files
  • /usr/local/share/whisper-cpp-server/samples/: the bundled test clip
  • /opt/whisper/: MIT licence notices and the model provenance record
  • /etc/nginx/whisper-cpp-server.htpasswd: credential store for the reverse proxy
  • /root/whisper-cpp-server-credentials.txt: per-machine credentials, root only

Managing the service

sudo systemctl status whisper-server.service --no-pager

Restart the inference engine:

sudo systemctl restart whisper-server.service

View the engine log:

sudo journalctl -u whisper-server.service --no-pager -n 20

Scripts and log files

  • /usr/local/sbin/whisper-cpp-server-firstboot.sh: generates the per-machine credential on first boot
  • /var/log/sac/sac-firstboot.log: first-boot log
  • /var/lib/sac/data/whisper-cpp-server-firstboot.done: first-boot completion marker

On startup

On the first boot only, whisper-cpp-server-firstboot.service runs before the application and nginx start. It generates the password unique to this machine, writes it into the reverse-proxy credential store and into the root-only credentials file, resolves the machine's public address for the login banner, and then writes its completion marker. Subsequent boots skip it, so your password is never regenerated behind your back.

Read the first-boot log with:

sudo cat /var/log/sac/sac-firstboot.log

Troubleshooting

Every request returns HTTP 401

Confirm that you are using the password from this machine. Passwords are per-machine and are not shared between deployments:

sudo grep -c WHISPER_PASSWORD /root/whisper-cpp-server-credentials.txt

If the credentials file is missing, first boot did not complete. Check the log:

sudo journalctl -u whisper-cpp-server-firstboot.service --no-pager -n 20

A request returns HTTP 200 but the transcript is empty

An empty transcript with a healthy status usually means the model file is missing or corrupt. Verify it against its checksum as shown in Verifying the bundled model. Genuinely silent audio also returns an empty transcript, which is correct behaviour, so confirm your input actually contains speech.

The /load endpoint returns HTTP 403

This is intentional. The endpoint is blocked on this image because it allows an arbitrary server side file to be loaded as a model. To change the model, edit the service unit as described in Using a different model.

Transcription is slower than expected

If you use a B-series size, confirm that the machine has not run out of CPU credits, and consider a compute-optimised size. See Performance.

The browser will not prompt for credentials again

Browsers cache Basic authentication credentials for the session. Open a private window to be prompted again.

Security recommendations

  • Restrict port 80 to trusted source addresses in your network security group.
  • Terminate TLS in front of the service before you use it across any untrusted network.
  • Keep the credentials file readable by root only, and do not copy it into shared storage.
  • Leave the inference engine bound to the loopback interface.
  • Apply operating system updates regularly. Unattended security upgrades are enabled in this image.

Support

For assistance, contact SecureAnyCloud support:

  • Email: sales@secureanycloud.com
  • Response time: 24/7 with a guaranteed 24-hour response SLA
  • Website: https://secureanycloud.com

Deploy on Azure

Launch whisper.cpp Server with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure