SecureAnyCloud
Deployment guide

Prometheus 3

Overview

Prometheus is the de facto open source time-series metrics server in production observability stacks. The 3.x line improves on the 2.x series in PromQL evaluation, native histograms, OTLP metrics ingestion and remote-write reliability.

This guide covers the deployment and configuration of Prometheus 3 on Ubuntu 24.04 on Azure, using SecureAnyCloud Azure Marketplace images.

The image ships the latest non-prerelease 3.x release of Prometheus, resolved at build time from the official github.com/prometheus/prometheus/releases API. The binary lives at /usr/local/bin/prometheus and the config at /etc/prometheus/prometheus.yml. The TSDB lives at /var/lib/prometheus/data, not /mnt, because Azure mounts the ephemeral resource disk at /mnt and that location does not survive SIG capture.

The default config monitors Prometheus itself. Example node-exporter and Grafana scrape jobs are included as comments for you to uncomment and edit.

Prometheus has no built-in authentication. This is by upstream design: auth and TLS are concerns for an external reverse proxy. For any non-loopback exposure, terminate TLS and basic auth (or AAD) at Nginx, Caddy, Apache or an Azure Application Gateway. The SecureAnyCloud nginx-ssl-certbot-ubuntu-24-04 image is purpose-built as a companion reverse proxy.

What you get:

  • Prometheus 3 server installed from the official github.com/prometheus/prometheus/releases tarball (latest 3.x at build time)
  • promtool at /usr/local/bin/promtool for config validation, rule testing and tsdb administration
  • Console templates and console libraries at /etc/prometheus/consoles/ and /etc/prometheus/console_libraries/
  • prometheus.service systemd unit that starts on boot and runs as the prometheus system user
  • prometheus-firstboot.service systemd oneshot that writes endpoint info to /stage/scripts/prometheus-credentials.log
  • Default scrape config at /etc/prometheus/prometheus.yml with the prometheus job pre-seeded and example node and grafana jobs commented in
  • TSDB at /var/lib/prometheus/data
  • Listener on TCP 9090 (HTTP, no built-in auth)
  • Lifecycle API enabled (--web.enable-lifecycle), so curl -X POST http://localhost:9090/-/reload reloads the config without a restart
  • Ubuntu 24.04 LTS base with the latest security patches applied at build time
  • Azure Linux Agent for cloud integration and SSH key injection
  • 24/7 SecureAnyCloud support with guaranteed 24 hour response SLA

Prerequisites

  • An active Azure subscription
  • A subscription to the Prometheus 3 on Ubuntu 24.04 listing on Azure Marketplace
  • An SSH public key for VM authentication
  • A virtual network and subnet in the target region
  • Recommended virtual machine size: Standard_B2s (2 vCPU, 4 GB RAM) for development and small fleets. For production deployments that scrape more than about 50 targets at 15s intervals, use Standard_D2s_v3 or larger, with attached Premium SSD data disks for the TSDB.
1

Deploy from the Azure Portal

In the Azure Portal, open the Marketplace and search for Prometheus 3. Select the SecureAnyCloud publisher entry and choose Create.

On the Networking tab, attach a network security group. Allow inbound TCP 22 from your management IP range. Allow TCP 9090 from your scrape clients (such as the Grafana VM private IP) and admin networks.

Do not expose port 9090 to the public internet without a TLS-terminating reverse proxy in front. Prometheus has no auth, and the /api endpoints expose all metrics and the lifecycle API.

Choose Review + create, wait for validation, then choose Create. Deployment takes around two minutes.

2

Deploy from the Azure CLI

RG="prometheus-prod" LOCATION="eastus" VM_NAME="prometheus-01" ADMIN_USER="azureuser" GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-sac/providers/Microsoft.Compute/galleries/sacGallery/images/prometheus-3-ubuntu-24-04/versions/<version>" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create \
  --resource-group "$RG" \
  --name prom-vnet --address-prefix 10.95.0.0/16 \
  --subnet-name prom-subnet --subnet-prefix 10.95.1.0/24 az network nsg create --resource-group "$RG" --name prom-nsg az network nsg rule create \
  --resource-group "$RG" --nsg-name prom-nsg \
  --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" \
  --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create \
  --resource-group "$RG" --nsg-name prom-nsg \
  --name allow-prometheus-vnet --priority 110 \
  --source-address-prefixes 10.95.0.0/16 \
  --destination-port-ranges 9090 --access Allow --protocol Tcp az vm create \
  --resource-group "$RG" --name "$VM_NAME" \
  --image "$GALLERY_IMAGE_ID" \
  --size Standard_B2s --storage-sku StandardSSD_LRS \
  --admin-username "$ADMIN_USER" --ssh-key-values "$SSH_KEY" \
  --vnet-name prom-vnet --subnet prom-subnet --nsg prom-nsg \
  --public-ip-sku Standard
3

Connect via SSH

ssh azureuser@<vm-ip>

prometheus.service is already running, and prometheus-firstboot.service has already written the endpoint info file.

4

Verify the Prometheus Service

sudo systemctl status prometheus.service --no-pager

The expected state is active (running). Confirm the firstboot sentinel:

sudo test -f /var/lib/sac/data/prometheus-firstboot.done && echo FIRSTBOOT_DONE

Confirm the listener:

sudo ss -tln | grep 9090

Confirm health and readiness:

curl -s http://localhost:9090/-/healthy echo
curl -s http://localhost:9090/-/ready

Confirm the version via the API:

curl -s http://localhost:9090/api/v1/status/buildinfo | jq -r '.data.version'
5

Open the Expression UI

Open http://<vm-ip>:9090 in your browser. Prometheus 3 opens on the expression UI, ready for PromQL queries:

Type up in the expression box, or choose Execute on any pre-filled query. This is the most basic PromQL probe. Every active scrape target reports 1 if it answered the last scrape and 0 if it did not:

6

Inspect Active Targets

Open Status > Target health in the top navigation, or go to /targets directly. On a fresh VM, the prometheus self-monitoring job is the only configured target:

To add more targets, edit /etc/prometheus/prometheus.yml (Step 7), then reload:

sudo curl -s -X POST http://localhost:9090/-/reload

The --web.enable-lifecycle flag is set in the systemd unit, so a reload does not require a service restart.

7

Add Scrape Targets

The default config includes two example jobs as comments. Edit /etc/prometheus/prometheus.yml:

sudo cat /etc/prometheus/prometheus.yml

You can view the active config in the UI under Status > Configuration:

Common pairings:

  • node-exporter on every VM you want to monitor. Install it on each target VM (apt-get install -y prometheus-node-exporter) and point a Prometheus job at port 9100.
  • Grafana for dashboards. Pair Prometheus with the SecureAnyCloud grafana-12-ubuntu-24-04 image, and add Prometheus as a data source in Grafana with the URL http://<this-vm-private-ip>:9090.
  • Application metrics from Spring Boot, Go, Python (prometheus_client) and Node.js (prom-client) frameworks. Each can expose a /metrics endpoint in the Prometheus text format.

After you edit the config, validate it with promtool before you reload:

sudo -u prometheus promtool check config /etc/prometheus/prometheus.yml
sudo curl -s -X POST http://localhost:9090/-/reload
8

Server Status

The Status page shows runtime and build information, command-line flags and feature flags:

9

Server Components

Component Path

Prometheus binary /usr/local/bin/prometheus

promtool /usr/local/bin/promtool

Config file /etc/prometheus/prometheus.yml

Console templates /etc/prometheus/consoles/

Console libraries /etc/prometheus/console_libraries/

TSDB data /var/lib/prometheus/data

Systemd unit /etc/systemd/system/prometheus.service

Firstboot script /usr/local/sbin/prometheus-firstboot.sh

Firstboot service /etc/systemd/system/prometheus-firstboot.service

Endpoint info file /stage/scripts/prometheus-credentials.log

Firstboot sentinel /var/lib/sac/data/prometheus-firstboot.done

Inspect installed version:

/usr/local/bin/prometheus --version 2>&1 | head -1
10

Managing the Prometheus Service

Status:

sudo systemctl status prometheus.service --no-pager

Stop, start or restart:

sudo systemctl stop prometheus.service
sudo systemctl start prometheus.service
sudo systemctl restart prometheus.service

Reload config without restart:

sudo curl -s -X POST http://localhost:9090/-/reload

View the journal:

sudo journalctl -u prometheus.service --no-pager -n 50

Validate the config:

sudo -u prometheus /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
11

Add Authentication via Reverse Proxy

For any non-loopback exposure, put Nginx with basic auth in front of Prometheus:

sudo apt-get update -y sudo apt-get install -y nginx apache2-utils
sudo htpasswd -c /etc/nginx/.prometheus.htpasswd <admin-user>

Create a config at /etc/nginx/sites-available/prometheus.conf:

server { listen 80; server_name <your-domain>; auth_basic "Prometheus"; auth_basic_user_file /etc/nginx/.prometheus.htpasswd; location / { proxy_pass http://127.0.0.1:9090; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } }

Then enable the site and restart Nginx:

sudo ln -s /etc/nginx/sites-available/prometheus.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Restrict the NSG so that ports 80 and 443 are public and port 9090 is reachable from localhost only. The SecureAnyCloud nginx-ssl-certbot-ubuntu-24-04 image deploys all of this, plus Let's Encrypt TLS, in one step.

12

Troubleshooting

Cannot reach Prometheus on port 9090

  • Confirm the service is running: sudo systemctl status prometheus.service
  • Confirm the listener is bound: sudo ss -tln | grep 9090
  • Check the journal: sudo journalctl -u prometheus.service --no-pager -n 50
  • Confirm the NSG allows TCP 9090 from your client source IP.

Self-monitoring target shows DOWN on /targets

  • Open /targets and check the Last Error column.
  • Confirm Prometheus can reach localhost:9090. It always should, so this is a quick sanity check.
  • If the target is DOWN, run curl -s http://localhost:9090/-/ready and check the journal.

PromQL queries return empty

  • Wait one scrape interval (15s default) after a fresh start before you query.
  • Confirm targets are UP at /targets.
  • Confirm the metric name exists: curl -s 'http://localhost:9090/api/v1/label/__name__/values' | jq '.data | length'

Config reload fails

  • Validate first: sudo -u prometheus /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
  • Check the journal for the parse error: sudo journalctl -u prometheus.service --no-pager -n 20

TSDB disk full

  • Check usage: sudo du -sh /var/lib/prometheus/data
  • Reduce retention: add --storage.tsdb.retention.time=15d to the systemd unit ExecStart and restart.
  • Consider remote-write to Cortex, Mimir or Thanos for long-term storage instead of growing the local TSDB.
13

Security Recommendations

  • Front Prometheus with TLS and auth before any non-loopback exposure (Step 11).
  • Restrict the NSG so port 9090 is reachable only from your scrape clients (Grafana VM, alertmanager VM) and admin networks.
  • Disable the lifecycle API in production if you do not need hot reloads. Remove --web.enable-lifecycle from the systemd unit. After that, config changes need a systemctl restart prometheus.
  • Use a separate user for Alertmanager and federation reads by terminating those at the reverse proxy with their own credentials.
  • Enable Prometheus TLS for federation (the --web.config.file= flag) if you have a multi-Prometheus setup.
  • Back up the TSDB with promtool tsdb create-blocks-from snapshots for important long-term metrics.
  • Keep Prometheus updated by periodically re-installing the latest tarball from github.com/prometheus/prometheus/releases.
14

Support and Licensing

Prometheus is licensed under the Apache License 2.0. There is no per-target, per-metric or per-server fee. Prometheus is a graduated CNCF project.

SecureAnyCloud provides commercial support for this image separately from the upstream project.

Email: sales@secureanycloud.com

Website: secureanycloud.com

Support hours: 24/7 with guaranteed 24 hour response SLA

Deploy on Azure

Launch Prometheus 3 on Ubuntu 24.04 with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Launch Prometheus 3 on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure