Cloudprober 0.14.3

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Cloudprober 0.14.3 on Ubuntu 24.04 as the software version, select your Region, and choose Launch through EC2. Choose m5.large (a smaller instance is fine for a few dozen probes), your key pair and a security group as described in the prerequisites.

To launch with the AWS CLI, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=cloudprober}]'

First boot takes a few seconds after the instance is running.

Version: Cloudprober 0.14.3 on Ubuntu 24.04. SSH user: ubuntu.

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot sets the password and the certificate, and then nginx and Cloudprober start. Check the services:

systemctl is-active cloudprober nginx cloudprober-firstboot
sudo test -f /var/lib/sac/data/cloudprober-firstboot.done && echo FIRSTBOOT_DONE

All three units are active and the first boot marker is present:

active active active FIRSTBOOT_DONE

Only SSH and nginx listen on network ports, and Cloudprober listens on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:9313 users:(("cloudprober")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

sudo cat /root/cloudprober-credentials.txt

The file is readable by root only:

# Cloudprober 0.14.3 on Ubuntu 24.04 (sac) - generated on this instance's first boot. # This password is unique to this instance; none was baked into the image. Keep this file secret. CLOUDPROBER_URL=https://<public-ip>/ CLOUDPROBER_USER=sac CLOUDPROBER_PASSWORD=<CLOUDPROBER_PASSWORD> # Status page: https://<host>/status Metrics: https://<host>/metrics (Prometheus format) # Health check: https://<host>/healthz (no login; answers "ok") # The certificate is self signed (minted on this instance): accept the browser warning once, or install # your own. Serve a DNS name instead:
sudo cloudprober-set-hostname <your-dns-name>
# Cloudprober listens on 127.0.0.1:9313 only; nginx on 443 is the only way in.
# Probes live in /etc/cloudprober.cfg; apply changes with:
sudo systemctl restart cloudprober
# Self-test:
sudo /usr/local/sbin/cloudprober-selftest.sh
# User guide: https://secureanycloud.com/stacks/cloudprober-0-14-3/

The URL uses the instance's public IP address from the EC2 instance metadata. The certificate also names the public DNS name and the private IP address, so https://<public-dns-name>/ and https://<private-ip>/ work too. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it. To serve Cloudprober under your own DNS name, see Step 8.

Browse to the URL from the credentials file. Your browser warns once about the self-signed certificate. Accept the warning, or install your own certificate (Step 8). Sign in as SecureAnyCloud with the password from the credentials file.

The status page shows every probe with its success ratio over the last minute, five minutes, half hour and longer, and a graph per probe. The links at the top lead to the running configuration, Cloudprober's recent log lines and the raw metrics.

The image ships one probe, sac_selfcheck. It checks Cloudprober's own status endpoint every five seconds, so metrics flow from the first minute. The page below shows it alongside the example probes from Step 7.

A request without the password gets nginx's 401, and the health check answers without one:

curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/metrics
curl -sk https://127.0.0.1/healthz

401 ok

Every probe exports total, success, failure, timeouts, latency (cumulative microseconds) and, for HTTP probes, resp_code counters. Read the metrics on the instance:

curl -sk -u "sac:$(sudo sed -n 's/^CLOUDPROBER_PASSWORD=//p' /root/cloudprober-credentials.txt)" https://127.0.0.1/metrics | grep 'probe="sac_selfcheck"'

total{ptype="http",probe="sac_selfcheck",dst="127.0.0.1"} 30 success{ptype="http",probe="sac_selfcheck",dst="127.0.0.1"} 30 latency{ptype="http",probe="sac_selfcheck",dst="127.0.0.1"} 29967.474 timeouts{ptype="http",probe="sac_selfcheck",dst="127.0.0.1"} 0 resp_code{ptype="http",probe="sac_selfcheck",dst="127.0.0.1",code="200"} 30 failure{ptype="http",probe="sac_selfcheck",dst="127.0.0.1"} 0

To scrape the metrics from a Prometheus server, add a job like the following to prometheus.yml. Keep the password from the credentials file in a file that only Prometheus can read. Use the address in the certificate (the private IP or your DNS name), and either trust the instance's certificate with ca_file or install your own certificate (Step 8):

scrape_configs: - job_name: cloudprober scheme: https basic_auth: username: sac password_file: /etc/prometheus/cloudprober.pass tls_config: ca_file: /etc/prometheus/cloudprober.crt
# a copy of /etc/nginx/tls/cloudprober.crt static_configs: - targets: ['<private-ip>:443']

The success ratio of a probe over five minutes is rate(success[5m]) / rate(total[5m]), and its mean latency is rate(latency[5m]) / rate(success[5m]).

Probes live in /etc/cloudprober.cfg. Open the file with sudo nano /etc/cloudprober.cfg and add a probe block per check. The following example adds an HTTPS check of a website, a DNS check against the VPC resolver and a TCP check of a port:

probe { name: "website_https" type: HTTP targets { host_names: "secureanycloud.com" } http_probe { protocol: HTTPS relative_url: "/" } interval_msec: 15000 timeout_msec: 5000 } probe { name: "vpc_dns" type: DNS targets { host_names: "169.254.169.253" } dns_probe { resolved_domain: "aws.amazon.com." } interval_msec: 15000 timeout_msec: 3000 } probe { name: "ssh_port" type: TCP targets { host_names: "127.0.0.1" } tcp_probe { port: 22 } interval_msec: 15000 timeout_msec: 3000 }

Check the file and apply it. Cloudprober reads its configuration at start, so restart the service:

sudo /opt/cloudprober/cloudprober --config_file=/etc/cloudprober.cfg --configtest && echo CONFIG_OK
sudo systemctl restart cloudprober

CONFIG_OK

The config link on the status page shows the configuration Cloudprober is running:

Cloudprober also has PING, gRPC, UDP and external (run a command) probe types, target discovery and many more options. See the Cloudprober documentation for details.

The instance metadata service (169.254.169.254) is blocked for Cloudprober by design, so a probe aimed at it always fails. The VPC DNS resolver (169.254.169.253) and Amazon Time Sync (169.254.169.123) are not blocked.

Point a DNS record at the instance (an Elastic IP keeps the address stable), and then set the name that Cloudprober is served as. This command mints a new self-signed certificate for the name, updates nginx and the URL in /root/cloudprober-credentials.txt, and keeps the name across reboots:

sudo cloudprober-set-hostname <your-domain>

To use a certificate from a public or internal certificate authority instead (for example, one issued by Let's Encrypt for <your-domain>), copy it over the generated pair and reload nginx. Running cloudprober-set-hostname again replaces it with a self-signed certificate:

sudo install -m 0644 <your-domain>.crt /etc/nginx/tls/cloudprober.crt
sudo install -m 0600 <your-domain>.key /etc/nginx/tls/cloudprober.key
sudo nginx -t && sudo systemctl reload nginx

The shipped self-test verifies the following:

• The exact set of listening ports
• An unknown host name is dropped
• nginx refuses a request without a password and a request with a wrong password, and it accepts this instance's password
• The self-check probe is succeeding
• The instance metadata service cannot be reached from inside the cloudprober and nginx services, while it can be reached from your shell

The password is handed to curl in a root-only temporary file, never on a command line:

sudo /usr/local/sbin/cloudprober-selftest.sh

selftest OK: listening tcp lo:53 lo:9313 pub:22 pub:443 pub:80 | healthz ok, unknown Host dropped (444); anonymous 401 and wrong password 401 (nginx auth), per-instance password 200; selfcheck probe success=30; instance metadata blocked inside cloudprober and nginx (000), reachable from the host shell (200)

The metadata block is a systemd setting on each service. Display the settings with:

systemctl show -p IPAddressDeny cloudprober nginx cloudprober-firstboot

IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32 IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32 IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32

A request for a host name that the instance does not serve gets no answer at all. nginx closes the connection, so curl exits with code 52:

curl -sk -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000 curl exit 52

The image generated the certificate on this instance for its own addresses:

sudo openssl x509 -in /etc/nginx/tls/cloudprober.crt -noout -subject -ext subjectAltName

subject=CN = <public-ip>, O = sac X509v3 Subject Alternative Name: IP Address:<public-ip>, DNS:<public-dns-name>, IP Address:<private-ip>, DNS:<private-dns-name>, IP Address:127.0.0.1, DNS:localhost

/opt/cloudprober/cloudprober --version nginx -v 2>&1

v0.14.3 nginx version: nginx/1.24.0 (Ubuntu)

The image uses these files:

• /etc/cloudprober.cfg: the Cloudprober configuration (probes, surfacers, the loopback web server)
• /opt/cloudprober/cloudprober: the Cloudprober binary, run by cloudprober.service as the cloudprober user
• /etc/nginx/sites-available/sac-cloudprober: the nginx site (HTTPS, basic authentication, proxy to 127.0.0.1:9313)
• /etc/nginx/cloudprober.htpasswd: the basic authentication user and password hash (SHA-512 crypt)
• /etc/nginx/tls/: this instance's certificate and private key
• /root/cloudprober-credentials.txt: the URL, user and password (root only)

Run these lines in your SSH session. The new password is read at a prompt and reaches openssl on standard input, so it never appears on a command line or in your shell history:

read -rsp 'New password: ' NEWPASS; echo printf 'sac:%s\n' "$(printf '%s\n' "$NEWPASS" | openssl passwd -6 -stdin)" | sudo tee /etc/nginx/cloudprober.htpasswd >/dev/null unset NEWPASS

nginx reads the file on every request, so the new password works at once. Then update or delete CLOUDPROBER_PASSWORD in /root/cloudprober-credentials.txt (sudo nano /root/cloudprober-credentials.txt). To add a second user, append another user:hash line in the same format.

• Keep port 443 open only to the networks that need the status page and metrics, and keep port 22 open only to your admin network. Never open 9313.
• Use your own DNS name and a certificate from a certificate authority that your Prometheus servers trust (Step 8).
• Keep the password in a secrets store, and change it when people leave (Step 11).
• Probe only targets that you are allowed to probe, and keep probe intervals reasonable for third-party services.
• The instance applies Ubuntu security updates automatically. To update Cloudprober itself, replace /opt/cloudprober/cloudprober with a newer upstream release after you check its checksum, and then restart the service.