rumqttd 0.20.0

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick rumqttd 0.20.0 on Ubuntu 24.04 as the software version, choose your Region, then choose Launch through EC2.

Select m5.large (or larger for many thousands of connections or high message rates), your key pair and a security group as described above.

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=rumqttd-broker}]'

The image requires IMDSv2 (session tokens).

First boot takes under a minute after the instance is running. The boot service reads the instance's EC2 names and creates the login and the certificate. Then rumqttd and HAProxy start, and the guard checks them.

Connect over SSH with these values:

Version: rumqttd 0.20.0 on Ubuntu 24.04; SSH user: ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

The login banner shows where to find the login and the certificate:

rumqttd 0.20.0 on Ubuntu 24.04 (sac) Login, endpoint, certificate fingerprint:
sudo cat /root/rumqttd-credentials.txt Clients: MQTT over TLS only, mqtts://<this host>:8883 (pin /etc/haproxy/sac/rumqttd.crt; TLS by HAProxy) Broker: systemctl status rumqttd haproxy Verify:
sudo rumqttd-selfcheck Users:
sudo rumqttd-user list User guide: https://secureanycloud.com/stacks/rumqttd-0-20-0/

systemctl is-active rumqttd-metadata-names rumqttd-firstboot rumqttd-egress-guard rumqttd haproxy
sudo ss -tlnp | grep -E ':(22|80|443|1883|3030|8080|8404|8883|9042) '
sudo ss -ulnp | grep -E 'rumqttd|haproxy' || echo 'no UDP socket'

active active active active active LISTEN 0 1024 127.0.0.1:1883 0.0.0.0:* users:(("rumqttd",pid=13609,fd=6)) LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=891,fd=3),("systemd",pid=1,fd=258)) LISTEN 0 4096 0.0.0.0:8883 0.0.0.0:* users:(("haproxy",pid=13921,fd=5)) LISTEN 0 4096 [::]:22 [::]:* users:(("sshd",pid=891,fd=4),("systemd",pid=1,fd=259)) no UDP socket

rumqttd-metadata-names reads this instance's EC2 names for the certificate. rumqttd-firstboot prepares the login and the certificate. rumqttd-egress-guard loads the firewall table. rumqttd is the broker, and haproxy is its TLS front.

HAProxy listens on 8883 (every IPv4 address) for clients. rumqttd listens on 127.0.0.1:1883 only, where nothing outside the instance can reach it.

Everything generated for your instance is in /root/rumqttd-credentials.txt (readable by root only). View it all with sudo cat /root/rumqttd-credentials.txt. These lines are not secret:

sudo grep -E '^RUMQTTD_(HOST|URL|PORT|PRIVATE_IP|USER|TLS_CA_FILE|TLS_CERT_SHA256|TLS_PUBKEY_SHA256)=' /root/rumqttd-credentials.txt

RUMQTTD_HOST=3.89.221.187 RUMQTTD_URL=mqtts://3.89.221.187:8883 RUMQTTD_PORT=8883 RUMQTTD_PRIVATE_IP=172.31.88.68 RUMQTTD_USER=sac-app RUMQTTD_TLS_CA_FILE=/etc/haproxy/sac/rumqttd.crt RUMQTTD_TLS_CERT_SHA256=BB:3E:77:5A:1A:24:AA:8C:7F:F4:C9:20:F6:7B:06:61:B1:C9:56:66:F2:C7:F7:C6:AF:D2:88:50:8A:F0:AC:E9 RUMQTTD_TLS_PUBKEY_SHA256=ae72fdda1faf340dc4ff6e00a919bf42013b0aaa5206c013aa726d738d4b47cb

The file also holds RUMQTTD_PASSWORD, the password of sac-app. Give each device type or application a login of its own (Step 8) rather than sharing this one.

Keep passwords in files with mode 0600, not in shell history or on command lines. Every user on a machine can see a command line through ps.

The broker's certificate is self-signed, made on your instance:

sudo openssl x509 -in /etc/haproxy/sac/rumqttd.crt -noout -subject -enddate -ext subjectAltName,basicConstraints
sudo openssl x509 -in /etc/haproxy/sac/rumqttd.crt -noout -fingerprint -sha256

subject=CN = rumqttd-broker, O = sac notAfter=Feb 10 13:04:43 2029 GMT X509v3 Subject Alternative Name: IP Address:3.89.221.187, DNS:ec2-3-89-221-187.compute-1.amazonaws.com, IP Address:172.31.88.68, DNS:ip-172-31-88-68.ec2.internal, IP Address:127.0.0.1, DNS:localhost X509v3 Basic Constraints: critical CA:FALSE sha256 Fingerprint=BB:3E:77:5A:1A:24:AA:8C:7F:F4:C9:20:F6:7B:06:61:B1:C9:56:66:F2:C7:F7:C6:AF:D2:88:50:8A:F0:AC:E9

CA:FALSE means it is an end-entity certificate that cannot sign anything else. The SHA-256 fingerprint is the RUMQTTD_TLS_CERT_SHA256 value in your credentials file.

Copy the certificate to each client machine. The certificate is public, and the private key never leaves the instance:

client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/haproxy/sac/rumqttd.crt' > rumqttd.crt client-host$ openssl x509 -in rumqttd.crt -noout -fingerprint -sha256

Compare the fingerprint with RUMQTTD_TLS_CERT_SHA256 before you trust the file.

Then give your client only this file as its CA file, with host name checking on. Use --cafile for mosquitto_pub and mosquitto_sub, or tls_set(ca_certs=...) in Paho. Do not add the file to the operating system's trust store, because the client should trust this one broker, not every program on the machine. Your client then refuses any other server, including one in the middle that presents a different certificate.

A client that uses only its system trust store is refused. Paho raises CERTIFICATE_VERIFY_FAILED, and mosquitto_pub prints Error: Protocol error.

The certificate covers the instance's public and private IPv4 addresses and EC2 DNS names, localhost and 127.0.0.1. An EC2 stop and start gives the instance a new public IP. At the next boot, the certificate is re-issued for the new names with the same key. Clients that pin the public key (RUMQTTD_TLS_PUBKEY_SHA256) keep working, while clients that pin the certificate file need the new copy.

rumqttd-probe is a small MQTT client. It reads passwords from a file, or from the credentials file with --identity app, and trusts only the broker's certificate:

sudo rumqttd-probe connect --identity app
sudo rumqttd-probe roundtrip --identity app --topic guide/test
sudo rumqttd-probe connect --anonymous || true
sudo rumqttd-probe plain || true
sudo rumqttd-probe connect --anonymous --no-tls --port 1883 || true

CONNACK-OK proto=4 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 ROUNDTRIP OK topic=guide/test qos=1 bytes=43 proto=4 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 CLOSED-NO-CONNACK NOT-MQTT-PLAIN no-bytes CLOSED-NO-CONNACK

The login connects over TLS 1.3, subscribes to guide/test, publishes a message there and receives it back.

A client without a login, with a wrong password or with an unknown user name is closed without a CONNACK. rumqttd does not say which part was wrong. Clear-text MQTT sent to 8883 gets a TLS alert, never an MQTT answer. On the loopback address, rumqttd itself refuses the same anonymous client. rumqttd-probe exits non-zero on a refusal, hence || true.

The broker speaks MQTT 3.1.1. rumqttd 0.20.0's MQTT 5 support is experimental and is not enabled on this image. Configure your client for protocol version 3.1.1: use -V mqttv311 for the Mosquitto tools and protocol=mqtt.MQTTv311 in Paho.

On the client machine, install the Mosquitto command-line clients. Use sudo apt-get install mosquitto-clients on Ubuntu or Debian, or brew install mosquitto on macOS. Then copy the certificate as in Step 5.

Next, put the connection settings in an options file that only you can read. mosquitto_pub and mosquitto_sub read their default options, one per line, from mosquitto_pub and mosquitto_sub files in the directory named by XDG_CONFIG_HOME (normally ~/.config), so the password is never on a command line. Keeping them in a directory of their own lets you have one per broker and login:

client-host$ umask 077; mkdir -p ~/mqtt-app client-host$ printf -- '-h <public-ip>\n-p 8883\n-V mqttv311\n--cafile %s/rumqttd.crt\n-u sac-app\n' "$PWD" > ~/mqtt-app/mosquitto_pub client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> "sudo sed -n 's/^RUMQTTD_PASSWORD=/-P /p' /root/rumqttd-credentials.txt" >> ~/mqtt-app/mosquitto_pub client-host$ cp ~/mqtt-app/mosquitto_pub ~/mqtt-app/mosquitto_sub

Subscribe in one terminal and publish from another:

client-host$ XDG_CONFIG_HOME=~/mqtt-app mosquitto_sub -t 'demo/#' -q 1 -v client-host$ XDG_CONFIG_HOME=~/mqtt-app mosquitto_pub -t demo/hello -q 1 -m 'hello from mosquitto_pub'

demo/hello hello from mosquitto_pub

Use a name the certificate covers as -h: the public or private IP, an EC2 DNS name, or your own DNS name once you have added it (Step 13).

A wrong or missing password, or an unknown user, is refused. rumqttd closes the connection without a CONNACK, and mosquitto_pub and mosquitto_sub print Error: The connection was lost. A client that does not use TLS, or trusts only the system certificate store, gets Error: Protocol error.

To do the same from Python with Eclipse Paho, install it with sudo apt-get install python3-paho-mqtt or pip install paho-mqtt:

# paho_demo.py - publish and receive through your sac rumqttd broker over MQTT/TLS (paho-mqtt 1.6 or 2.x).
# Usage:
python3 paho_demo.py <broker-host> <rumqttd.crt> <user> <password-file> <topic>
# The password is read from a file, never the command line; TLS trusts ONLY the broker's own certificate. import ssl import sys import threading import paho.mqtt.client as mqtt host, cafile, user, pwfile, topic = sys.argv[1:6] with open(pwfile) as f: password = f.read().strip() got = threading.Event() try: c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv311) except AttributeError: # paho-mqtt 1.x c = mqtt.Client(protocol=mqtt.MQTTv311) c.username_pw_set(user, password) c.tls_set(ca_certs=cafile, tls_version=ssl.PROTOCOL_TLS_CLIENT) # verify against this one certificate, host name checked c.on_connect = lambda cl, *a: cl.subscribe(topic, qos=1) c.on_subscribe = lambda cl, *a: cl.publish(topic, b"hello from paho", qos=1) c.on_message = lambda cl, u, m: (print("received:", m.topic, m.payload.decode()), got.set()) c.connect(host, 8883) c.loop_start() ok = got.wait(15) c.loop_stop() c.disconnect() sys.exit(0 if ok else 1)

client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> "sudo sed -n 's/^RUMQTTD_PASSWORD=//p' /root/rumqttd-credentials.txt" > app.password client-host$ python3 paho_demo.py <public-ip> rumqttd.crt sac-app app.password demo/paho received: demo/paho hello from paho

The URL form is mqtts://<public-ip>:8883. Some clients accept the login inside the URL (mqtts://sac-app:<password>@...). Build that URL inside your program from the password file. Do not write it in a configuration file or on a command line.

Give each device type or application its own login rather than sharing one. rumqttd-user reads the new password from standard input or a hidden prompt. It adds the login to rumqttd's configuration through the configuration gate and restarts rumqttd and HAProxy, so connected clients reconnect:

umask 077 openssl rand -hex 16 > ~/sensors.password
sudo rumqttd-user add sensors < ~/sensors.password
sudo rumqttd-user list
sudo rumqttd-probe roundtrip --user sensors --password-file ~/sensors.password --topic sensors/17/temperature

added sensors rumqttd and its TLS front restarted (connected clients reconnect) sac-app (publish and subscribe to any topic: rumqttd has no topic ACLs) sensors (publish and subscribe to any topic: rumqttd has no topic ACLs) ROUNDTRIP OK topic=sensors/17/temperature qos=1 bytes=43 proto=4 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256

User names are 1 to 32 characters (a-z, 0-9, _ and -). The names root, admin, sys, all, anonymous, rumqttd and haproxy are reserved, and an instance holds at most 64 logins. Passwords are 16 to 128 printable ASCII characters without a quote or a backslash.

rumqttd keeps its logins in clear text in /etc/rumqttd/rumqttd.toml. Only root and the broker can read that file (root:rumqttd, mode 0640). rumqttd-user rewrites the file whole, so never edit the logins by hand.

rumqttd has no topic access rules. sensors can publish and subscribe to every topic, including sac-app's and #, exactly as sac-app can. Treat every login as trusted with all of the broker's messages. Run a separate broker for each party that must not see the others' traffic.

Hand ~/sensors.password to your application's secret store, then delete the local copy.

sudo rumqttd-user passwd <name> changes a password, read from standard input as for add. The old password is refused from the next login. sudo rumqttd-user remove <name> deletes a login, but the last login cannot be removed. To remove the example user again:

sudo rumqttd-user remove sensors shred -u ~/sensors.password

HAProxy enforces the limits that keep slow, idle or misbehaving clients from shutting real clients out. Show the limits with rumqttd-limits:

sudo rumqttd-limits show

handshake_timeout_s 10 (2..10, default 10) seconds a client may take from TCP connect to a finished TLS handshake idle_timeout_m 15 (1..1440, default 15) minutes an established MQTT session may be silent in both directions per_source_max_conn 64 (1..10000, default 64) concurrent connections from one source IPv4 address per_source_conn_rate_10s 300 (1..100000, default 300) new connections from one source address per 10 seconds max_conn 4096 (64..10000, default 4096) concurrent connections in all (rumqttd itself takes at most 10010)

• handshake_timeout_s (10 seconds): a client must finish its TLS handshake within this time of connecting, or HAProxy closes it. HAProxy 2.8 has no separate handshake timer, so this value is its timeout client, which covers everything before the connection to rumqttd is made. It is capped at 10 seconds.
• idle_timeout_m (15 minutes): once connected, a session that is silent in both directions for this long is closed (HAProxy's timeout tunnel). MQTT clients send a keep-alive ping well within this time, and 60 seconds is the usual default. Set the keep-alive of a client below this value.
• per_source_max_conn (64) and per_source_conn_rate_10s (300): the connections one source IPv4 address may hold at once, and may open in any 10 seconds. Clients behind one NAT gateway share one source address, so raise these values if a fleet of devices connects through NAT.
• max_conn (4,096): the connections HAProxy accepts in all. rumqttd itself takes at most 10,010.

rumqttd sees every client as 127.0.0.1, so HAProxy applies these limits before a client reaches rumqttd. You can watch them work. A client that connects and never starts TLS is closed after the handshake timeout, and while it waits, a real login is still answered at once:

sudo rumqttd-probe idle --hold 20
sudo rumqttd-probe stallcheck --identity app --count 20

IDLE-DROPPED seconds=10.0 STALL-CONNACK n=20 seconds=0.0 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256

Change a limit with rumqttd-limits set. It checks the value, renders /etc/haproxy/haproxy.cfg again from /etc/haproxy/rumqttd-limits.conf and restarts HAProxy, so connected clients reconnect:

sudo rumqttd-limits set per_source_max_conn 256

Do not edit /etc/haproxy/haproxy.cfg by hand. The configuration gate refuses any text other than the shipped configuration rendered with your limits, and HAProxy then does not start.

rumqttd.service and haproxy.service each run rumqttd-config-gate before every start and rumqttd-guard after it. At any time, check the gates and the whole broker as a client sees it:

sudo rumqttd-config-gate --unit rumqttd && sudo rumqttd-config-gate --unit haproxy && echo gate-ok
sudo rumqttd-selfcheck

rumqttd-config-gate: OK (rumqttd) rumqttd-config-gate: OK (haproxy) gate-ok SELFCHECK OK login=sac-app tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256; anonymous, wrong password, unknown user and clear text refused; 1883 refuses anonymous; STALL-CONNACK n=5 seconds=0.0 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256

Suppose you edit /etc/rumqttd/rumqttd.toml and do any of the following: move rumqttd's listener off 127.0.0.1, add a second listener, remove the logins, enable the console or the metrics exporter, lengthen the CONNECT timeout, or loosen the file's permissions. The gate then refuses the change and rumqttd does not start.

HAProxy does not start if its configuration differs from the shipped one rendered with your limits (a stats page or socket, TLS 1.0 or 1.1, a longer handshake timeout, another listener). It also does not start if its key is missing, does not match the certificate or is readable by others. journalctl -u rumqttd -u haproxy shows the reason.

If the running services ever listen anywhere they should not, accept an anonymous login or run without the firewall table, the guard stops them. HAProxy cannot run without rumqttd. If first boot cannot create the login or the certificate, neither service starts and nothing listens on 8883 or 1883.

rumqttd's example configuration opens clear-text MQTT on 1883 for every address, MQTT 5 on 1884, a console on 3030 and a Prometheus exporter on 9042. On a cloud instance each is a way around TLS or the logins. User names, passwords and every message cross the network in clear text on 1883, and the console reports the broker's state to anyone who asks.

This image configures exactly one rumqttd listener, MQTT 3.1.1 on 127.0.0.1:1883. It sets this in rumqttd's own configuration rather than relying on a firewall, and HAProxy is the only service on 8883:

• 1883 (MQTT): bound to 127.0.0.1 only, behind HAProxy. Every maintained MQTT client supports TLS on 8883.
• 3030 (console) and 9042 (metrics): not configured, and the gate refuses them.
• 8404 (HAProxy statistics): HAProxy has no stats page, stats socket or master socket.
• Anonymous access: refused. Every connection must carry a login from the configuration.

From another machine, even inside a security group that allows all traffic, a connection to 1883, 3030, 9042, 8404, 80 or 443 is refused. Clear-text MQTT sent to 8883 gets no MQTT answer. On the instance itself:

IP=$(sudo sed -n 's/^RUMQTTD_PRIVATE_IP=//p' /root/rumqttd-credentials.txt) for p in 1883 3030 9042 8404 8883; do nc -z -w 3 "$IP" "$p" 2>/dev/null && echo "$p open" || echo "$p closed"; done

1883 closed 3030 closed 9042 closed 8404 closed 8883 open

The EC2 instance metadata service (169.254.169.254) hands out the instance's IAM role credentials to any process that asks, so a network service must not be able to reach it. rumqttd, haproxy and rumqttd-firstboot run with the metadata addresses blocked by systemd (IPAddressDeny), and rumqttd may exchange packets with the loopback address only.

Only rumqttd-metadata-names, which reads the instance's names for the certificate, may reach the metadata service, and it may reach nothing else:

for u in rumqttd haproxy rumqttd-firstboot rumqttd-metadata-names; do printf '%-24s deny=%s allow=%s\n' "$u" "$(systemctl show -p IPAddressDeny --value "$u")" "$(systemctl show -p IPAddressAllow --value "$u")"; done

rumqttd deny=0.0.0.0/0 ::/0 allow=127.0.0.0/8 ::1/128 haproxy deny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 allow= rumqttd-firstboot deny=fd00:ec2::254/128 169.254.169.254/32 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 168.63.129.16/32 allow= rumqttd-metadata-names deny=0.0.0.0/0 ::/0 allow=fd00:ec2::254/128 169.254.169.254/32

On top of that, the nftables table inet sac_rumqttd_egress denies all outbound traffic, IPv4 and IPv6, for the users rumqttd and haproxy. Only their answers to their clients and HAProxy's connections to rumqttd on 127.0.0.1:1883 are allowed. Neither user can reach the metadata service, the internet, your VPC or DNS, whatever its configuration says. Other users are unaffected:

IP=$(sudo sed -n 's/^RUMQTTD_PRIVATE_IP=//p' /root/rumqttd-credentials.txt) for u in rumqttd haproxy; do for t in http://169.254.169.254/latest/meta-data/ https://1.1.1.1/ "http://$IP:22/"; do
sudo -u "$u" curl -s -m 5 -o /dev/null -w "$u -> $t: %{http_code} %{errormsg}\n" "$t" || true; done; done curl -s -m 5 -o /dev/null -w "ubuntu -> https://1.1.1.1/: %{http_code}\n" https://1.1.1.1/

rumqttd -> http://169.254.169.254/latest/meta-data/: 000 Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to server rumqttd -> https://1.1.1.1/: 000 Failed to connect to 1.1.1.1 port 443 after 0 ms: Couldn't connect to server rumqttd -> http://172.31.88.68:22/: 000 Failed to connect to 172.31.88.68 port 22 after 0 ms: Couldn't connect to server haproxy -> http://169.254.169.254/latest/meta-data/: 000 Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to server haproxy -> https://1.1.1.1/: 000 Failed to connect to 1.1.1.1 port 443 after 0 ms: Couldn't connect to server haproxy -> http://172.31.88.68:22/: 000 Failed to connect to 172.31.88.68 port 22 after 0 ms: Couldn't connect to server ubuntu -> https://1.1.1.1/: 301

rumqttd-egress-guard.service loads the table before rumqttd and HAProxy start, and both require it. If the table is removed, the guard stops them.

Point a DNS name at the instance. An Elastic IP keeps the address across stop and start. Then add the name to the certificate:

sudo rumqttd-set-hostname <dns-name>

This re-issues the self-signed certificate with the same key to cover the name, updates the endpoint in /root/rumqttd-credentials.txt and restarts HAProxy. Copy the new certificate to clients that pin the certificate file.

To use a certificate from your CA, copy the chain (server certificate first) and its unencrypted PEM key to the instance. The CA can be your own or a public one such as Let's Encrypt. Then run:

sudo rumqttd-install-cert <certificate-chain.pem> <private-key.pem> <dns-name>

Port 8883 then serves your certificate, and clients trust your CA instead of pinning this instance. rumqttd-selfcheck checks it under the name you gave. The boot-time renewal never replaces an installed certificate.

To go back to a new self-signed certificate, run sudo rumqttd-set-hostname <name>.