Deployment guide

rsyslog

Overview

This guide covers the deployment and use of rsyslog 8.2312.0 on Ubuntu 24.04 on Azure, using SecureAnyCloud Azure Marketplace images.

rsyslog is the syslog daemon that ships as the system logger on most Linux distributions. This image configures it as a central log server. Instead of writing only this machine's own logs, it listens for syslog from your other machines and files everything it receives under the host and the program that sent it.

Centralising logs pays off the first time a machine fails. Logs that live only on the machine that produced them are unreadable when that machine is the problem, and they are gone when the instance is replaced. A collector gives you one place to search, one retention policy, and a copy of the evidence that survives the machine.

The shape of the deployment

The collector exposes these ports:

514 UDP: classic syslog, spoken by switches, firewalls, routers and older Unix hosts. It is lossy and spoofable, but universal. Access is restricted by sender address.
514 TCP: syslog over a stream. A full buffer cannot cause message loss, but there is no application-level acknowledgement, so a collector restart can still lose messages in flight. Access is restricted by sender address.
2514 RELP over TLS: the one to use wherever you can. The collector acknowledges every message at the application layer, so nothing is lost across a restart or a broken connection, and the session is encrypted and authenticated in both directions.
22 TCP: SSH.

Everything the collector receives is written to:

/var/log/remote/<sending-host>/<program>.log

Security by design

This image is not an open relay. That is a deliberate design decision. A syslog receiver reachable from the public internet is found and abused within hours. UDP syslog is trivially spoofable, the protocol has no authentication, and a few megabits of junk aimed at port 514 will fill a disk. The image therefore applies these controls:

Restricted plaintext receivers. The plaintext receivers accept traffic only from loopback and the three RFC1918 private ranges. One clearly commented file controls this, and this guide shows you how to narrow it further to your own subnet, which you should do.
Client certificates for the encrypted receiver. On first boot, your VM mints its own certificate authority, a server certificate and one client certificate. That authority exists on no other machine, so the only hosts that can deliver over RELP are the ones you issue certificates to.
Rate-limited plaintext receivers. A single host stuck in a logging loop is the ordinary way a collector falls over.
A bounded queue. The queue is bounded in memory and on disk, and it sheds informational and debug messages before it can fill the volume, rather than blocking the receiver.
No start before first boot completes. The collector does not start until first boot has created this VM's certificates, so a listener is never up without encryption material or with material inherited from the image.
No cryptographic material in the image. No two customers share a trust root.

Why this image uses the distro build

This image installs rsyslog from the Ubuntu 24.04 archive rather than from the upstream project's own package repository or from source. A log collector is infrastructure that must keep taking security fixes for years with nobody logged into it. The distro build is wired into Ubuntu's unattended-upgrades security pocket, with support through 2029.

The upstream project's newest release line moves faster. This image deliberately trades that for a receiver that keeps patching itself.

What you get:

rsyslog 8.2312.0 from the Ubuntu 24.04 archive, with rsyslog-relp and rsyslog-gnutls, configured as a central collector
A per-host, per-program log tree under /var/log/remote, with the path components sanitised so a hostile sender cannot write outside it
A per-VM certificate authority, server certificate and client certificate, generated on first boot
A retention policy that rotates on size, compresses and keeps 14 generations, driven by its own hourly timer
Two self-test tools, so you can prove reception and inspect the exposed surface at any time
A paired deployment guide and 24/7 support from SecureAnyCloud

Before you start

You need an Azure subscription, permission to create a virtual machine, and an SSH key pair. Nothing needs to be installed locally beyond an SSH client.

Decide up front which network your log sources live on. You will restrict the collector to that network shortly, and it is easier to do so once than to remember later.

Deploy the virtual machine

Deploy from the Azure Marketplace listing and choose the following:

Size: Standard_B2s is the recommended starting point. A collector is I/O-bound rather than CPU-bound. If you receive from hundreds of hosts, give it more disk rather than more cores.
Authentication: an SSH public key (recommended) or a password, with the admin username azureuser. The image honours whichever you choose in the portal.
Disk: the default operating system disk holds the received logs at /var/log/remote. For a busy fleet, attach a larger disk and mount it there.

Network Security Group rules

By default, the Marketplace deployment opens only SSH. That is the correct starting point. The collector is also restricted internally, and there is no reason to expose a receiver before you decide who may reach it.

Add rules for the receivers you intend to use. Scope each rule to the source address range of your log sources, never to Any:

# Replace 10.20.0.0/16 with the network your log sources are on.
az network nsg rule create --resource-group <your-resource-group> \
  --nsg-name <your-nsg> --name allow-syslog-udp --priority 1001 \
  --source-address-prefixes 10.20.0.0/16 --destination-port-ranges 514 \
  --protocol Udp --access Allow

Connect and read this VM's credentials

Connect to the VM over SSH:

ssh azureuser@<collector-ip>

Your VM generated its own certificate authority and fingerprints on first boot and recorded them in a file that only root can read. Print it:

sudo cat /root/rsyslog-credentials.txt

The file lists the collector's address, the three ports, the paths to the certificate authority and the client certificate and key, and the SHA-256 fingerprints of all three certificates. The fingerprints are unique to this virtual machine. Nothing in that file exists on any other SecureAnyCloud deployment.

The private key of the authority lives at /etc/rsyslog.d/tls/ca.key. Treat it as a secret. All of your log sources' certificates are issued from it, so back it up somewhere you can reach if this VM is ever lost.

Confirm the collector is running

Check the services and the listening sockets:

systemctl is-active rsyslog.service
systemctl is-active rsyslog-firstboot.service ss -lntu | grep -E ':(514|2514)'

rsyslog.service should be active. The first-boot unit should also be active: it is a one-shot unit that stays active after it succeeds. You should see three receivers: UDP 514, TCP 514 and TCP 2514.

If rsyslog.service is not active, check the first-boot unit first. The collector is deliberately gated on it, so a failed first boot shows up as a collector that never started, not as a collector listening without encryption material:

sudo systemctl status rsyslog-firstboot.service --no-pager | head -20

Send your first message

The quickest proof is to have the collector receive a message over the network from itself. logger is part of the base system on every Ubuntu host. Send a test message:

logger --server 127.0.0.1 --port 514 --tcp \
  --tag guidetest --priority local4.warning \
  "sac guide test message"

Then find it. The collector files messages by the hostname in the message and the program that sent them, so this message lands under this VM's own hostname:

sleep 3
sudo find /var/log/remote -type f -name 'guidetest.log'
sudo find /var/log/remote -type f -name 'guidetest.log' -exec tail -n 1 {} \;

The line that comes back looks like this:

2026-07-26T12:36:16.035147+00:00 collector-host 127.0.0.1 local4.warning guidetest sac guide test message

Read left to right, the fields are: an RFC3339 timestamp, the hostname the sender claimed, the address the message actually came from, the facility and severity, the program tag, and the message.

Both the claimed hostname and the real source address are recorded on every line, on purpose. Syslog lets a sender put any value in the hostname field, so the source address is what makes a message attributable.

Where your logs land

List every received log file:

sudo find /var/log/remote -type f | sort

The tree holds one directory per sending host and one file per program inside it. It is owned by syslog:adm with mode 0750, so it is not world-readable.

A remote host controls both the hostname and the program tag in the messages it sends, and both are used to build a path. Each is sanitised before use, so a sender that puts ../../../../tmp/x in its hostname cannot write outside the tree. In practice, rsyslog's parser rejects a malformed hostname outright and files the message under the sender's IP address instead, which is the behaviour you want.

Open the collector to your own network

This is the only file you need to edit:

sudo cat /etc/rsyslog.d/10-sac-allowed-senders.conf

As shipped, it permits loopback and the three RFC1918 private ranges. Narrow it to the network your log sources actually live on. Replace the two directives with your own range:

$AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16 $AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16

Then restart the collector:

sudo systemctl restart rsyslog

Do not add 0.0.0.0/0. No configuration of a plaintext syslog receiver is safe to expose to the public internet. To receive logs from outside your own network, use the RELP receiver on 2514, which is encrypted and authenticated.

That file deliberately has no entry for RELP. Its access control is the client certificate, not the source address.

Point a Linux host at the collector

On each host whose logs you want, add one file to its rsyslog configuration.

Plaintext, for hosts inside your private network: create /etc/rsyslog.d/90-forward-to-collector.conf:

# Forward everything to the sac central log server.
# The disk-assisted queue means the sending host buffers rather than
# dropping messages if the collector is briefly unreachable. *.* action(type="omfwd" target="COLLECTOR_ADDRESS" port="514" protocol="tcp" queue.type="LinkedList" queue.filename="fwd-collector" queue.maxDiskSpace="256m" queue.saveOnShutdown="on" action.resumeRetryCount="-1")

Replace COLLECTOR_ADDRESS with your collector's private address, then run sudo systemctl restart rsyslog on the sending host. Within seconds, a directory named after that host appears under /var/log/remote on the collector.

Encrypted, authenticated delivery with RELP

RELP is what makes rsyslog worth choosing for central collection. The sender holds each message until the collector acknowledges it, so a collector restart, a network interruption or a full buffer does not silently lose the log lines in flight.

On this image, RELP is TLS-only and requires a client certificate. Copy three files from the collector to the sending host:

sudo scp /etc/rsyslog.d/tls/ca.crt /etc/rsyslog.d/tls/client.crt \
  /etc/rsyslog.d/tls/client.key user@sending-host:/etc/rsyslog.d/tls/

On the sending host, install rsyslog-relp and rsyslog-gnutls, and put the three files in /etc/rsyslog.d/tls/, owned by root:syslog with mode 0640. Then create /etc/rsyslog.d/90-forward-relp.conf:

module(load="omrelp") *.* action(type="omrelp" target="COLLECTOR_ADDRESS" port="2514" tls="on" tls.caCert="/etc/rsyslog.d/tls/ca.crt" tls.myCert="/etc/rsyslog.d/tls/client.crt" tls.myPrivKey="/etc/rsyslog.d/tls/client.key" tls.authMode="name" tls.permittedPeer=["sac-rsyslog-collector"] queue.type="LinkedList" queue.filename="relp-collector" queue.maxDiskSpace="256m" queue.saveOnShutdown="on" action.resumeRetryCount="-1")

Both ends now authenticate each other. The sender checks the collector's certificate against the authority, and the collector checks the sender's.

A certificate signed by anything other than this VM's own authority is refused at the handshake, including one that carries the exact permitted name.

This matters because rsyslog offers a certvalid authentication mode that sounds stricter than the mode used here. On the librelp version that Ubuntu 24.04 ships, certvalid does not enforce the client certificate at all. This image uses name mode, which was measured rejecting both kinds of forgery.

Issuing certificates for more log sources

Every log source can share the one client certificate. Alternatively, issue one certificate per source, so that a single compromised host can be revoked on its own. Issue certificates from the collector:

sudo openssl req -newkey ec:<(openssl ecparam -name prime256v1) -nodes \
  -keyout /etc/rsyslog.d/tls/host2.key -subj "/O=sac/CN=sac-relp-client" \
  -out /tmp/host2.csr

Keep the common name as sac-relp-client, unless you also add the new name to tls.permittedPeer in /etc/rsyslog.d/30-sac-collector-inputs.conf.

Verify the whole thing, at any time

The image carries its own end-to-end check. It stands up simulated remote log sources on loopback addresses, sends a message from a different source address over each of the three transports, and asserts that each one landed in the correct file with the right facility, severity and timestamp. It then proves the refusals: a sender outside the permitted networks, and two forged certificates.

Run the check:

sudo /usr/local/sbin/rsyslog-roundtrip-check.sh

A healthy collector prints a single ROUNDTRIP_OK line. Any other output names the specific check that failed.

To see what this machine actually exposes, enumerated from the live socket table rather than from what the configuration claims, run:

sudo /usr/local/sbin/rsyslog-surface-audit.sh

Retention and disk safety

A collector without retention fills its disk and takes itself down. This image ships with a retention policy:

cat /etc/sac/logrotate-rsyslog-remote.conf systemctl list-timers sac-rsyslog-logrotate.timer --no-pager

As shipped, the policy rotates a file when it passes 128 MB or once an hour, whichever comes first. It keeps 14 generations, compresses everything older than the current one, and discards anything over 30 days old.

Two details are worth knowing:

The policy is not in /etc/logrotate.d, on purpose. It has its own hourly timer and its own state file. If it were in the usual place, the daily system logrotate run would also process it against a different state file, and the two runs would rotate each other's files out of step.
The timer is hourly, not daily. maxsize only takes effect when logrotate is actually invoked, and a busy collector can write many gigabytes between two daily runs.

To change retention, edit that file. It uses ordinary logrotate syntax.

SSH and system hardening

Root logins, empty passwords and X11 forwarding are switched off in /etc/ssh/sshd_config.d/00-sac.conf, which sorts before every other SSH setting. Whether SSH accepts a password follows the choice you made when you created the VM (SSH key or password). Check the effective values:

sudo sshd -T | grep -E '^(permitrootlogin|permitemptypasswords|x11forwarding|pubkeyauthentication) '

permitrootlogin no pubkeyauthentication yes x11forwarding no permitemptypasswords no

The firmware update refresh timer (fwupd-refresh) is masked. An Azure VM has no firmware that LVFS can update, so the timer would only add an outbound call. The fwupd package itself stays installed.

systemctl show -p Id -p UnitFileState fwupd-refresh.timer fwupd-refresh.service

Id=fwupd-refresh.timer UnitFileState=masked Id=fwupd-refresh.service UnitFileState=masked

Nothing from the image build ships on your VM: no certificate authority or key (yours is minted on first boot), no firmware client key, no SSH host key and no received logs. The free space of the disk was zeroed before the image was captured.

Troubleshooting

Nothing is arriving from a remote host. Work outwards. On the collector, check that the sender is permitted (/etc/rsyslog.d/10-sac-allowed-senders.conf must include its address range) and look for the refusal:

sudo journalctl -u rsyslog -n 30 --no-pager | grep -i 'disallowed sender' \
  || echo "no disallowed-sender entries in the last 30 lines"

A UDP message from disallowed sender discarded line means the collector received the message and dropped it on policy, which is the sender policy working. If there is no such line, the packet never arrived, so the problem is the Network Security Group rule or the sending host's own configuration.

RELP connections are refused. The collector logs the reason:

sudo journalctl -u rsyslog -n 40 --no-pager | grep -i 'authentication error' \
  || echo "no RELP authentication errors in the last 40 lines"

A certificate validation failed message means the client certificate was not issued by this VM's authority. Copy ca.crt, client.crt and client.key from this collector to the sending host again. A certificate from a different SecureAnyCloud deployment does not work, by design.

Do not delete a live log file. rsyslog keeps the per-host output files open for performance, so removing one sends the next messages to a deleted file, and they are lost silently until the handle is recycled. Truncate the file instead:

sudo truncate -s 0 /var/log/remote/some-host/some-program.log

This is also why the retention policy uses copytruncate.

The collector is not running after a reboot. Check the first-boot unit, because the collector is gated on it by design:

systemctl is-active rsyslog-firstboot.service ls -l /var/lib/sac/data/

Local logs stopped. This machine is a log server and also logs itself. Confirm that local logging still works:

logger --tag localcheck "local capture check" sleep 2
sudo grep -c localcheck /var/log/syslog

Support

Every SecureAnyCloud image comes with 24/7 support. Contact us at sales@secureanycloud.com or visit secureanycloud.com.

Include the output of sudo /usr/local/sbin/rsyslog-surface-audit.sh and sudo /usr/local/sbin/rsyslog-roundtrip-check.sh with any report about reception. Between them, they identify almost every configuration problem immediately.

Deploy on Azure

Launch rsyslog 8.2312.0 on Ubuntu 24.04 on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure