rsyslog 8.2312.0

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick rsyslog 8.2312.0 on Ubuntu 24.04 as the software version and select your Region. Then choose Launch through EC2.

Choose m5.large, your key pair and a security group as described above. A collector is I/O bound, so for hundreds of hosts give it more disk rather than more cores. The AMI is IMDSv2 only.

To launch with the AWS CLI instead, replace the values in angle brackets and run:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=log-collector}]'

Received logs are stored on the root volume under /var/log/remote. Size the volume for your retention policy (see Retention and disk safety), or attach a larger volume and mount it there.

Connect over SSH as the ubuntu user. This is the SSH user for rsyslog 8.2312.0 on Ubuntu 24.04:

ssh -i <your-key.pem> ubuntu@<public-ip>

The login banner shows the receivers, where the logs are stored and where to find this guide.

On first boot, the instance creates its certificate authority and keys, and then releases rsyslog. Check the first-boot unit and rsyslog:

systemctl is-active rsyslog.service
systemctl is-active rsyslog-firstboot.service rsyslogd -v | head -1

active active rsyslogd 8.2312.0 (aka 2023.12) compiled with:

rsyslog-firstboot.service is a one-shot unit that stays active after it succeeds. Three receivers listen: UDP 514, TCP 514 and TCP 2514. SSH is the only other public listener, and the DNS resolver and the time client stay on loopback:

sudo ss -Hlntu | awk '{print $1, $5}' | sort -u

tcp 0.0.0.0:22 tcp 0.0.0.0:2514 tcp 0.0.0.0:514 tcp 127.0.0.53%lo:53 tcp 127.0.0.54:53 tcp [::]:22 tcp [::]:2514 tcp [::]:514 udp 0.0.0.0:514 udp 127.0.0.1:323 udp 127.0.0.53%lo:53 udp 127.0.0.54:53 udp <private-ip>%ens5:68 udp [::1]:323 udp [::]:514

If rsyslog.service is not active, check the first-boot unit first. The collector is deliberately gated on that unit. A failed first boot therefore shows up as a collector that never started, not as one that listens without encryption material:

sudo systemctl status rsyslog-firstboot.service --no-pager | head -20

The address, ports, certificate paths and fingerprints that your log sources need are in a root-only file written on first boot. Read it with:

sudo grep -E '^(COLLECTOR_PRIVATE_IP|SYSLOG_UDP_PORT|SYSLOG_TCP_PORT|RELP_TLS_PORT|RELP_CA_CERT|RELP_CLIENT_CERT|RELP_CLIENT_KEY|CA_FINGERPRINT_SHA256)=' /root/rsyslog-credentials.txt

COLLECTOR_PRIVATE_IP=<collector-ip> SYSLOG_UDP_PORT=514 SYSLOG_TCP_PORT=514 RELP_TLS_PORT=2514 RELP_CA_CERT=/etc/rsyslog.d/tls/ca.crt RELP_CLIENT_CERT=/etc/rsyslog.d/tls/client.crt RELP_CLIENT_KEY=/etc/rsyslog.d/tls/client.key CA_FINGERPRINT_SHA256=<ca-fingerprint>

The fingerprints are unique to this instance. Send logs to the private address, which does not change when the instance stops and starts. The server certificate names the instance's private and public addresses:

sudo openssl x509 -in /etc/rsyslog.d/tls/server.crt -noout -subject -issuer -ext subjectAltName

subject=O = sac, CN = sac-rsyslog-collector issuer=O = sac, OU = rsyslog Central Log Server, CN = sac-rsyslog-ca-<random-id> X509v3 Subject Alternative Name: IP Address:<private-ip>, IP Address:127.0.0.1, DNS:ip-<private-ip-dashed>, DNS:localhost, IP Address:<public-ip>

Other users cannot read the credentials file or the three private keys:

sudo stat -c '%n mode=%a owner=%U:%G' /root/rsyslog-credentials.txt /etc/rsyslog.d/tls/ca.key /etc/rsyslog.d/tls/server.key /etc/rsyslog.d/tls/client.key

/root/rsyslog-credentials.txt mode=600 owner=root:root /etc/rsyslog.d/tls/ca.key mode=640 owner=root:syslog /etc/rsyslog.d/tls/server.key mode=640 owner=root:syslog /etc/rsyslog.d/tls/client.key mode=640 owner=root:syslog

The certificate authority's private key, /etc/rsyslog.d/tls/ca.key, issues every client certificate. Treat it as a secret and back it up somewhere private (see Backing up, below).

The quickest test is to have the collector receive a message over the network from itself. logger is part of every Ubuntu host:

logger --server 127.0.0.1 --port 514 --tcp \
  --tag guidetest --priority local4.warning \
  "sac guide test message" sleep 3
sudo find /var/log/remote -type f -name 'guidetest.log' -exec tail -n 1 {} \;

2026-10-20T21:55:09.456770+00:00 ip-<private-ip-dashed> 127.0.0.1 local4.warning guidetest sac guide test message

Each line contains, from left to right, an RFC3339 timestamp, the hostname the sender claimed, the address the message actually came from, the facility and severity, the program tag and the message. Syslog lets a sender put anything in the hostname field, so the real source address on every line is what makes a message attributable.

The collector creates one directory per sending host and one file per program inside it. The tree is owned by syslog:adm with mode 0750, so it is not world-readable:

sudo find /var/log/remote -type f | sort
sudo stat -c '%n %a %U:%G' /var/log/remote

/var/log/remote/ip-<private-ip-dashed>/guidetest.log /var/log/remote 750 syslog:adm

A remote host controls both the hostname and the program tag that it sends, and the collector uses both to build a path. It sanitises each one first, so a sender that puts ../../../../tmp/x in its hostname cannot write outside the tree. In practice, rsyslog's parser rejects a malformed hostname and files the message under the sender's IP address instead.

Security group. Add inbound rules for the receivers that you will use. Scope them to the address range that your log sources live on, for example your VPC CIDR:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol udp --port 514 --cidr <your-vpc-cidr>
aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 514 --cidr <your-vpc-cidr>
aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 2514 --cidr <your-vpc-cidr>

Sender policy. The plaintext receivers also check the sender's address in the collector itself. You edit one file to change this policy:

sudo grep '^\$AllowedSender' /etc/rsyslog.d/10-sac-allowed-senders.conf

$AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 $AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

As shipped, the file permits loopback and the three RFC1918 private ranges. Narrow it to the network that your log sources actually live on, for example:

$AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16 $AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16

Then run sudo systemctl restart rsyslog.

Never add 0.0.0.0/0. No plaintext syslog receiver is safe to expose to the internet. To receive from outside your private network, use RELP on 2514, which is encrypted and authenticated. The file has no entry for RELP on purpose, because RELP access control is the client certificate, not the source address.

On each host whose logs you want to collect, add one file to its rsyslog configuration. For hosts inside your private network, create /etc/rsyslog.d/90-forward-to-collector.conf:

# Forward everything to the sac central log server.
# The disk-assisted queue buffers rather than drops messages
# if the collector is briefly unreachable. *.* action(type="omfwd" target="COLLECTOR_ADDRESS" port="514" protocol="tcp" queue.type="LinkedList" queue.filename="fwd-collector" queue.maxDiskSpace="256m" queue.saveOnShutdown="on" action.resumeRetryCount="-1")

Replace COLLECTOR_ADDRESS with the collector's private address (COLLECTOR_PRIVATE_IP from Step 4). Then run sudo systemctl restart rsyslog on the sending host. Within seconds, a directory named after that host appears under /var/log/remote on the collector.

RELP is what makes rsyslog worth choosing for central collection. The sender holds each message until the collector acknowledges it, so a collector restart or a network blip does not silently lose the lines in flight. On this image, RELP is TLS only and requires a client certificate.

To configure a sending host, complete these steps:

1. Copy three files from the collector to the sending host: ca.crt, client.crt and client.key from /etc/rsyslog.d/tls/. For example, use scp from your workstation.
2. Install rsyslog-relp and rsyslog-gnutls on the sending host.
3. Put the files in /etc/rsyslog.d/tls/, owned by root:syslog with mode 0640.
4. Create /etc/rsyslog.d/90-forward-relp.conf:

module(load="omrelp") *.* action(type="omrelp" target="COLLECTOR_ADDRESS" port="2514" tls="on" tls.caCert="/etc/rsyslog.d/tls/ca.crt" tls.myCert="/etc/rsyslog.d/tls/client.crt" tls.myPrivKey="/etc/rsyslog.d/tls/client.key" tls.authMode="name" tls.permittedPeer=["sac-rsyslog-collector"] queue.type="LinkedList" queue.filename="relp-collector" queue.maxDiskSpace="256m" queue.saveOnShutdown="on" action.resumeRetryCount="-1")

Both ends now authenticate each other. The sender checks the collector's certificate against the authority, and the collector checks the sender's. A certificate signed by anything other than this instance's own authority is refused at the handshake, including one that carries the exact permitted name.

rsyslog also offers a certvalid mode that sounds stricter. On the librelp version that Ubuntu 24.04 ships, that mode does not enforce the client certificate at all. This image therefore uses name mode, which was measured rejecting both kinds of forgery.

More log sources. Every source can share the one client certificate, or you can issue one to each source so that you can revoke a single compromised host on its own. Issue certificates from the collector and keep the common name sac-relp-client, or add the new name to tls.permittedPeer in /etc/rsyslog.d/30-sac-collector-inputs.conf:

sudo bash -c 'cd /etc/rsyslog.d/tls && umask 077 && openssl ecparam -name prime256v1 -genkey -noout -out host2.key && openssl req -new -key host2.key -subj "/O=sac/CN=sac-relp-client" -out host2.csr && printf "basicConstraints=CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=clientAuth\n" > host2.ext && openssl x509 -req -in host2.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -sha256 -days 825 -extfile host2.ext -out host2.crt && rm -f host2.csr host2.ext'

Copy ca.crt, host2.crt and host2.key to the new source and delete host2.key from the collector.

The image includes its own end-to-end check. The check stands up simulated remote log sources on loopback addresses and sends a message from a different source address over each of the three transports. It asserts that each message landed in the right file with the right facility, severity and timestamp.

It then verifies the refusals (a sender outside the permitted networks, a self-signed client certificate, and a certificate with the permitted name from another authority) and confirms that retention rotates and compresses logs. Run it with:

sudo /usr/local/sbin/rsyslog-roundtrip-check.sh

ROUNDTRIP_OK udp=10.99.99.11->simhost-udp-<pid> tcp=10.99.99.12->simhost-tcp-<pid> relp_tls=10.99.99.13->simhost-relp-<pid> denied_sender=dropped rogue_cert=refused rotation=rotated+compressed

To see what the instance actually exposes, enumerated from the live socket table rather than from the configuration, run:

sudo /usr/local/sbin/rsyslog-surface-audit.sh | tail -n 14

== the shipped sender policy must not be an open relay == $AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 $AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 restricted - ok == rate limiting and the bounded queue must be configured == input rate limiting configured - ok bounded disk-assisted queue configured - ok queue discard guard configured - ok == RELP must be TLS with mutual authentication == relp listener is TLS - ok relp requires a certificate issued by this VM's authority - ok SURFACE_AUDIT_OK expected=22/tcp 514/udp 514/tcp 2514/tcp