Fluentd 1.19.4

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Fluentd 1.19.4 on Ubuntu 24.04 as the software version and select your Region. Then choose Launch through EC2.

Choose m5.large, your key pair and a security group as above. If you will use the S3 output, also choose your IAM role. Under Advanced details, set Metadata version to V2 only (token required) and Metadata response hop limit to 1.

The AMI is IMDSv2 only. Set the hop limit to 1 at launch, because an AMI cannot carry it. This way a container on the instance can never borrow the instance role. To do the same with the AWS CLI, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --iam-instance-profile Name=<your-instance-profile> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled,HttpPutResponseHopLimit=1 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fluentd}]'

Version SSH user

Fluentd 1.19.4 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate and the shared key. Fluentd then starts as the _fluentd user:

systemctl is-active fluentd.service
sudo test -f /var/lib/sac/data/fluentd-firstboot.done && echo FIRSTBOOT_DONE /opt/fluent/bin/fluentd --version dpkg-query -W -f='fluent-package ${Version}\n' fluent-package

active FIRSTBOOT_DONE fluentd 1.19.4 fluent-package 6.0.5-1

The monitor_agent API answers on loopback and lists the running plugins. These are the secure forward input, the file output for received records, and the two housekeeping inputs that fluent-package adds. One housekeeping input warns about obsolete plugins, and the other logs when a newer fluent-package LTS release is available:

curl -s http://127.0.0.1:24220/api/plugins.json | /opt/fluent/bin/ruby -rjson -e 'JSON.parse(STDIN.read)["plugins"].each { |p| puts "#{p["plugin_category"]} #{p["type"]}" }'

input monitor_agent input forward input obsolete_plugins input fluent_package_update_notifier output file output null

Only SSH and the forward input listen on public addresses. The monitoring API and the local DNS resolver stay on loopback:

sudo ss -Hltn | awk '{print $4}' | sort -u

0.0.0.0:22 0.0.0.0:24224 127.0.0.1:24220 127.0.0.53%lo:53 127.0.0.54:53 [::]:22

The forward input also answers the UDP heartbeat on 24224, which carries no data. You only need to open TCP 24224 for agents.

SSH accepts key pairs only. The effective SSH server settings confirm that password logins, root logins and X11 forwarding are off:

sudo sshd -T | grep -E '^(passwordauthentication|permitrootlogin|x11forwarding) ' | sort

passwordauthentication no permitrootlogin no x11forwarding no

The address, port, host name and certificate path your agents need are in the root-only credentials file:

sudo grep -E '^FLUENTD_(FORWARD_HOST|FORWARD_PORT|SERVER_HOSTNAME|CA_CERT)=' /root/fluentd-credentials.txt

FLUENTD_FORWARD_HOST=<private-ip> FLUENTD_FORWARD_PORT=24224 FLUENTD_SERVER_HOSTNAME=ip-<private-ip-dashed>.ec2.internal FLUENTD_CA_CERT=/etc/fluent/tls/forward.crt

The shared key is the FLUENTD_SHARED_KEY line of the same file. Read it when you configure an agent with sudo grep '^FLUENTD_SHARED_KEY=' /root/fluentd-credentials.txt, and keep it in your secrets store. It is unique to this instance.

The certificate names the instance's private and public addresses and EC2 host names. Agents should connect to the private DNS name (FLUENTD_SERVER_HOSTNAME). It is in the certificate, it resolves to the private address inside the VPC, and it does not change when the instance stops and starts.

sudo openssl x509 -in /etc/fluent/tls/forward.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ip-<private-ip-dashed>.ec2.internal, O = sac X509v3 Subject Alternative Name: IP Address:<private-ip>, IP Address:<public-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:127.0.0.1, DNS:localhost notAfter=<about 27 months after first boot>

The credentials file, the input configuration that holds the key and the private key are not readable by other users:

sudo stat -c '%n mode=%a owner=%U:%G' /root/fluentd-credentials.txt /etc/fluent/forward-secure.conf /etc/fluent/tls/forward.key

/root/fluentd-credentials.txt mode=600 owner=root:root /etc/fluent/forward-secure.conf mode=640 owner=root:_fluentd /etc/fluent/tls/forward.key mode=640 owner=root:_fluentd

fluentd-forward-test sends records to the local forward input with a throwaway Fluentd client and checks that one arrived. Run it with TLS and the right key, then with a wrong key, and then without TLS. Only the first run is accepted, and each refused run takes about 20 seconds:

sudo fluentd-forward-test
sudo fluentd-forward-test --wrong-key || echo "refused as expected"
sudo fluentd-forward-test --no-tls || echo "refused as expected"

ACCEPTED sac-forward-test-<timestamp> NOT-ACCEPTED sac-forward-test-<timestamp>-wrong-key refused as expected NOT-ACCEPTED sac-forward-test-<timestamp>-no-tls refused as expected

Fluentd logs why it refused the wrong key:

sudo grep -a 'Shared key mismatch' /var/log/fluent/fluentd.log | tail -1

<date> <time> +0000 [warn]: #0 Shared key mismatch address="127.0.0.1" hostname="forward-test-client"

Records the aggregator receives are written to /var/log/fluentd/records.log (time, tag, record) until you add your own outputs (Step 8):

sudo tail -n 1 /var/log/fluentd/records.log

<date>T<time>+00:00 sac.forward.test {"message":"sac-forward-test-<timestamp>"}

Add an inbound rule for TCP 24224 from the address range your agents send from, for example your VPC CIDR:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 24224 --cidr <your-vpc-cidr>

Copy the instance's certificate to each agent host so the agent can verify it is talking to your aggregator:

ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/fluent/tls/forward.crt' > forward.crt

Fluentd agent (the forward output is built in). self_hostname names the agent, and the shared key is the one from Step 4:

<match **> @type forward transport tls tls_ca_cert_path /etc/fluent/forward.crt tls_verify_hostname true <security> self_hostname <agent-name> shared_key <FLUENTD_SHARED_KEY> </security> <server> host ip-<private-ip-dashed>.ec2.internal port 24224 </server> </match>

If the agent must connect by IP address instead, add name <private-ip> inside <server>. When host is an IP address, Fluentd verifies the certificate against name, and without it the TLS connection fails.

Fluent Bit agent (classic configuration):

[OUTPUT] Name forward Match * Host ip-<private-ip-dashed>.ec2.internal Port 24224 tls on tls.verify on tls.ca_file /etc/fluent-bit/forward.crt Shared_Key <FLUENTD_SHARED_KEY> Self_Hostname <agent-name>

Restart the agent; its records appear in /var/log/fluentd/records.log on the aggregator with the agent's tags.

The pipeline is /etc/fluent/fluentd.conf. Its final <match **> writes everything to /var/log/fluentd/records.log; replace it or put your own <match> blocks above it. The forward input itself lives in forward-secure.conf, which first boot generated; leave it as it is.

Fluentd uses the instance's IAM role through IMDSv2, and no other service on the image can reach IMDS. With a role attached, the bundled S3 output (fluent-plugin-s3) needs no access keys:

<match **> @type s3 s3_region <region> s3_bucket <your-bucket> path fluentd/%Y/%m/%d/ <buffer tag,time> @type file path /var/log/fluent/buffer/s3 timekey 300 timekey_wait 60 chunk_limit_size 64m </buffer> </match>

fluent-package also bundles outputs for OpenSearch, Elasticsearch, Kafka, OpenTelemetry and Prometheus. You add other destinations, for example CloudWatch Logs, as plugins with sudo /opt/fluent/bin/fluent-gem install fluent-plugin-cloudwatch-logs. Plugins from RubyGems run with Fluentd's privileges, so install only the ones you trust.

Edit the file with sudo nano /etc/fluent/fluentd.conf, check it, and restart the service:

sudo -u _fluentd /opt/fluent/bin/fluentd -c /etc/fluent/fluentd.conf --dry-run sudo systemctl restart fluentd.service

Buffers on disk must live under /var/log/fluent, /var/log/fluentd or /var/lib/fluent. The sandbox keeps the rest of the file system read-only for Fluentd.

Watch for output errors in the Fluentd log with sudo tail -f /var/log/fluent/fluentd.log.

To present a certificate from your own certificate authority (for example one that names your DNS record), replace /etc/fluent/tls/forward.crt and /etc/fluent/tls/forward.key, keep their ownership and modes, then restart:

sudo install -o root -g _fluentd -m 0640 your.key /etc/fluent/tls/forward.key
sudo install -o root -g root -m 0644 your.crt /etc/fluent/tls/forward.crt
sudo systemctl restart fluentd.service

Give your agents the new CA certificate instead of forward.crt.

To rotate the shared key, put a new value on the shared_key line of /etc/fluent/forward-secure.conf (generate one with openssl rand -hex 20), restart the service and update every agent. The credentials file keeps the first boot value, so update your own records.

The service runs with no capabilities, cannot gain privileges and sees the file system as read-only:

systemctl show -p User -p NoNewPrivileges -p ProtectSystem -p CapabilityBoundingSet fluentd.service

CapabilityBoundingSet= User=_fluentd ProtectSystem=strict NoNewPrivileges=yes

Fluentd is the one service allowed to reach the instance metadata service, because its S3 output signs requests with the instance role. It therefore carries no metadata deny rule:

systemctl show -p IPAddressDeny fluentd.service

IPAddressDeny=

Also note the following:

• The forward input accepts TLS 1.2 and 1.3 only; older protocols and plaintext connections are refused.
• The key is never on a command line, in the unit's environment or in the monitoring API; Fluentd reads it from forward-secure.conf, which only root and the _fluentd group can read. Fluentd also masks the key in its own log.
• Fluentd logs a warning at start that private_key_passphrase is recommended. The private key is protected by its file permissions instead (0640, root and the _fluentd group only), because a passphrase would have to sit next to it in the configuration.
• Keep TCP 24224 restricted to your agents' networks in the security group. Use the private DNS name. There is no need to expose the aggregator to the internet.
• The instance is IMDSv2 only. Launch it with a metadata hop limit of 1 (Step 1) so containers cannot reach the role.
• If you add an input that reads another agent protocol (syslog, HTTP, OpenTelemetry), bind it to 127.0.0.1 or protect it with TLS before opening its port.
• fluent-package 6.0.5 also bundles the Treasure Data td command line client, which depends on rubyzip 1.3.0 (CVE-2026-85396, path traversal when extracting a crafted archive; fixed in rubyzip 3.4.0). Fluentd, its plugins and the aggregator pipeline never load rubyzip, so the running service is not exposed. Only running td by hand to unpack an untrusted archive would be. When a fluent-package 6 LTS release carries the fix, sudo apt-get install --only-upgrade fluent-package picks it up.