Fluent Bit 5.1.4

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Fluent Bit 5.1.4 on Ubuntu 24.04 as the software version, choose your Region, then choose Launch through EC2.

Choose m5.large, your key pair and a security group as described above. If you will use the AWS outputs, choose your IAM role. Under Advanced details, set Metadata version to V2 only (token required) and Metadata response hop limit to 1.

The AMI is IMDSv2 only. Set the hop limit to 1 at launch, because an AMI cannot carry it. This stops a container on the instance from borrowing the instance role. To launch with the AWS CLI, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --iam-instance-profile Name=<your-instance-profile> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled,HttpPutResponseHopLimit=1 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fluent-bit}]'

Connect as the ubuntu user (Fluent Bit 5.1.4 on Ubuntu 24.04):

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate and the shared key. Then Fluent Bit starts as the fluentbit user. Check the service:

systemctl is-active fluent-bit.service
sudo test -f /var/lib/sac/data/fluent-bit-firstboot.done && echo FIRSTBOOT_DONE /opt/fluent-bit/bin/fluent-bit --version | head -1 ps -o user=,args= -C fluent-bit

active FIRSTBOOT_DONE Fluent Bit v5.1.4 fluentb+ /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf

Check the health endpoint of the monitoring API. It answers on loopback:

curl -s http://127.0.0.1:2020/api/v1/health; echo

ok

Confirm what listens on the network. Only SSH and the forward input listen on public addresses. The monitoring API and the local DNS resolver stay on loopback:

sudo ss -Hltn | awk '{print $4}' | sort -u

0.0.0.0:22 0.0.0.0:24224 127.0.0.1:2020 127.0.0.53%lo:53 127.0.0.54:53 [::]:22

The address, port and certificate path your agents need are in the root-only credentials file:

sudo grep -E '^FLUENT_BIT_(FORWARD_HOST|FORWARD_PORT|SERVER_HOSTNAME|CA_CERT)=' /root/fluent-bit-credentials.txt

FLUENT_BIT_FORWARD_HOST=<private-ip> FLUENT_BIT_FORWARD_PORT=24224 FLUENT_BIT_SERVER_HOSTNAME=ip-<private-ip-dashed>.ec2.internal FLUENT_BIT_CA_CERT=/etc/fluent-bit/tls/forward.crt

The shared key is the FLUENT_BIT_SHARED_KEY line of the same file. Read it when you configure an agent, sudo grep '^FLUENT_BIT_SHARED_KEY=' /root/fluent-bit-credentials.txt. Keep it in your secrets store. It is unique to this instance.

The certificate names the instance's private and public addresses and EC2 host names. Send agents to the private address, because it does not change when the instance stops and starts.

sudo openssl x509 -in /etc/fluent-bit/tls/forward.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ip-<private-ip-dashed>.ec2.internal, O = sac X509v3 Subject Alternative Name: IP Address:<private-ip>, IP Address:<public-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:127.0.0.1, DNS:localhost notAfter=<about 27 months after first boot>

Other users cannot read the credentials file, the input configuration that holds the key, or the private key:

sudo stat -c '%n mode=%a owner=%U:%G' /root/fluent-bit-credentials.txt /etc/fluent-bit/forward-secure.conf /etc/fluent-bit/tls/forward.key

/root/fluent-bit-credentials.txt mode=600 owner=root:root /etc/fluent-bit/forward-secure.conf mode=640 owner=root:fluentbit /etc/fluent-bit/tls/forward.key mode=640 owner=root:fluentbit

fluent-bit-forward-test sends one record to the local forward input with a throwaway Fluent Bit client and checks that it arrived. Run it with TLS and the right key, then with a wrong key, then without TLS. Only the first is accepted:

sudo fluent-bit-forward-test
sudo fluent-bit-forward-test --wrong-key || echo "refused as expected"
sudo fluent-bit-forward-test --no-tls || echo "refused as expected"

ACCEPTED sac-forward-test-<timestamp> NOT-ACCEPTED sac-forward-test-<timestamp>-wrong-key refused as expected NOT-ACCEPTED sac-forward-test-<timestamp>-no-tls refused as expected

Check the log. The forward input records why it refused the wrong key:

sudo journalctl -u fluent-bit.service --no-pager -o cat | grep -a 'shared_key mismatch' | tail -1

[<date> <time>] [error] [input:forward:forward.0] shared_key mismatch

The aggregator writes the records it receives to /var/log/fluent-bit/records.log until you add your own outputs (Step 8):

sudo tail -n 1 /var/log/fluent-bit/records.log

sac.forward.test: [<epoch>.<ns>, {"message":"sac-forward-test-<timestamp>"}]

Add an inbound rule for TCP 24224 from the address range your agents send from, for example your VPC CIDR:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 24224 --cidr <your-vpc-cidr>

Copy the instance's certificate to each agent host. The agent uses it to verify that it is talking to your aggregator:

ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/fluent-bit/tls/forward.crt' > forward.crt

Fluent Bit agent (classic configuration). Self_Hostname names the agent. The shared key is the one from Step 4:

[OUTPUT] Name forward Match * Host <private-ip> Port 24224 tls on tls.verify on tls.ca_file /etc/fluent-bit/forward.crt Shared_Key <FLUENT_BIT_SHARED_KEY> Self_Hostname <agent-name>

If the agent connects by a name that is not in the certificate, such as your own DNS record, add tls.vhost ip-<private-ip-dashed>.ec2.internal or issue a certificate for that name (Step 9).

Fluentd agent (fluent-plugin-forward is built in):

<match **> @type forward transport tls tls_cert_path /etc/fluent/forward.crt tls_verify_hostname true <security> self_hostname <agent-name> shared_key <FLUENT_BIT_SHARED_KEY> </security> <server> host <private-ip> port 24224 </server> </match>

Restart the agent. Its records appear in /var/log/fluent-bit/records.log on the aggregator with the agent's tags.

The pipeline is /etc/fluent-bit/fluent-bit.conf. Its [OUTPUT] writes everything to /var/log/fluent-bit/records.log. Replace it, or add outputs next to it. The forward input itself lives in forward-secure.conf, which first boot generated. Leave that file as it is.

Fluent Bit uses the instance's IAM role via IMDSv2, and no other service on the image can reach IMDS. With a role attached, the AWS outputs need no access keys. This example ships to CloudWatch Logs:

[OUTPUT] Name cloudwatch_logs Match * region <region> log_group_name /sac/fluent-bit log_stream_prefix agent- auto_create_group On

This example ships to Amazon S3:

[OUTPUT] Name s3 Match * region <region> bucket <your-bucket> total_file_size 50M upload_timeout 5m s3_key_format /fluent-bit/%Y/%m/%d/$TAG-%H%M%S

Edit the file with sudo nano /etc/fluent-bit/fluent-bit.conf. Then check it and restart the service:

sudo -u fluentbit /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run sudo systemctl restart fluent-bit.service

Watch the journal for output errors: sudo journalctl -u fluent-bit.service -f.

To present a certificate from your own certificate authority, such as one that names your DNS record, replace /etc/fluent-bit/tls/forward.crt and /etc/fluent-bit/tls/forward.key. Keep their ownership and modes, then restart:

sudo install -o root -g fluentbit -m 0640 your.key /etc/fluent-bit/tls/forward.key
sudo install -o root -g root -m 0644 your.crt /etc/fluent-bit/tls/forward.crt
sudo systemctl restart fluent-bit.service

Give your agents the new CA certificate instead of forward.crt.

To rotate the shared key, put a new value on the Shared_Key line of /etc/fluent-bit/forward-secure.conf. You can generate one with openssl rand -hex 20. Restart the service and update every agent. The credentials file keeps the first boot value, so update your own records.

The service runs with no capabilities, cannot gain privileges and sees the file system as read-only:

systemctl show -p User -p NoNewPrivileges -p ProtectSystem -p CapabilityBoundingSet fluent-bit.service

CapabilityBoundingSet= User=fluentbit ProtectSystem=strict NoNewPrivileges=yes

Fluent Bit is the one service allowed to reach the instance metadata service, because its AWS outputs sign requests with the instance role. It therefore carries no metadata deny rule:

systemctl show -p IPAddressDeny fluent-bit.service

IPAddressDeny=

SSH accepts key logins only. Password and keyboard-interactive logins, root logins and X11 forwarding are off. The settings live in /etc/ssh/sshd_config.d/00-sac.conf, which sshd reads first. Check the settings sshd actually uses:

sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|x11forwarding) '

permitrootlogin no passwordauthentication no kbdinteractiveauthentication no x11forwarding no

Also note the following:

• The forward input accepts TLS 1.2 and 1.3 only; TLS 1.1 and plaintext connections are refused.
• The key is never on a command line or in the unit's environment. Fluent Bit reads it from forward-secure.conf, which only root and the fluentbit group can read.
• Keep TCP 24224 restricted to your agents' networks in the security group. Use the private address. You do not need to expose the aggregator to the internet.
• The instance is IMDSv2 only. Launch it with a metadata hop limit of 1 (Step 1) so that containers cannot reach the role.
• If you add an input that reads another agent protocol (syslog, HTTP or OpenTelemetry), bind it to 127.0.0.1 or protect it with TLS before you open its port.