Apache Kafka 4.4

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Apache Kafka 4.4 on Ubuntu 24.04 as the software version and choose your Region. Then choose Launch through EC2. Choose m5.large, your key pair and a security group as described above.

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=kafka}]'

First boot takes about two minutes after the instance is running. The boot service generates the passwords and the certificate and formats the KRaft storage. Kafka then starts and writes its first metadata snapshot, and AKHQ and nginx start after it.

Version SSH user

Apache Kafka 4.4 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-dns>

systemctl is-active kafka-metadata-names kafka-firstboot kafka akhq nginx
sudo test -f /var/lib/sac/data/kafka-firstboot.done && echo FIRSTBOOT_DONE

All five units are active, and the first-boot marker is present:

active active active active active FIRSTBOOT_DONE

Only SSH, nginx and the Kafka SASL_SSL listener (9094) listen on network ports. Kafka's local SASL listener (9092), its controller (9093) and AKHQ (8080, 28081) listen on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

*:9094 users:(("java")) 0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx")) [::ffff:127.0.0.1]:28081 users:(("java")) [::ffff:127.0.0.1]:8080 users:(("java")) [::ffff:127.0.0.1]:9092 users:(("java")) [::ffff:127.0.0.1]:9093 users:(("java"))

Everything you need is in /root/kafka-credentials.txt, which only root can read. Show it with sudo cat /root/kafka-credentials.txt. This is the file with the two passwords hidden:

sudo sed -E 's/(PASSWORD=).*/\1<hidden>/' /root/kafka-credentials.txt

# Apache Kafka 4.4 on Ubuntu 24.04 - generated on first boot by kafka-firstboot.service.
# Every password below is unique to this instance; none was baked into the image. Keep this file secret.
#
# Kafka from other hosts: SASL_SSL with SCRAM-SHA-512 on KAFKA_BOOTSTRAP_TLS (open port 9094 in your security group
# to your client networks only). Client config template: /root/kafka-client-tls.properties. The broker certificate is
# self-signed (minted on this instance, never a CA): pin it in a truststore file used only by your Kafka client and
# check it is this one -
# openssl x509 -in kafka-broker.pem -noout -fingerprint -sha256 must print KAFKA_TLS_CERT_SHA256
# or install a CA-issued certificate:
sudo kafka-install-cert <chain.pem> <key.pem>
# Kafka on this instance: 127.0.0.1:9092 (sudo kafka-local topics --list). More users:
sudo kafka-scram-user add <name> KAFKA_BOOTSTRAP_TLS=<public-dns>:9094 KAFKA_BOOTSTRAP_LOCAL=127.0.0.1:9092 KAFKA_SASL_MECHANISM=SCRAM-SHA-512 KAFKA_ADMIN_USER=admin KAFKA_ADMIN_PASSWORD=<hidden> KAFKA_TLS_CERT_SHA256=<sha256-fingerprint>
#
# AKHQ web UI: open AKHQ_URL and log in as AKHQ_ADMIN_USER (same certificate; compare the fingerprint your browser shows). AKHQ_URL=https://<public-dns>/ AKHQ_ADMIN_USER=admin AKHQ_ADMIN_PASSWORD=<hidden>
# Change the advertised/served name:
sudo kafka-set-hostname <dns-name> Re-verify the appliance:
sudo kafka-selfcheck
# User guide: https://secureanycloud.com/stacks/apache-kafka-4-4/

The bootstrap address is the instance's public DNS name from the EC2 instance metadata. Inside your VPC it resolves to the private address, and outside it resolves to the public one. Without a public name, the public IP or private address is used. The certificate names the public and private DNS names and IP addresses too.

If you stop and start the instance without an Elastic IP, the boot service advertises the new name and issues a new certificate for it. To keep one name, use your own DNS name (Step 9).

These commands print the broker certificate's SHA-256 fingerprint from the public copy of the certificate:

openssl x509 -in /etc/sac/kafka-broker.pem -noout -fingerprint -sha256

sha256 Fingerprint=<sha256-fingerprint>

kafka-local runs any Kafka command line tool against the broker's loopback SASL listener with your admin client configuration (/root/kafka-client.properties), so no password ever goes on a command line. The image creates one topic, SecureAnyCloud, to get you started. Topics are never created automatically:

sudo kafka-local topics --list

sac

Produce three keyed messages and read them back as a consumer group:

printf '%s\n' 'order-1001:{"item":"SKU-0042","qty":2}' 'order-1002:{"item":"SKU-0007","qty":1}' 'order-1003:{"item":"SKU-0042","qty":5}' \
  | sudo kafka-local console-producer --topic sac --reader-property parse.key=true --reader-property key.separator=:
sudo kafka-local console-consumer --topic sac --from-beginning --max-messages 3 --formatter-property print.key=true --group quickstart

order-1001 {"item":"SKU-0042","qty":2} order-1002 {"item":"SKU-0007","qty":1} order-1003 {"item":"SKU-0042","qty":5} Processed a total of 3 messages

Create your own topics the same way, for example sudo kafka-local topics --create --topic orders --partitions 3 --replication-factor 1. On a single node, the replication factor is always 1. The tools themselves are in /opt/kafka/bin. Run them with --bootstrap-server 127.0.0.1:9092 --command-config /root/kafka-client.properties.

Clients on other hosts connect to KAFKA_BOOTSTRAP_TLS (port 9094) with SASL_SSL and SCRAM-SHA-512. You need two things: a password, and a way for the client to trust the broker's certificate.

Trust the broker by pinning its certificate. The certificate is self-signed and unique to this instance. Copy its public half to the client host and check its SHA-256 fingerprint against KAFKA_TLS_CERT_SHA256 in /root/kafka-credentials.txt (Step 4). Then pin it in a truststore file that only this Kafka client uses. Never add it to the operating system's, a browser's or the Java runtime's own trust store. If you prefer certificates your clients already trust, install a certificate from a certificate authority with kafka-install-cert (Step 9) and leave the truststore lines out.

On the client host, copy the certificate and check the fingerprint. It must print the same value as KAFKA_TLS_CERT_SHA256:

scp -i <your-key.pem> ubuntu@<public-dns>:/etc/sac/kafka-broker.pem ./kafka-broker.pem openssl x509 -in kafka-broker.pem -noout -fingerprint -sha256

Then write a client configuration file on the client host, readable by you only. Put the password in the file, never on a command line. Use KAFKA_ADMIN_PASSWORD, or better, a user of its own from Step 7:

# on <client-host> umask 077 cat > client.properties <<'EOF' security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="<password>"; ssl.truststore.type=PEM ssl.truststore.location=/path/to/kafka-broker.pem EOF

Produce and consume over the network with the Kafka 4.4 command line tools on the client host:

# on <client-host> echo 'hello from another host' | kafka-console-producer.sh --bootstrap-server <public-dns>:9094 --command-config client.properties --topic sac kafka-console-consumer.sh --bootstrap-server <public-dns>:9094 --command-config client.properties --topic sac --from-beginning --max-messages 1

Client libraries use the same settings. In librdkafka-based clients (Python confluent-kafka, Go, .NET, Node.js), set security.protocol=SASL_SSL, sasl.mechanism=SCRAM-SHA-512, sasl.username, sasl.password and ssl.ca.location=/path/to/kafka-broker.pem. The broker checks the client's host name against the certificate, so connect with one of the names it carries (openssl x509 -in kafka-broker.pem -noout -ext subjectAltName). A ready-made template, with the admin password and the on-instance path of the certificate, is at /root/kafka-client-tls.properties.

Amazon MSK's IAM authentication does not apply here. The broker has no route to the instance metadata service, and clients authenticate with SCRAM over TLS.

kafka-scram-user adds, rotates and deletes SCRAM-SHA-512 users. It generates each password on the instance and writes it into a ready-to-use client configuration, /root/kafka-users/<name>-client-tls.properties (readable by root only). It never prints the password and never puts it on a command line, because it reaches Kafka through a root-only file. The users the image itself manages, broker and akhq, cannot be changed with it.

sudo kafka-scram-user list

admin SCRAM-SHA-512=iterations=4096 broker SCRAM-SHA-512=iterations=4096 akhq SCRAM-SHA-512=iterations=4096

Give each application its own user, then copy its client file and the broker certificate to the application host (Step 6):

sudo kafka-scram-user add orders-service
sudo kafka-scram-user rotate orders-service
sudo kafka-scram-user delete orders-service

rotate admin also updates /root/kafka-credentials.txt and /root/kafka-client*.properties. Do not create or delete SCRAM users with kafka-configs.sh --add-config-file. It mangles the SCRAM mechanism name and prints the password in its error message.

Authorization. Like Amazon MSK's default, this broker has no authorizer, so every authenticated user can create and delete topics and read every topic. Before you give a login to a less trusted application, enable Kafka's StandardAuthorizer with ACLs (authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer and super.users=User:admin;User:broker;User:akhq;User:ANONYMOUS in /etc/kafka/server.properties, where ANONYMOUS is the loopback-only controller listener), then grant each user its own topics with sudo kafka-local acls --add .... See the Kafka authorization documentation.

Open AKHQ_URL (https://<public-dns>/) in your browser. The browser warns that the certificate is self-signed. Open the certificate details, check that its SHA-256 fingerprint matches the one from Step 4, and only then continue to the site. Log in as admin with AKHQ_ADMIN_PASSWORD.

The Topics page lists your topics. A topic's Data tab shows its messages with keys, partitions and offsets, including a live tail.

Consumer Groups shows every group and its lag.

Nodes shows the broker as AKHQ sees it over the loopback listener.

AKHQ logs in to Kafka as its own SCRAM user, akhq, whose password is in /etc/akhq/secrets.yml (root and the akhq group only). To add AKHQ users or manage other clusters, edit /etc/akhq/secrets.yml (logins) or /etc/akhq/application.yml (connections) and run sudo systemctl restart akhq. AKHQ refuses to start with HTTP Basic or anonymous access switched on, on a public address, or without its own session secret.

To advertise Kafka and serve AKHQ under your own DNS name, point the name at the instance's Elastic IP first. Then run:

sudo kafka-set-hostname kafka.example.com

This sets Kafka's advertised listener to kafka.example.com:9094, issues a new self-signed certificate for the name, restarts Kafka and reloads nginx. Clients pin the new certificate (Step 6).

To use a certificate from a certificate authority instead, for example from your company CA or Let's Encrypt, put the certificate chain (server certificate first) and its private key on the instance. Then run:

sudo kafka-install-cert /path/to/fullchain.pem /path/to/privkey.pem kafka.example.com

Kafka (9094) and AKHQ (443) then use your certificate. Clients that already trust your CA connect without a pinned truststore, and the boot service never replaces the certificate. sudo kafka-set-hostname <name> returns to a self-signed certificate.

kafka-selfcheck re-verifies the whole appliance with your instance's own credentials. It runs a produce and consume round trip over the loopback listener and over SASL_SSL with the pinned certificate, and it proves that wrong passwords are refused. It also checks that port 9094 speaks only TLS, and that the AKHQ login, session secret, origin guard and cross-site guard work. It confirms that the internal services refuse network connections and that the metadata service is blocked. Finally, it creates, uses and deletes a temporary Kafka user while checking that no password reaches a process command line, the journal or a log file:

sudo kafka-selfcheck

Apache Kafka appliance self-check (SASL_SSL <public-dns>:9094, AKHQ https://<public-dns>/) [ok] loopback SASL (SCRAM-SHA-512, admin) on 127.0.0.1:9092: topics listed (sac present), produce + consume round trip [ok] wrong SCRAM password on 127.0.0.1:9092 -> refused by SCRAM (invalid credentials) [ok] SASL_SSL (TLS + SCRAM-SHA-512, admin) on <public-dns>:9094: produce + consume round trip with the PINNED certificate and hostname verification [ok] wrong SCRAM password on SASL_SSL -> refused by SCRAM (invalid credentials) [ok] KRaft metadata snapshot present (<snapshot>); snapshots every 60 s keep SCRAM users safe across restarts (Kafka 4.4.0 ScramDelta bug) [ok] wire: 9092 answers ApiVersions but drops Metadata before SASL; 9094 speaks TLS only (TLS 1.3 default, TLS 1.2 accepted, nothing older enabled; plaintext Kafka request: no response), drops Metadata before SASL; a system-CA-only client refuses the self-signed certificate [ok] HTTPS AKHQ login page -> 200 [ok] topic API without a session -> 401; HTTP Basic (even with the right password) -> 401 [ok] wrong password -> 303 back to the login page, no session cookie [ok] admin login with the per-instance password -> session; AKHQ (its own SCRAM login) lists the topics and the self-check consumer group [ok] session token is HS256-signed with this instance's own secret (re-signed copy -> 200) [ok] tokens signed with AKHQ's built-in default secret or a random secret -> 401 [ok] port 80 -> 301 https://<public-dns>/; unknown Host -> 444; CONNECT refused; cross-origin POST -> 403; WebSocket upgrade -> 403 [ok] AKHQ (8080/28081) and Kafka LOCAL/CONTROLLER (9092/9093) refuse the private address; SASL_SSL 9094 answers on it; AKHQ speaks no TLS itself [ok] instance metadata refused inside the kafka, akhq and nginx cgroups (loopback control 200) [ok] TLS certificate: end-entity (CA:FALSE, serverAuth, no certificate signing), names <public-dns>; the same on 443 and 9094; SHA-256 <sha256-fingerprint> [ok] cross-site GET guard: 13/13 (/logout); Kafka Connect restart/pause/resume/task-restart: 24/24 foreign, none or missing signals -> nginx 403 [ok] SCRAM user sac-selfcheck-<random>: created by kafka-scram-user (ConfigCommand via a 0600 java @argfile), logged in over SASL_SSL, deleted -> refused; argv sampler snapshots=662 distinct=48 values=3 hits=0; journal + /var/log: 0 hits for the new user's, admin and AKHQ passwords SELFCHECK OK - Kafka SASL_SSL (pinned end-entity certificate, SCRAM-SHA-512) and loopback SASL round trips, wrong passwords refused, TLS-only off-host listener, AKHQ login and session secret, origin and cross-site guards, loopback-only internals, metadata block verified.

The metadata block is a systemd IPAddressDeny on every service. The one exception is kafka-metadata-names. It runs at boot, only reads the instance's DNS names and addresses for the advertised listener, and may reach nothing but the metadata service:

for u in kafka akhq nginx kafka-firstboot kafka-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u)" "$(systemctl show -p IPAddressAllow --value $u)"; done

kafka: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow= akhq: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow= nginx: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow= kafka-firstboot: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow= kafka-metadata-names: deny=::/0 0.0.0.0/0 allow=fd00:ec2::254/128 169.254.169.254/32

/opt/kafka/bin/kafka-topics.sh --version 2>/dev/null | tail -1 java -version 2>&1 | head -1 sudo kafka-local metadata-quorum describe --status 2>/dev/null | grep -E 'LeaderId|CurrentVoters'

4.4.0 openjdk version "21.0.12.1" 2026-08-18 LeaderId: 1 CurrentVoters: [{"id": 1, "endpoints": ["CONTROLLER://127.0.0.1:9093"]}]

What Where

Kafka configuration /etc/kafka/server.properties (listeners, advertised listener, retention)

Kafka data (topics) and KRaft metadata /var/lib/kafka/data/logs, /var/lib/kafka/data/metadata

Kafka logs /var/log/kafka and journalctl -u kafka

Broker TLS keystore (key + certificate, Kafka group only) /etc/kafka/tls/kafka-keystore.pem

Public broker certificate (to pin) /etc/sac/kafka-broker.pem

Your passwords and addresses /root/kafka-credentials.txt, /root/kafka-client.properties, /root/kafka-client-tls.properties, /root/kafka-users/

AKHQ configuration and secrets /etc/akhq/application.yml, /etc/akhq/secrets.yml

Build provenance and licences /usr/share/doc/apache-kafka/, /usr/share/doc/akhq/

Two settings in server.properties work around a bug in Kafka 4.4.0: metadata.log.max.snapshot.interval.ms=60000 and metadata.max.retention.ms=3600000. Without frequent metadata snapshots, deleting a SCRAM user could lock every user out at the next restart. Keep both settings, because Kafka refuses to start without the first.

• Back up /var/lib/kafka/data (with Kafka stopped, or with EBS snapshots of the volume) together with /etc/kafka, /etc/akhq and /root/kafka-*. The SCRAM users and the cluster ID live in the KRaft metadata, so a data copy without those files cannot be opened with new passwords.
• Storage. Topics live on the root volume. Grow the EBS volume (and the file system with sudo growpart and sudo resize2fs) as your retention needs, or change log.retention.hours (7 days by default).
• Updates. Ubuntu security updates install automatically. Kafka and AKHQ are pinned releases, and SecureAnyCloud publishes new AMI versions with newer releases.
• Logs. Read them with journalctl -u kafka -u akhq -u nginx and in /var/log/kafka/server.log.

• Restrict ports 9094, 443 and 22 in your security group to the networks that need them.
• Give each application its own Kafka user (Step 7), and keep the admin password for administration. Rotate passwords with kafka-scram-user rotate.
• Enable an authorizer and ACLs before you share the cluster with less trusted clients (Step 7).
• Pin the broker certificate in each client's own truststore after checking the fingerprint, or install a CA-issued certificate (Steps 6 and 9).
• Keep /root/kafka-credentials.txt and the client files private. They are root-only on the instance.