Apache Kafka 4.2 on Ubuntu 24.04

sudo systemctl is-active kafka.service akhq.service nginx.service ls /opt/kafka/libs/ | grep -E '^kafka-clients-' | head -1

sudo cat /stage/scripts/kafka-credentials.log

Note the values of KAFKA_BOOTSTRAP, KAFKA_ADMIN_USER and KAFKA_ADMIN_PASSWORD. The same password is used for the AKHQ web UI basic auth.

PASS=$(sudo grep '^KAFKA_ADMIN_PASSWORD=' /stage/scripts/kafka-credentials.log | cut -d= -f2-)
cat > /tmp/sac.properties <<EOF security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="sac" password="${PASS}"; EOF /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties | head -3

Without --command-config, the first call fails. With the SecureAnyCloud SCRAM credentials, it returns the broker API matrix.

Browse to http://<vm-ip>/ and sign in as SecureAnyCloud with the password from Step 5. The Topics tab lists every topic, including the pre-created SecureAnyCloud topic.

The Nodes tab shows the single broker with id 1 and host 127.0.0.1:9092, and it shows the controller leader.

The Consumer Groups tab shows consumer groups and their lag once you have consumers running. On a fresh deployment, the list is empty until you produce and consume a message.

PASS=$(sudo grep '^KAFKA_ADMIN_PASSWORD=' /stage/scripts/kafka-credentials.log | cut -d= -f2-)
cat > /tmp/sac.properties <<EOF security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="sac" password="${PASS}"; EOF echo "hello sac $(date +%s)" | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --topic sac /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --topic sac --from-beginning --max-messages 1 --timeout-ms 10000

The producer sends a single message through a pipe. The consumer reads it back from offset 0.

Replace <topic-name> with your topic name, and replace <partitions> and <replication> with the numbers you want. Replication cannot exceed the number of brokers, so keep it at 1 for a single VM:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --create --topic <topic-name> --partitions 1 --replication-factor 1

Edit /etc/default/kafka and restart:

sudo systemctl restart kafka.service

The defaults on Standard_B2s are:

• KAFKA_HEAP_OPTS: -Xms512m -Xmx1024m
• G1GC tuning enabled
• Log directory: /var/lib/kafka/data/logs
• Metadata directory: /var/lib/kafka/data/metadata

For production, raise -Xmx to 4096m or higher on D4s or D8s instances.

Replace <new-password> with the operator's chosen password:

sudo htpasswd -bB /etc/nginx/akhq.htpasswd alice <new-password>
sudo systemctl reload nginx

Each operator signs in with their own basic-auth credential. The underlying Kafka cluster credentials remain the SecureAnyCloud SCRAM admin defined in /etc/akhq/application.yml.

/opt/kafka/bin/kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --alter --add-config 'SCRAM-SHA-256=[password=<new-password>]' --entity-type users --entity-name <new-username>

Kafka data lives at /var/lib/kafka/data/logs. To take a consistent backup, stop the broker, copy the directory and restart the broker:

sudo systemctl stop kafka.service
sudo tar czf /var/backups/kafka-$(date +%F).tgz -C /var/lib/kafka data
sudo systemctl start kafka.service

Copy /var/backups to Azure Blob Storage (az storage blob upload-batch) periodically for off-VM retention. For zero-downtime backups, add a second broker and use kafka-mirror-maker.sh.

sudo journalctl -u kafka.service --no-pager -n 80
sudo journalctl -u akhq.service --no-pager -n 30
sudo journalctl -u nginx.service --no-pager -n 30
sudo ls /var/log/kafka/

Kafka writes its main log to /var/log/kafka/server.log and its controller log to /var/log/kafka/controller.log. AKHQ's stdout goes to the systemd journal. The nginx access log is at /var/log/nginx/access.log, and the nginx error log is at /var/log/nginx/error.log.