Apache Kafka 4.2

Open Marketplace in the Azure Portal and search for Apache Kafka 4.2. Select the SecureAnyCloud publisher entry, then choose Create to start the wizard.

On the Basics tab, choose your subscription, target resource group and region. The region must match the region your Azure Compute Gallery exposes the image in. Set the virtual machine name. Choose SSH public key as the authentication type, set a username of your choice, and paste your SSH public key.

On the Disks tab, the recommended OS disk type is Premium SSD. Leave the OS disk size at the default. You can attach a separate Premium SSD data disk now if you plan to move the Kafka log directory to it later. You can also do that after the broker is running, by following Step 12.

On the Networking tab, select your existing virtual network and subnet. Attach a network security group that allows inbound TCP 22 from your management IP range and inbound TCP 9093 only from the virtual network CIDR. Do not expose 9093 to the public internet. The broker uses SASL_PLAINTEXT, which authenticates clients but does not encrypt traffic. Any exposure beyond a private network requires the TLS upgrade described in Step 13.

On the Management, Monitoring and Advanced tabs, the defaults are appropriate. Choose Review + create and wait for validation to pass. Then choose Create. Deployment takes around three minutes.

If you prefer the command line, accept the Marketplace terms once. Then deploy from the Marketplace image URN:

RG="kafka-prod" LOCATION="eastus" VM_NAME="kafka-broker-1" ADMIN_USER="kafkaops" IMAGE_URN="sac1647283583153:apache-kafka-4-2-ubuntu-22-04:default:latest" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create \
  --resource-group "$RG" \
  --name kafka-vnet \
  --address-prefix 10.20.0.0/16 \
  --subnet-name kafka-subnet \
  --subnet-prefix 10.20.1.0/24 az network nsg create --resource-group "$RG" --name kafka-nsg az network nsg rule create \
  --resource-group "$RG" --nsg-name kafka-nsg \
  --name allow-ssh-mgmt --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" \
  --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create \
  --resource-group "$RG" --nsg-name kafka-nsg \
  --name allow-kafka-vnet --priority 110 \
  --source-address-prefixes 10.20.0.0/16 \
  --destination-port-ranges 9093 --access Allow --protocol Tcp
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:apache-kafka-4-2-ubuntu-22-04:default:latest
az vm create \
  --resource-group "$RG" \
  --name "$VM_NAME" \
  --image "$IMAGE_URN" \
  --size Standard_D4s_v3 \
  --admin-username "$ADMIN_USER" \
  --ssh-key-values "$SSH_KEY" \
  --vnet-name kafka-vnet --subnet kafka-subnet \
  --nsg kafka-nsg \
  --public-ip-address "" \
  --os-disk-size-gb 64

The --public-ip-address "" flag keeps the broker off the public internet. Use a bastion host or your existing private connectivity to reach it.

After deployment, find the private IP of the new virtual machine. Then connect from a host inside the same virtual network:

ssh kafkaops@<private-ip>

The first login may take a few seconds while cloud-init finishes. When you get a shell, systemd has already started the broker, and the first-boot one-shot service has already generated the per-VM credentials.

The kafka-firstboot.service systemd one-shot writes the admin SASL credentials the very first time the virtual machine boots. They live in a single file. Read it with:

sudo cat /stage/scripts/kafka-credentials.log

The output looks similar to this:

username=admin password=4f8aB2dQ9pK1xVnE7sT3uYrL6cZw0jHm bootstrap=10.20.1.4:9093 mechanism=SCRAM-SHA-512 security.protocol=SASL_PLAINTEXT

These credentials are unique to this virtual machine. Store them somewhere safe immediately, and rotate them next, as described in Step 10. The first-boot one-shot does not run again on later reboots, so this file is your only readable copy of the generated password until you rotate it.

The deployed image contains the following components:

Component Version Purpose

Apache Kafka 4.2.0 (Scala 2.13) Single-node KRaft broker, controller and broker in one JVM

OpenJDK 17 (headless) JVM runtime

Ubuntu 22.04 LTS Base operating system

systemd units kafka.service, kafka-firstboot.service Process supervision

The broker process is kafka.Kafka running under the dedicated kafka system user, with no login shell, started by /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties.

Path Owner Purpose

/opt/kafka-4.2.0/ root:root Unpacked Apache Kafka tarball

/opt/kafka symlink Stable upgrade path that points at the active version

/etc/kafka/server.properties kafka:kafka 0640 Broker and KRaft controller configuration

/etc/kafka/kafka_server_jaas.conf kafka:kafka 0640 JAAS file consumed by the broker JVM, populated on first boot

/etc/default/kafka root:kafka 0640 Environment variables (heap size, JVM flags, log directory)

/etc/systemd/system/kafka.service root:root 0644 Main broker unit

/etc/systemd/system/kafka-firstboot.service root:root 0644 First-boot one-shot unit

/usr/local/sbin/kafka-firstboot.sh root:root 0750 First-boot logic (credential generation, KRaft format)

/var/lib/kafka/data/ kafka:kafka 0750 Kafka log directories

/var/log/kafka/ kafka:kafka 0750 Server and GC logs

/stage/scripts/kafka-credentials.log kafka:kafka 0600 Generated admin credentials, readable by root only

systemd starts the broker at boot. Manage it with these commands:

# Status
sudo systemctl status kafka.service
# Stop
sudo systemctl stop kafka.service
# Start
sudo systemctl start kafka.service
# Restart
sudo systemctl restart kafka.service
# Tail live logs
sudo journalctl -u kafka.service -f

The systemd unit appends to the on-disk log file /var/log/kafka/kafka.log, which contains the same content as journalctl. To check the first-boot one-shot, run:

sudo systemctl status kafka-firstboot.service
sudo journalctl -u kafka-firstboot.service

The oneshot is expected to remain in the active (exited) state after a successful first boot. The file /var/lib/kafka/.firstboot-done gates it, so it does not run again.

Build a client properties file from your generated credentials. The command reads the password from /stage/scripts/kafka-credentials.log:

PRIVATE_IP="$(hostname -I | awk '{print $1}')" ADMIN_PW="$(sudo awk -F= '/^password=/ {print $2}' /stage/scripts/kafka-credentials.log)" cat > /tmp/client.properties <<EOF security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="${ADMIN_PW}"; EOF chmod 600 /tmp/client.properties

List the topics:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --list

The same client.properties file works as --producer.config and --consumer.config for the console producer and consumer. It is equivalent to the consumer.properties and producer.properties overrides you would set in a Java or Python client.

Here is a minimal Java producer config block:

Properties props = new Properties(); props.put("bootstrap.servers", "10.20.1.4:9093"); props.put("security.protocol", "SASL_PLAINTEXT"); props.put("sasl.mechanism", "SCRAM-SHA-512"); props.put("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required " + "username=\"admin\" password=\"<your-password>\";"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Here is the equivalent block for the confluent-kafka Python client:

from confluent_kafka import Producer producer = Producer({ "bootstrap.servers": "10.20.1.4:9093", "security.protocol": "SASL_PLAINTEXT", "sasl.mechanism": "SCRAM-SHA-512", "sasl.username": "admin", "sasl.password": "<your-password>", })

Create a topic, produce a message to it and consume the message back:

# Create a topic with 6 partitions /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --create --topic orders --partitions 6 --replication-factor 1
# Produce a message echo "hello sac" | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --producer.config /tmp/client.properties \
  --topic orders
# Consume it back /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --consumer.config /tmp/client.properties \
  --topic orders --from-beginning --max-messages 1 --timeout-ms 10000

The consumer echoes hello SecureAnyCloud back. The replication factor is fixed at 1 because this is a single-broker image.

Rotate the admin password generated on first boot before you put any real workload through the broker. Generate a new password, alter the SCRAM credential, update the JAAS file and the credentials file, and restart the broker:

NEW_PW="$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)" /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --alter \
  --add-config "SCRAM-SHA-512=[password=${NEW_PW}]" \
  --entity-type users --entity-name admin
# Update the broker JAAS file with the new password
sudo tee /etc/kafka/kafka_server_jaas.conf >/dev/null <<EOF KafkaServer { org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="${NEW_PW}"; }; EOF
sudo chown kafka:kafka /etc/kafka/kafka_server_jaas.conf
sudo chmod 0640 /etc/kafka/kafka_server_jaas.conf
# Update the operator credentials file
sudo sed -i "s|^password=.*|password=${NEW_PW}|" /stage/scripts/kafka-credentials.log
sudo systemctl restart kafka.service

After the restart, regenerate /tmp/client.properties with the new password and confirm that kafka-topics.sh --list still works.

The shipped image has only the admin user, with no ACLs configured. Kafka ACLs are off by default in server.properties. Add an application user with a generated password:

APP_PW="$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)" /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --alter \
  --add-config "SCRAM-SHA-512=[password=${APP_PW}]" \
  --entity-type users --entity-name app1 echo "app1 password: ${APP_PW}"

To turn ACLs on, add the following two lines to /etc/kafka/server.properties, then restart the broker:

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer super.users=User:admin

Then grant app1 produce and consume rights on the orders topic:

/opt/kafka/bin/kafka-acls.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --add --allow-principal User:app1 \
  --producer --consumer --group '*' --topic orders

By default, the broker writes its log directories to /var/lib/kafka/data on the OS disk. For any non-trivial workload, attach a Premium SSD data disk and move the log directory onto it:

# Stop the broker
sudo systemctl stop kafka.service
# (In Azure) attach a new Premium SSD via the portal or `az vm disk attach`,
# then on the VM identify the new device. It will typically be /dev/sdc. lsblk
# Format and mount
sudo mkfs.xfs /dev/sdc
sudo mkdir -p /data echo "/dev/sdc /data xfs defaults,nofail 0 2" | sudo tee -a /etc/fstab
sudo mount /data
sudo mkdir -p /data/kafka
sudo chown kafka:kafka /data/kafka
sudo chmod 0750 /data/kafka
# Move existing data (cheap on a freshly deployed broker)
sudo rsync -aHAX /var/lib/kafka/data/ /data/kafka/
sudo mv /var/lib/kafka/data /var/lib/kafka/data.old
# Point Kafka at the new directory
sudo sed -i 's|^log.dirs=.*|log.dirs=/data/kafka|' /etc/kafka/server.properties
sudo systemctl start kafka.service
sudo systemctl status kafka.service

Confirm that the broker comes back up cleanly and your topics still list. Then you can run sudo rm -rf /var/lib/kafka/data.old to reclaim the OS disk space.

The shipped image is SASL_PLAINTEXT only. The wire is authenticated but not encrypted. Do not expose the broker beyond a private network until you have completed this step.

Adding TLS to a Kafka broker is a multistep task. It depends on whether you want a self-signed certificate, a private CA-issued certificate or a public CA-issued certificate. The high-level steps are:

• Generate or obtain a server certificate and private key for the broker, plus a truststore for clients.
• Add a SASL_SSL listener to /etc/kafka/server.properties on a new port (typically 9094), and update advertised.listeners accordingly.
• Configure ssl.keystore.location, ssl.keystore.password and ssl.key.password on the broker. If you use a private CA, also configure ssl.truststore.location and ssl.truststore.password.
• Update the network security group to allow the new port, and remove the SASL_PLAINTEXT port from the allow list.
• Distribute the truststore (or root CA certificate) to all your clients, and change their security.protocol from SASL_PLAINTEXT to SASL_SSL.
• Restart the broker and validate the new listener with a test client before you cut traffic over.

The Apache Kafka security documentation at https://kafka.apache.org/documentation/#security_overview is the authoritative reference. SecureAnyCloud does not ship a one-click TLS enabler in this image, on purpose. Your security team should own the certificate lifecycle, not the marketplace publisher.

The broker will not start. Check sudo journalctl -u kafka.service -n 200 --no-pager. The most common cause is a syntax error in /etc/kafka/server.properties after a manual edit.

The first-boot one-shot never ran. Check sudo systemctl status kafka-firstboot.service and ls -la /var/lib/kafka/.firstboot-done. If the sentinel is present but the credentials file is empty or holds no generated password, something interrupted the one-shot. Remove the sentinel with sudo rm /var/lib/kafka/.firstboot-done, then run sudo systemctl start kafka-firstboot.service to retry.

Client authentication fails with "Authentication failed during authentication due to invalid credentials". The most common cause is a sasl.mechanism mismatch. The broker uses SCRAM-SHA-512, and your client must use exactly the same string. A client configured with SCRAM-SHA-256 or PLAIN cannot authenticate against this broker.

Client connects locally but a remote client gets "Connection refused". Check ss -tlnp | grep 9093 on the broker. The listener should be bound to 0.0.0.0. The image ships with listeners=SASL_PLAINTEXT://0.0.0.0:9093, so 0.0.0.0 is correct. Also check the network security group rule that controls inbound 9093 traffic.

Producer fails with "Bootstrap broker returned wrong host". This means advertised.listeners is wrong. The first-boot one-shot rewrites advertised.listeners to the value of hostname -I | awk '{print $1}'. If your VM has multiple network interfaces, it may have picked the wrong one. Edit /etc/kafka/server.properties and fix the advertised.listeners line. Then run sudo systemctl restart kafka.service.

Out of disk on /var/lib/kafka/data. Either move the log directory to a data disk as described in Step 12, or reduce log.retention.hours in /etc/kafka/server.properties and restart the broker.

• Rotate the admin password immediately after first boot (Step 10).
• Restrict the network security group on port 9093 to the smallest possible source CIDR.
• Add TLS before you expose the broker beyond a private network (Step 13).
• Use the application user pattern in Step 11 instead of reusing the admin user for application clients.
• Enable ACLs and set super.users=User:admin, so that application users have only the rights they need.
• Take regular snapshots of /var/lib/kafka/data (or your attached data disk) using Azure Disk Snapshots.
• Subscribe to the Apache Kafka security mailing list at https://kafka.apache.org/contact and apply security patches as they are published.

Apache Kafka is licensed under the Apache License 2.0. The full text is in /opt/kafka/LICENSE on the deployed image. SecureAnyCloud distributes the unmodified upstream Apache Kafka 4.2.0 binary tarball published by the Apache Software Foundation. SecureAnyCloud-authored configuration, systemd units and first-boot tooling are layered on top.

For support with the SecureAnyCloud image itself, contact sales@secureanycloud.com or visit https://secureanycloud.com/support. For Apache Kafka questions outside the scope of SecureAnyCloud packaging, the Apache Kafka community resources at https://kafka.apache.org/contact are the authoritative reference.

Deploy on Azure

Launch Apache Kafka 4.2 on Ubuntu 22.04 on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com