Apache Kafka

Sign in to the AWS Management Console, open the EC2 service and choose Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Apache Kafka. Choose Select on the SecureAnyCloud listing, then choose Continue on the subscription summary.

Choose an instance type of m5.large or larger, because the Kafka broker and the bundled web console are both JVM processes. Choose your EC2 key pair under Key pair (login). Under Network settings, choose your VPC and subnet, then create or choose a security group. The group must allow inbound port 22 from your management network, inbound port 80 from the networks that will reach the web console, and inbound port 9092 from your Kafka client network. Keep the root volume at the default size or larger.

Choose Launch instance. First boot initialization takes about one minute after the instance state becomes Running and the status checks pass.

The following command launches an instance from the SecureAnyCloud Apache Kafka Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22, 80 and 9092 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":30,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-kafka-01}]'

On success, the command prints a JSON document. Note the instance ID, then get the public address once the instance is running with aws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].PublicIpAddress" --output text.

Connect over SSH with the key pair you chose and the public IP address from step 2. The SSH login user depends on the operating system of the AMI variant you launched:

AMI variant SSH login user

Apache Kafka on Ubuntu 24.04 ubuntu

The first boot service runs before the SSH daemon is ready, so the credentials file is in place when you first log in. Display it:

ssh <login-user>@<public-ip>
sudo cat /root/kafka-credentials.txt

The plain text file contains the broker bootstrap address, the administrator username (SecureAnyCloud), the administrator password, and the security protocol and SASL mechanism the broker expects. Copy these values to a secure place such as a password manager or encrypted vault. Do not commit them to source control. The same password authenticates Kafka clients on port 9092 and the web management console on port 80.

The image runs three services: kafka.service (the broker and controller), akhq.service (the web management console) and nginx.service (the reverse proxy in front of the console). Confirm that all three are active and check the installed Kafka version:

sudo systemctl is-active kafka.service akhq.service nginx.service ls /opt/kafka/libs/ | grep -E '^kafka-clients-' | head -1

You should see active three times, followed by the Kafka client library version:

active active active kafka-clients-4.3.0.jar

Every Kafka command-line tool needs a small properties file that carries the SASL credentials. Build one from the values in the credentials file. This block reads the password directly from /root/kafka-credentials.txt, so you never paste it:

PASS=$(sudo grep '^kafka.admin.pass=' /root/kafka-credentials.txt | cut -d= -f2-)
cat > /tmp/sac.properties <<EOF security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="sac" password="${PASS}"; EOF

The broker enforces authentication, so any tool invoked without --command-config /tmp/sac.properties is rejected. This confirms that authentication is working.

With the client configuration in place, query the broker API versions and list the topics. The broker API call confirms that the SASL handshake succeeds end to end:

sudo /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties | head -3 sudo /opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --list

The first command prints the broker id and its supported API range. The second lists the topics, including the sample SecureAnyCloud topic created on first boot:

127.0.0.1:9092 (id: 1 rack: null isFenced: false) -> ( Produce(0): 0 to 13 [usable: 13], Fetch(1): 4 to 18 [usable: 18], __consumer_offsets sac

Open a web browser and go to http://<public-ip>/. When the browser prompts for HTTP basic authentication, sign in as the SecureAnyCloud user with the password from /root/kafka-credentials.txt. The console opens on the Topics view. It lists every topic with its message count, size on disk, partition count, replication factor and any consumer groups attached to it.

Choose the magnifying glass on a topic row to open the topic detail. There you can browse individual messages, view per-partition offsets and edit topic configuration.

The Nodes view shows every broker in the cluster. A single-instance deployment has one node: broker id 1, host 127.0.0.1:9092, marked as the controller and holding all of the cluster's partitions.

When you scale to a multi-broker cluster, each broker appears here with its own share of the partitions. The controller column shows which broker currently holds the controller role.

The Consumer Groups view lists every consumer group, its state, the broker coordinating it, the number of live members and the topics it consumes. The lag badge next to each topic shows how far behind the group is. A lag of zero means the group has consumed every available message.

A group whose consumers have all disconnected shows the EMPTY state but keeps its committed offsets. When a consumer rejoins, consumption resumes exactly where it stopped.

Confirm the full streaming path from the command line. The first command pipes a single message into the SecureAnyCloud topic. The second reads it back from the start of the topic:

echo "hello sac $(date +%s)" | sudo /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --topic sac
sudo /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --topic sac --from-beginning --max-messages 1 --timeout-ms 10000

The consumer prints the earliest message in the topic, then exits. Both commands succeed only because the client configuration carries valid SASL credentials.

Create your own topics with kafka-topics.sh. Replace <topic-name> with your topic name, and <partitions> and <replication> with your chosen numbers. On a single broker, the replication factor cannot exceed one. Add brokers first if you need replicas.

sudo /opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --create --topic <topic-name> --partitions <partitions> --replication-factor <replication>

Refresh the Topics view in the web console. The new topic appears immediately.

External producers and consumers connect to the broker on port 9092 with the same SASL SCRAM SHA 256 credentials. Make sure your security group allows inbound port 9092 from the client network, then point the client at <public-ip>:9092 with these properties:

bootstrap.servers=<public-ip>:9092 security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="sac" password="<KAFKA_ADMIN_PASSWORD>";

For traffic that crosses an untrusted network, terminate TLS at an upstream Application Load Balancer, or switch the broker listener to SASL_SSL and supply a keystore. The Kafka documentation describes the SASL_SSL configuration in detail.

The KAFKA_HEAP_OPTS variable in /etc/default/kafka sets the broker heap. The image ships with a conservative heap suitable for development and light production. For a busier broker, raise the heap to roughly half of the instance memory, then restart the service:

sudo systemctl restart kafka.service

While the broker restarts, follow its logs with sudo journalctl -u kafka.service -f to confirm that it comes back cleanly.

Kafka data lives under /var/lib/kafka/data on the dedicated EBS data volume. The simplest consistent backup is an EBS snapshot of that volume, which you can schedule with Amazon Data Lifecycle Manager. For a file-level copy, stop the broker first so the log segments are quiescent:

sudo systemctl stop kafka.service
sudo tar czf /var/tmp/kafka-data-backup.tgz -C /var/lib/kafka data
sudo systemctl start kafka.service

For zero-downtime backups, scale to a multi-broker cluster and rely on replication, or mirror topics to a second cluster.

This image is a single broker and controller. It suits development, testing, single-tenant production streaming, edge and IoT brokering, and proof-of-concept work.

To scale horizontally, launch additional brokers and give each a unique node.id. List every controller in controller.quorum.voters, and raise the replication factor on your topics so that partitions are spread across brokers. The Apache Kafka documentation describes multi-node KRaft cluster configuration step by step.