Kafka Karapace

Confirm that all four services are active, and check the installed Kafka and Karapace versions.

for s in kafka karapace-registry karapace-rest nginx; do printf '%-26s : ' "$s.service"; systemctl is-active "$s.service" done echo "Kafka : $(ls /opt/kafka/libs/ | grep -oE '^kafka-clients-[0-9.]+' | head -1 | sed 's/kafka-clients-//')" echo "Karapace: $(/opt/karapace/venv/bin/pip show karapace 2>/dev/null | awk '/^Version:/{print $2}')"

The first-boot service generates a unique password for this VM. The same password authenticates the Kafka broker and the Karapace endpoints. Read it from the credentials file:

sudo cat /stage/scripts/kafka-karapace-credentials.log

Note these values: KAFKA_ADMIN_USER and KAFKA_ADMIN_PASSWORD, KARAPACE_USER and KARAPACE_PASSWORD, and the SCHEMA_REGISTRY_URL and REST_PROXY_URL endpoints.

The broker rejects anonymous clients. Build an admin client configuration from the per-VM password, then produce a message to the SecureAnyCloud topic and consume it back.

PASS=$(sudo grep '^KAFKA_ADMIN_PASSWORD=' /stage/scripts/kafka-karapace-credentials.log | cut -d= -f2-)
cat > /tmp/sac.properties <<EOF security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="sac" password="${PASS}"; EOF echo -n 'anonymous client : ' timeout 12 /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server 127.0.0.1:9092 >/dev/null 2>&1 && echo 'ACCEPTED' || echo 'REJECTED (auth required)' MSG="sac-demo-$(date +%s)" echo "${MSG}" | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --topic sac echo "produced: ${MSG}" echo -n 'consumed: ' timeout 20 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/sac.properties --topic sac --from-beginning --timeout-ms 12000 2>/dev/null | grep -F "${MSG}" | head -1 rm -f /tmp/sac.properties

The Schema Registry is exposed on port 8081 behind nginx basic authentication. Register an Avro schema for a subject, then read it back.

The commands below run on the VM and use 127.0.0.1. From another host, replace it with your VM's IP address, shown as <vm-ip> elsewhere in this guide.

PASS=$(sudo grep '^KARAPACE_PASSWORD=' /stage/scripts/kafka-karapace-credentials.log | cut -d= -f2-)
echo -n 'anonymous GET /subjects : HTTP ' curl -s -o /dev/null -w '%{http_code}\n' -m 8 http://127.0.0.1:8081/subjects
echo 'register schema for subject demo-value:' curl -s -u "sac:${PASS}" -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  -d '{"schema":"{\"type\":\"record\",\"name\":\"demo\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}' \
  http://127.0.0.1:8081/subjects/demo-value/versions echo;
echo 'read it back:' curl -s -u "sac:${PASS}" http://127.0.0.1:8081/subjects
echo curl -s -u "sac:${PASS}" http://127.0.0.1:8081/subjects/demo-value/versions/1 echo

The Kafka REST proxy on port 8082 lets applications produce and consume without a Kafka client library. Produce a JSON record to the SecureAnyCloud topic over HTTP.

PASS=$(sudo grep '^KARAPACE_PASSWORD=' /stage/scripts/kafka-karapace-credentials.log | cut -d= -f2-)
echo -n 'anonymous GET /topics : HTTP ' curl -s -o /dev/null -w '%{http_code}\n' -m 8 http://127.0.0.1:8082/topics
echo 'produce a JSON record to topic sac:' curl -s -u "sac:${PASS}" -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  -d '{"records":[{"value":{"event":"order.created","id":42}}]}' \
  http://127.0.0.1:8082/topics/sac echo;
echo 'topics visible to the REST proxy:' curl -s -u "sac:${PASS}" http://127.0.0.1:8082/topics echo