Karapace

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet with a subnet in the target region.

Standard_B2s (2 vCPU / 4 GiB RAM) is the size the image is tuned for, and it is sufficient for the bundled broker plus both Karapace services. Choose a larger size for heavier REST proxy workloads or larger schema volumes.

Configure the network security group (NSG) to allow inbound 22/tcp from your management network, 80/tcp for the schema registry API, and 8082/tcp if you intend to use the REST proxy. Karapace serves plain HTTP, so for production terminate TLS in front of it with your own domain. The bundled Kafka broker listens on 127.0.0.1 only, so ports 9092 and 9093 stay off the network.

Step 1 - Deploy the image

Option A: Azure portal

• In the Azure portal, choose Create a resource and search for Karapace on Ubuntu 24.04 LTS by SecureAnyCloud.
• Select the plan and click Create.
• On the Basics tab, choose your subscription, resource group and region, name the VM, and select Standard_B2s.
• Set Authentication type to SSH public key, enter azureuser as the username, and provide your public key.
• On the Networking tab, allow inbound 22/tcp (from your IP) and 80/tcp. Add 8082/tcp if you need the REST proxy.
• Select Review + create, then Create. When deployment completes, note the VM's public IP address.

Option B: Azure CLI

az vm image terms accept --urn sac1647283583153:karapace-ubuntu-24-04:default:latest
az vm create \
  --resource-group my-karapace-rg \
  --name karapace \
  --image sac1647283583153:karapace-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
az vm open-port --resource-group my-karapace-rg --name karapace --port 80 --priority 900
az vm open-port --resource-group my-karapace-rg --name karapace --port 8082 --priority 910

Step 2 - Confirm the services are running

Connect over SSH as azureuser and confirm that the four systemd units are active:

• kafka.service is the bundled broker.
• karapace-registry.service is the schema registry.
• karapace-rest.service is the Kafka REST proxy.
• nginx.service is the authenticated reverse proxy.

systemctl is-active kafka.service karapace-registry.service karapace-rest.service nginx.service

Every service reports active.

Check the listener layout, which is the core of the security model. Only nginx is on a public interface, and everything behind it is loopback-only.

ss -lnt | grep -E ':(80|8082|8081|8083|9092|9093)\b'

nginx listens on 0.0.0.0:80 and 0.0.0.0:8082. The Karapace registry (127.0.0.1:8081), the Karapace REST proxy (127.0.0.1:8083) and the Kafka broker (127.0.0.1:9092 and 127.0.0.1:9093) are all bound to loopback. The broker's listener may appear in the IPv4-mapped form [::ffff:127.0.0.1]:9092, which is the same loopback address.

An unauthenticated health endpoint is available for load balancer probes.

curl -s -o /dev/null -w 'GET /healthz -> HTTP %{http_code}\n' http://127.0.0.1/healthz

This returns HTTP 200.

The registry also exposes its own readiness detail. This endpoint answers 200 with "healthy":true from the moment the process starts. schema_registry_ready stays false until the registry has finished replaying the _schemas topic and elected itself primary. Writes succeed only when both inner flags are true, which normally takes a few seconds after boot.

curl -s http://127.0.0.1:8081/_health | python3 -m json.tool | head -12

Step 3 - Retrieve the per-VM password

Both APIs require HTTP Basic authentication. The username is karapace. First boot generated the password on this VM and stored it in a root-only file, and no other VM shares it.

sudo cat /root/karapace-info.txt

The file records the registry URL, the REST proxy URL, the username and the generated password. It is mode 600 and owned by root. Verify this with:

stat -c '%n mode=%a owner=%U:%G' /root/karapace-info.txt

Confirm the secure-by-default posture. The registry refuses a request with no credentials or a wrong password, and answers a request with the per-VM password.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -o /dev/null -w 'no credentials -> HTTP %{http_code}\n' http://127.0.0.1/subjects
curl -s -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' -u karapace:wrong-password http://127.0.0.1/subjects
curl -s -o /dev/null -w 'per VM password -> HTTP %{http_code}\n' -u "karapace:$KPASS" http://127.0.0.1/subjects

The first two return HTTP 401 and the third returns HTTP 200.

From your workstation, use the VM's public IP address in place of 127.0.0.1, for example curl -u karapace:<karapace-password> http://<vm-public-ip>/subjects.

Step 4 - Register your first schema

A subject is the namespace a schema lives under. The common convention is <topic>-value for the message value and <topic>-key for the key. Register an Avro schema for a payments topic.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}' \
  http://127.0.0.1/subjects/payments-value/versions

The registry replies with the global identifier it assigned to the schema, for example {"id":1}. Registering the identical schema again is idempotent and returns the same id.

Step 5 - Read schemas back

List every subject the registry knows about.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/subjects

Read the latest version of a subject. The response contains the subject, its version, the schema id and the schema itself.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/subjects/payments-value/versions/latest

List the versions of a subject, then fetch a schema directly by its global id.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/subjects/payments-value/versions
curl -s -u "karapace:$KPASS" http://127.0.0.1/schemas/ids/1

Step 6 - Check compatibility before you break a consumer

This is the reason a schema registry exists. Each subject is governed by a compatibility rule, and the registry evaluates a proposed schema against the existing version before you deploy it. The image default is BACKWARD, meaning a consumer using the new schema must still be able to read data written with the old one.

Check the current rule.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/config

Add a field with a default. This is backward compatible, because a consumer reading old data simply uses the default.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"GBP\"}]}"}' \
  http://127.0.0.1/compatibility/subjects/payments-value/versions/latest

This returns {"is_compatible":true}.

Now add a field without a default. This is not backward compatible, because a consumer using the new schema cannot populate that field when it reads data written before the field existed. The registry catches it.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"reference\",\"type\":\"string\"}]}"}' \
  http://127.0.0.1/compatibility/subjects/payments-value/versions/latest

This returns {"is_compatible":false} along with the offending field name, reference. This is the check to run in CI before a producer change is merged.

Know which direction each rule protects, because the intuition is easy to get backwards. Under BACKWARD, removing a field is allowed, since a consumer on the new schema ignores the extra data it finds in old records. Adding a mandatory field is what breaks, as shown above. Under FORWARD the reverse holds. Use FULL when you need both to be safe.

To change the rule for a single subject or globally, PUT a new compatibility level. Valid values are BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE and NONE.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X PUT \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"compatibility": "FULL"}' \
  http://127.0.0.1/config/payments-value

Step 7 - Use the Kafka REST proxy

The REST proxy on port 8082 exposes the bundled broker over plain HTTP, so a client with no native Kafka driver can still work with topics. List the topics.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1:8082/topics

Inspect a single topic's partitions.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1:8082/topics/_schemas

To produce a message, the topic must already exist. The REST proxy does not create it, and producing to an unknown topic returns {"error_code":40401,"message":"Topic payments not found"}. Create the topic first with the bundled Kafka tools.

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 \
  --create --if-not-exists --topic payments --partitions 1 --replication-factor 1

Produce a JSON message to the topic.

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  --data '{"records":[{"value":{"id":"pay-001","amount":42.5}}]}' \
  http://127.0.0.1:8082/topics/payments

The response reports the partition and offset of each record.

{"key_schema_id":null,"offsets":[{"offset":0,"partition":0}],"value_schema_id":null}

Step 8 - Work with the bundled broker from the command line

The standard Apache Kafka tools are installed at /opt/kafka/bin and point at the loopback broker. List the topics to confirm that the registry's data is stored in a Kafka topic.

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list

_schemas is the compacted topic where Karapace stores every schema. Describe it.

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic _schemas

Create your own topic.

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 \
  --create --if-not-exists --topic orders --partitions 3 --replication-factor 1

Because the broker is single node, the replication factor must be 1.

Step 9 - Point Karapace at an external Kafka cluster

The bundled broker makes the image self-contained, but Karapace works equally well with a cluster you already run. Karapace is configured entirely through KARAPACE_-prefixed environment variables in its systemd units.

Add a systemd drop-in for each service instead of editing the shipped unit files, so that your change survives future package updates. Create /etc/systemd/system/karapace-registry.service.d/external-kafka.conf with your own bootstrap address.

[Service] Environment=KARAPACE_BOOTSTRAP_URI=kafka-1.example.internal:9092,kafka-2.example.internal:9092 Environment=KARAPACE_REPLICATION_FACTOR=3

Set KARAPACE_REPLICATION_FACTOR to 3 on a real cluster. Karapace uses this factor when it creates the _schemas topic, and a single replica would put your entire schema history on one broker. Create the matching /etc/systemd/system/karapace-rest.service.d/external-kafka.conf with the same KARAPACE_BOOTSTRAP_URI line.

If your cluster requires TLS or SASL, set KARAPACE_SECURITY_PROTOCOL (for example SSL or SASL_SSL) in the same drop-in. Add the matching KARAPACE_SSL_CAFILE, KARAPACE_SSL_CERTFILE and KARAPACE_SSL_KEYFILE variables, or KARAPACE_SASL_MECHANISM, KARAPACE_SASL_PLAIN_USERNAME and KARAPACE_SASL_PLAIN_PASSWORD.

Apply the change, then confirm that the registry reports itself ready against the new cluster.

sudo systemctl daemon-reload
sudo systemctl restart karapace-registry.service karapace-rest.service curl -s http://127.0.0.1:8081/_health | python3 -m json.tool | head -12

When you use an external cluster, the bundled broker is redundant. Switch it off to free its memory.

sudo systemctl disable --now kafka.service

Step 10 - Security model

• No shipped credential. The nginx password file is locked in the image, so no known or blank password authenticates. On each VM's first boot, karapace-firstboot.service generates a unique password, writes it as a bcrypt hash into /etc/nginx/karapace.htpasswd, and records it in /root/karapace-info.txt with mode 600. Because the locked file is the shipped state, the appliance fails closed: if first boot has not completed, every request is refused rather than served anonymously.
• Authentication cannot be bypassed. Karapace serves both APIs with no authentication of its own here, so both processes are bound to 127.0.0.1. The only way in is through nginx, which requires HTTP Basic authentication on every path except /healthz.
• The broker is never exposed. The bundled Kafka broker listens on 127.0.0.1:9092 and 127.0.0.1:9093 only. Its sole client is Karapace on the same host, which is what makes an unauthenticated PLAINTEXT broker safe here. Do not change its listeners to a public address without adding Kafka-level authentication.
• Each VM has its own cluster. The image ships with the KRaft storage wiped, so every VM formats a fresh, empty cluster on first boot. No two deployments share a cluster identity, and no schemas from the build ever reach a customer instance.
• No build toolchain. Karapace is built from source at image build time, which needs a Go and C toolchain. Both are removed from the image afterwards, so no compiler ships to customers.
• Rotate the password at any time. htpasswd -B prompts for the new password twice and writes a bcrypt hash. Reload nginx to pick it up, then record the new password somewhere safe, because /root/karapace-info.txt still shows the original.

sudo htpasswd -B /etc/nginx/karapace.htpasswd karapace
sudo systemctl reload nginx

• Put TLS in front. Karapace serves plain HTTP. For anything beyond a trusted network, terminate TLS at nginx with your own certificate, or place the VM behind an Azure Application Gateway or Load Balancer that does.

Step 11 - Base image and versions

/opt/karapace/venv/bin/python -c 'from karapace.version import __version__; print("Karapace", __version__)'

ls -d /opt/kafka_* && java -version 2>&1 | head -1

The base is fully patched at build time, and unattended security upgrades stay enabled for the life of the instance. Confirm that no packages are held back.

apt-mark showhold | wc -l

This prints 0.

Managing the services

systemctl status kafka.service karapace-registry.service karapace-rest.service nginx.service --no-pager

Read the registry's log.

journalctl -u karapace-registry.service -n 50 --no-pager

The services start automatically on boot. karapace-firstboot.service runs once, before the others. It is skipped on later boots because it leaves a sentinel file at /var/lib/sac/data/karapace-firstboot.done.

Support

SecureAnyCloud publishes this image and supports it 24/7. SecureAnyCloud is not affiliated with or endorsed by Aiven or the Karapace project. Karapace and Apache Kafka are used here under the Apache License 2.0, and their names identify only the software included in this image.

For help with this image, contact sales@secureanycloud.com.

Deploy on Azure

Launch Karapace on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com