Debezium

Prerequisites

• An active Azure subscription, an SSH key pair, and a VNet and subnet in the target region.
• A recommended VM size of Standard_B4ms (4 vCPU / 16 GiB RAM) for the two JVMs.
• An NSG that allows inbound 22/tcp only, from your management network. Reach the Connect REST API through an SSH tunnel (Step 2).
• Do not open 80/tcp. nginx serves the API over plain HTTP, so the Basic auth password would cross the network in clear.
• If off-box clients need Kafka (9092/tcp) or the raw Connect REST port (8083/tcp), allow them only from your own client CIDR. See Step 11.

Step 1 - Deploy from the Azure Marketplace

• Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Debezium by SecureAnyCloud, and select Create.
• On the Basics tab, pick your subscription, resource group, region and size (Standard_B4ms).
• Under Administrator account, choose SSH public key and paste your key.
• Under Inbound port rules, allow SSH (22) and HTTP (80).
• Review the dedicated data disk on the Disks tab, then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

Create the VM from the command line. Replace the example resource group with your own:

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:debezium-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name debezium \
  --image sac1647283583153:debezium-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --public-ip-sku Standard

After the VM is created, do not open any port beyond SSH. To use the Connect REST API from your workstation, forward it over SSH. Then browse or curl to http://127.0.0.1:8080/:

ssh -N -L 8080:127.0.0.1:80 azureuser@<vm-public-ip>

Never open port 80 to the internet. The API behind it uses HTTP Basic auth over plain HTTP, so the password is sent in clear.

Step 3 - Connect to your VM

Connect over SSH as the administrator user:

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Check the three services:

systemctl is-active kafka.service kafka-connect.service nginx.service

All three services report active. Kafka starts in a few seconds. Kafka Connect takes a little longer to load the Debezium plugins and join the cluster.

Step 5 - Retrieve your Connect password

The first boot of your VM generates a unique Connect password and writes it to a root-only file. Read the file:

sudo cat /root/debezium-credentials.txt

The file contains DEBEZIUM_CONNECT_USER (admin), DEBEZIUM_CONNECT_PASSWORD, the Connect REST URL and the Kafka bootstrap address. Store the password somewhere safe.

Step 6 - Check the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes. Query it:

curl -s http://localhost/health

The endpoint returns ok.

Step 7 - Confirm the Kafka and Connect versions

Print the Kafka and Java versions:

/opt/kafka/bin/kafka-topics.sh --version java -version

Kafka reports 3.9.2 and Java reports a 17.x build (Eclipse Temurin).

Step 8 - List the Debezium connector plugins

The Kafka Connect REST API is available through nginx on port 80 behind HTTP Basic auth. List the loaded connector plugins and confirm that the Debezium classes are present:

PW=<DEBEZIUM_CONNECT_PASSWORD> curl -s -u admin:$PW http://localhost/connector-plugins | python3 -m json.tool | grep -E 'class|version' | grep -i debezium

The output lists io.debezium.connector.postgresql.PostgresConnector, io.debezium.connector.mysql.MySqlConnector and io.debezium.connector.mongodb.MongoDbConnector. A request without credentials returns 401. A request with the per-VM password returns 200.

Step 9 - Run a Kafka round trip (create a topic, produce and consume)

Prove that Kafka works end to end with the console tools:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic demo --partitions 1 --replication-factor 1 echo 'hello-cdc' | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic demo /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic demo --from-beginning --max-messages 1 --timeout-ms 15000

The consumer prints hello-cdc. To list your topics at any time, run /opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list.

Step 10 - Register a Debezium connector with the REST API

You register a Debezium connector by POSTing a JSON config to the Connect REST API. The example below registers a PostgreSQL connector. Replace the database host, credentials and slot.name with your source database, which must have logical replication enabled (wal_level=logical). The connector configuration documentation is at debezium.io.

PW=<DEBEZIUM_CONNECT_PASSWORD> curl -s -u admin:$PW -X POST http://localhost/connectors \
  -H 'Content-Type: application/json' \
  -d '{ "name": "inventory-postgres", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "<your-postgres-host>", "database.port": "5432", "database.user": "<replication-user>", "database.password": "<replication-password>", "database.dbname": "<your-db>", "topic.prefix": "inventory", "plugin.name": "pgoutput", "slot.name": "debezium", "schema.history.internal.kafka.bootstrap.servers": "127.0.0.1:9092", "schema.history.internal.kafka.topic": "schema-changes.inventory" } }'

Check that the connector reaches RUNNING:

curl -s -u admin:$PW http://localhost/connectors/inventory-postgres/status | python3 -m json.tool

The connector.state and each task state report RUNNING once Debezium connects to PostgreSQL. Row-level changes then stream into Kafka topics named <topic.prefix>.<schema>.<table>, which any Kafka consumer can read.

To delete a connector, run curl -u admin:$PW -X DELETE http://localhost/connectors/inventory-postgres.

Step 11 - Open Kafka and Connect to off-box clients

By default nothing beyond SSH is needed. Both ports below are unauthenticated and plaintext. Anyone who can reach 9092 can read and write every topic. Anyone who can reach 8083 can create, change or delete connectors, including their database credentials.

Never use az vm open-port for these ports, because it allows any source. If database clients, Kafka consumers or external Connect tooling must reach the VM directly, add NSG rules scoped to your own client CIDR only:

az network nsg rule create --resource-group <your-rg> --nsg-name <your-vm-nsg> \
  --name kafka-9092-clients --priority 910 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes <your-client-cidr> --destination-port-ranges 9092
az network nsg rule create --resource-group <your-rg> --nsg-name <your-vm-nsg> \
  --name connect-8083-clients --priority 920 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes <your-client-cidr> --destination-port-ranges 8083

The Kafka broker advertises the VM's public IP on 9092, which is set automatically at first boot. Allowed clients therefore connect with bootstrap.servers=<vm-public-ip>:9092.

For the Connect REST API, prefer the SSH tunnel from Step 2 (ssh -N -L 8083:127.0.0.1:8083 azureuser@<vm-public-ip>) over opening 8083 at all.

Step 12 - Confirm data lives on the dedicated disk

The Kafka logs and the Connect internal topics are stored on the dedicated Azure data disk, so they survive OS changes and can be resized independently. Check the mount:

findmnt /var/lib/debezium

A separate Azure data disk backs the mount. It is captured into the image and re-provisioned on every VM.

Maintenance

• Configuration: Kafka's KRaft config is /opt/kafka/config/sac-kraft.properties and Connect's is /opt/kafka/config/sac-connect-distributed.properties. JVM heaps are in /etc/default/kafka and /etc/default/kafka-connect. Edit the files, then run sudo systemctl restart kafka kafka-connect to apply the change.
• Connectors: manage connectors through the Connect REST API (GET/POST/DELETE /connectors). Registered connectors persist in the connect-configs Kafka topic on the data disk and survive restarts.
• Backups: snapshot the /var/lib/debezium data disk to preserve Kafka logs, connector configs and offsets.
• Upgrades: replace the Kafka release under /opt/kafka or the Debezium plugins under /opt/debezium-plugins, then restart the services.
• Security patches: unattended-upgrades stays enabled, so the OS continues to receive security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.

Debezium and Apache Kafka are trademarks of the Apache Software Foundation. SecureAnyCloud produces this image, and it is not affiliated with or endorsed by the Debezium project or the Apache Software Foundation. Debezium and Apache Kafka are distributed under the Apache License 2.0.

Deploy on Azure

Launch Debezium on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com