Apache ZooKeeper 3.9

Check the service and the ZooKeeper version:

sudo systemctl is-active zookeeper.service zoonavigator.service nginx.service /opt/zookeeper/bin/zkServer.sh version 2>&1 | tail -1

Read the credentials file:

sudo cat /stage/scripts/zookeeper-credentials.log

Note ZK_CONNECT, ZK_ADMIN_USER and ZK_ADMIN_PASSWORD. The same password is used for the ZooNavigator web UI basic auth.

Run a health check and an authenticated request:

echo "stat" | nc -q 2 127.0.0.1 2181 | head -10
PASS=$(sudo grep '^ZK_ADMIN_PASSWORD=' /stage/scripts/zookeeper-credentials.log | cut -d= -f2-) printf "addauth digest sac:%s\nls /\n" "$PASS" | /opt/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181 2>&1 | tail -5

The first call uses ZooKeeper's "stat" four-letter-word command to confirm Mode: standalone. The second call authenticates as SecureAnyCloud via digest ACL and lists the root znodes.

Browse to http://<vm-ip>/ and authenticate as SecureAnyCloud with the password from Step 5. ZooNavigator auto-connects to the local ZooKeeper ensemble and lands on the znode browser.

Click Create node in the ZooNavigator left pane. Name the node /myapp and set the mode to Persistent. The new znode appears in the tree.

Click any znode, for example the newly created /myapp, to see its data, ACL list, stat (czxid, mzxid, version and so on) and any children. The Edit tab lets operators set data inline.

Create a protected znode with the ZooKeeper CLI:

PASS=$(sudo grep '^ZK_ADMIN_PASSWORD=' /stage/scripts/zookeeper-credentials.log | cut -d= -f2-) printf "addauth digest sac:%s\ncreate /sac/data hello digest:sac:cdrwa\nls /sac\nget /sac/data\n" "$PASS" | /opt/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181 2>&1 | tail -10

This creates /sac/data with an ACL that grants the SecureAnyCloud digest user full CRUD permissions, then reads it back. For public-readable znodes, use the ACL world:anyone:r instead.

Edit /etc/zookeeper/conf/java.env and restart the service:

sudo systemctl restart zookeeper.service

Defaults on Standard_B2s:

• JVMFLAGS: -Xms256m -Xmx512m
• Snapshot retention: 5 snapshots, autopurge every 24 hours
• Data dir: /var/lib/zookeeper/data
• Transaction log dir: /var/lib/zookeeper/log

For production with a busy ensemble, raise -Xmx to 1024m–2048m and place transaction logs on a separate fast disk (dataLogDir in /etc/zookeeper/conf/zoo.cfg).

Replace <new-password> with the operator's chosen password:

sudo htpasswd -bB /etc/nginx/zoonavigator.htpasswd alice <new-password>
sudo systemctl reload nginx

Each operator authenticates with their own basic-auth credential against the same nginx vhost. The underlying ZooKeeper digest ACL is still SecureAnyCloud.

Generate the digest hash and create a znode owned by the second user:

PASS=$(sudo grep '^ZK_ADMIN_PASSWORD=' /stage/scripts/zookeeper-credentials.log | cut -d= -f2-) NEWUSER=alice NEWPASS=<new-password>
HASH=$(echo -n "$NEWUSER:$NEWPASS" | openssl dgst -binary -sha1 | base64) printf "addauth digest sac:%s\nsetAcl / digest:sac:cdrwa,digest:%s:%s:cdrwa\nls /\n" "$PASS" "$NEWUSER" "$HASH" | /opt/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181 2>&1 | tail -5

ZooKeeper writes snapshots to /var/lib/zookeeper/data/version-2 and transaction logs to /var/lib/zookeeper/log/version-2. Take a consistent backup while the service is running by copying the latest snapshot and all transaction logs since:

sudo tar czf /var/backups/zookeeper-$(date +%F).tgz -C /var/lib zookeeper

Periodically copy /var/backups to Azure Blob Storage (az storage blob upload-batch) for off-VM retention.

To restore, stop ZooKeeper, replace /var/lib/zookeeper and restart the service.

Use these commands to inspect the logs:

sudo journalctl -u zookeeper.service --no-pager -n 80
sudo journalctl -u zoonavigator.service --no-pager -n 30
sudo journalctl -u nginx.service --no-pager -n 30
sudo ls /var/lib/zookeeper/log/

ZooKeeper writes its main log to /var/lib/zookeeper/log/zookeeper-zookeeper-server-*.out and its data log to /var/lib/zookeeper/log/version-2/. ZooNavigator's stdout goes to the systemd journal. The nginx access log is at /var/log/nginx/access.log.