etcd 3

sudo systemctl is-active etcd nginx etcdkeeper
sudo test -f /var/lib/sac/data/etcd-firstboot.done && echo "firstboot: complete" || echo "firstboot: missing"
sudo test -f /var/lib/sac/data/etcd-postboot.done && echo "postboot: complete" || echo "postboot: missing" /usr/local/bin/etcd --version | head -1 /usr/local/bin/etcdctl version

etcd, nginx, and etcdkeeper all report active. The two boot oneshots write a sentinel file when they finish, then exit cleanly. etcd-firstboot.done and etcd-postboot.done confirm that per-VM credential generation, RBAC bootstrap and htpasswd generation completed successfully.

The etcd binary reports etcd Version: 3.6.11 and the CLI reports etcdctl version: 3.6.11 with API version: 3.6.

sudo cat /stage/scripts/etcd-credentials.log

Read two values from the file, which is root:root 0600:

• ETCD_PASSWORD is the SecureAnyCloud user's password. It is used for the etcdctl CLI, the nginx HTTP Basic-Auth prompt and the etcdkeeper login form.
• ETCD_ROOT_PASSWORD is the root user's password. You need it for snapshot save, auth status, user and role management, and the etcdkeeper Authentication dialog when you want full tree visibility.

ETCD_CLIENT_URL is the canonical client endpoint: http://127.0.0.1:2379 from the VM, or http://<vm-ip>:2379 from another VM in the VNet. ETCD_HEALTH_URL is http://127.0.0.1:2379/health.

curl -s http://127.0.0.1:2379/health | jq

Returns {"health":"true","reason":""} once etcd is fully ready to serve traffic. The /health endpoint is unauthenticated by design. Your load balancer and Kubernetes liveness probe call it. Every other endpoint requires credentials.

PASS=$(sudo grep '^ETCD_PASSWORD=' /stage/scripts/etcd-credentials.log | cut -d= -f2-) etcdctl --user=sac:$PASS put /sac/example "hello world" etcdctl --user=sac:$PASS get /sac/example etcdctl --user=sac:$PASS put /test should-fail 2>&1 | tail -1 || true

The first put returns OK. The get prints the key, then the value on the next line: /sac/example then hello world.

Keys in etcd are arbitrary byte strings, so /-prefixed paths are convention only. Values can be up to 1.5 MB each (the --max-request-bytes cap).

The third command would normally succeed. If you drop the --user= flag, etcd returns Error: etcdserver: user name is empty. This shows that RBAC is enforced and that credentials are the only thing standing between a client and the keyspace.

Browse to http://<vm-ip>/ from a workstation inside the allowed CIDR. nginx prompts with an HTTP Basic-Auth dialog. Enter SecureAnyCloud and the value of ETCD_PASSWORD from Step 5.

nginx accepts the credentials, and the etcdkeeper landing page loads with its own Authentication dialog on top. This application-level login proves you have access to the etcd RBAC API. Enter root and the value of ETCD_ROOT_PASSWORD from Step 5, then click Submit. The login flow is layered on purpose. See the Security section below.

After the Authentication dialog accepts your login, etcdkeeper renders the key tree in the left pane. The two keys written in Step 7, /sac/example and /sac/from-ui (added in the next step), appear under the root folder.

Click any key to load its value into the center editor. Click the disk icon at the top right to save edits back through the etcd API. The footer reports the etcd version (3.6.11), the live data-store size and the cluster member name.

Right-click the root folder in the tree pane and pick Create Node. The Create node dialog opens. Then:

• Set Name to the key path (for example /sac/from-ui).
• Leave Dir as FALSE. A directory node has no value, only children.
• Leave TTL blank for a permanent key, or enter seconds for an ephemeral lease-style key.
• Put the value in the Value editor.

Click Submit. The key is written to etcd in a single atomic API call, and the new entry shows up in the tree pane immediately.

In one SSH session, start a watch on the /sac/ key prefix.

PASS=$(sudo grep '^ETCD_PASSWORD=' /stage/scripts/etcd-credentials.log | cut -d= -f2-) etcdctl --user=sac:$PASS watch --prefix /sac/ & WATCH_PID=$! sleep 1 etcdctl --user=sac:$PASS put /sac/key1 v1 sleep 1 kill $WATCH_PID 2>/dev/null || true

The watcher prints the change event in real time: PUT, key and value, all delivered over a single long-lived gRPC stream. Kubernetes uses the same primitive to react to resource changes. Watches survive leader elections and resume from a known revision.

A lease is a TTL attached to one or more keys. When the lease expires, the keys are deleted automatically. This is how etcd implements ephemeral state such as service registration and distributed locks.

PASS=$(sudo grep '^ETCD_PASSWORD=' /stage/scripts/etcd-credentials.log | cut -d= -f2-)
LEASE_ID=$(etcdctl --user=sac:$PASS lease grant 60 | awk '{print $2}') etcdctl --user=sac:$PASS put /sac/ephemeral "expires in 60s" --lease=$LEASE_ID etcdctl --user=sac:$PASS lease list

lease grant 60 returns a 60-second lease. The put --lease= command ties the key to that lease.

To keep the key alive, run etcdctl lease keep-alive <lease-id> in a long-running process. If you stop refreshing the lease, or call etcdctl lease revoke <lease-id>, the key disappears at the next TTL boundary.

Transactions (etcdctl txn) compose conditional reads and writes. They are the building block for leader election and distributed locks.

A snapshot is etcd's first-class backup primitive. It is a consistent point-in-time copy of the entire keyspace.

ROOT_PASS=$(sudo grep '^ETCD_ROOT_PASSWORD=' /stage/scripts/etcd-credentials.log | cut -d= -f2-) sudo mkdir -p /var/backups/etcd
sudo etcdctl --user=root:$ROOT_PASS snapshot save /var/backups/etcd/etcd-snap-$(date +%F).db
sudo etcdctl --user=root:$ROOT_PASS --write-out=table snapshot status /var/backups/etcd/etcd-snap-$(date +%F).db

snapshot save requires the root user (the admin role). snapshot status prints the hash, revision, total keys and total size of the snapshot. Verify that the revision moves forward with each backup.

Periodically copy /var/backups/etcd off the VM to Azure Blob Storage with az storage blob upload-batch for disaster recovery retention.

To restore, stop etcd and run etcdctl snapshot restore <backup-dir>/etcd-snap-DATE.db --data-dir=/var/lib/etcd-restored. Swap /var/lib/etcd with the restored directory, run chown -R etcd:etcd /var/lib/etcd, and start etcd again.

For a multi-node cluster, restore is more involved. Each member restores from the same snapshot with cluster-specific flags. See https://etcd.io/docs/v3.5/op-guide/recovery/.

Add a read-only role for monitoring or for a Kubernetes-style service account:

ROOT_PASS=$(sudo grep '^ETCD_ROOT_PASSWORD=' /stage/scripts/etcd-credentials.log | cut -d= -f2-) etcdctl --user=root:$ROOT_PASS role add ro etcdctl --user=root:$ROOT_PASS role grant-permission ro --prefix=true read / etcdctl --user=root:$ROOT_PASS user list

user list prints SecureAnyCloud and root, the two users provisioned at first boot. To add a new monitoring user, run etcdctl --user=root:$ROOT_PASS user add monitoring --new-user-password=<new-password>, followed by user grant-role monitoring ro.

The new user can read every key (--prefix=true read /) but cannot write. You can scope permissions by exact key, key range (--from-key) or prefix.

This image runs a single-node etcd by design. That suits development, CI, edge deployments and any workload that does not need HA.

For production HA, use a 3 or 5 node Raft cluster. An odd number of nodes tolerates (N-1)/2 failures: 3 nodes tolerate 1 failure, and 5 tolerate 2. To stand up the cluster, deploy three of these images. Then edit /etc/etcd/etcd.conf on each node to set:

• ETCD_INITIAL_CLUSTER to the comma-separated list of name=http://<peer-ip>:2380 triples
• ETCD_LISTEN_PEER_URLS to http://<this-node-private-ip>:2380 (no longer loopback)
• ETCD_INITIAL_ADVERTISE_PEER_URLS to the same routable URL
• ETCD_INITIAL_CLUSTER_STATE to new on first boot, existing thereafter

Restart etcd on all three nodes within a few seconds of each other. Open 2380/tcp between the nodes in your NSG for peer-to-peer Raft traffic, and keep it closed to everyone else.

The etcdkeeper UI on each node shows that node's view of the keyspace. Once Raft converges, all three views are identical. Full reference: https://etcd.io/docs/v3.5/op-guide/clustering/.

ROOT_PASS=$(sudo grep '^ETCD_ROOT_PASSWORD=' /stage/scripts/etcd-credentials.log | cut -d= -f2-) sudo etcdctl --user=root:$ROOT_PASS --write-out=table endpoint status
sudo etcdctl --user=root:$ROOT_PASS alarm list

endpoint status prints the version, DB size, leader and Raft index in a table. It is the first command to run when triaging. alarm list reports any active NOSPACE or CORRUPT alarms. A NOSPACE alarm pins the cluster read-only, and it clears with alarm disarm once you have compacted and defragmented.

Use these commands to read the logs:

• sudo journalctl -u etcd -n 80 --no-pager for the long-running etcd daemon log
• sudo journalctl -u nginx -n 80 --no-pager for the reverse proxy (HTTP 401s for missing Basic-Auth show up here)
• sudo journalctl -u etcdkeeper -n 80 --no-pager for the web UI process
• sudo journalctl -u etcd-firstboot --no-pager and sudo journalctl -u etcd-postboot --no-pager for the one-shot bootstrap units that ran at first boot