etcd

Launch and connect to your instance

The default SSH login user for Ubuntu 24.04 is ubuntu. Connect over SSH on port 22:

ssh -i <your-ssh-key.pem> ubuntu@<instance-public-ip>

Retrieving the per-instance credentials

On the first boot of every instance, a one-shot service generates a fresh per-instance SecureAnyCloud user password and a separate emergency root password. It writes both to a root-only file and enables etcd RBAC. Retrieve them with:

sudo cat /root/etcd-credentials.txt

The file looks like:

# etcd 3.5 — Per-Instance Credentials ETCD_VERSION=etcd Version: 3.5.30 ETCD_CLIENT_URL=http://<instance-public-ip>:2379 ETCD_HEALTH_URL=http://<instance-public-ip>:2379/health ETCD_USER=sac ETCD_PASSWORD=<ETCD_PASSWORD> ETCD_ROOT_PASSWORD=<ETCD_ROOT_PASSWORD>

The SecureAnyCloud user holds readwrite on the entire / key prefix and is the canonical user for application traffic. ETCD_ROOT_PASSWORD is for emergency access only. Do not use it for normal operations.

Confirm the service is healthy

sudo systemctl status etcd.service --no-pager --lines=5

endpoint health and endpoint status need an authenticated etcdctl because RBAC is enforced. Read the password from /root/etcd-credentials.txt for each call:

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  endpoint health 2>/dev/null

endpoint status is a cluster-maintenance call that the SecureAnyCloud user is not granted. It needs the emergency root credential:

ROOT=$(sudo grep ^ETCD_ROOT_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=root:${ROOT} \
  endpoint status --write-out=table

The /health endpoint on port 2379 is anonymous by design. Kubernetes liveness and readiness probes call it:

curl -s http://127.0.0.1:2379/health

Authenticated put and get

Every key-value operation needs --user=sac:<password>. Load the password from the credentials file in each call:

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  put /apps/web/config '{"port":8080,"workers":4}'

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  get /apps/web/config --print-value-only

List every key under a prefix:

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  get /apps --prefix --keys-only

Delete a key:

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  del /apps/web/config

Watching a key

etcdctl watch streams every change to a key or prefix. Open a second SSH session and put into /apps/... to see the updates stream live in the first:

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  watch /apps --prefix

Verify RBAC is enforced

An unauthenticated put against the local endpoint is rejected, which shows that RBAC is active. The command below is expected to fail with Error: etcdserver: user name is empty:

ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  put /noauth-attempt should-fail

Listing users is a cluster-management call that requires the root credential. Use the emergency root password from the credentials file:

ROOT=$(sudo grep ^ETCD_ROOT_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=root:${ROOT} \
  user list

Inspect the sac-rw role. The SecureAnyCloud user has permission to read the roles assigned to it:

PASS=$(sudo grep ^ETCD_PASSWORD= /root/etcd-credentials.txt | cut -d= -f2-) && \
  ETCDCTL_API=3 /usr/local/bin/etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  --user=sac:${PASS} \
  role get sac-rw

Connecting from a remote client

The instance listens on 0.0.0.0:2379. At first boot, the advertise URL is published with the instance's public IPv4 address, resolved from EC2 IMDSv2. From any client whose security group opens port 2379, run:

ETCDCTL_API=3 etcdctl \
  --endpoints=http://<instance-public-ip>:2379 \
  --user=sac:<ETCD_PASSWORD> \
  put mykey myvalue

For production traffic on a public client port, we strongly recommend terminating TLS in front of etcd. See the TLS section below.

Security model

• Peer port 2380 is loopback only. A single-node deployment never needs peer traffic on the network, and exposing 2380 to the network is a CVE-class mistake. If you add cluster peers, change this deliberately. See the multi-node section.
• Client port 2379 listens on 0.0.0.0. RBAC enforces security at the etcd layer, and the AWS Security Group enforces it at the network layer. Keep your Security Group locked to the CIDR ranges that need to talk to etcd. Do not open 2379 to 0.0.0.0/0 in production.
• RBAC is enabled at first boot. A SecureAnyCloud user with readwrite on the / prefix and a separate emergency root user are created with per-instance passwords. The cluster refuses every unauthenticated key-value operation. The /health endpoint stays anonymous so Kubernetes probes work.

Expanding to a multi-node cluster

A single-node etcd suits development, proof-of-concept and small production workloads where the durability of the underlying EC2 instance is acceptable. For a quorum-replicated cluster, etcd recommends 3 or 5 nodes.

Cluster expansion is a runtime operation. Launch additional instances of this AMI, then run etcdctl member add from the existing node to introduce each peer to the cluster.

Before you start each new peer, rewrite its ETCD_LISTEN_PEER_URLS from 127.0.0.1 to the peer's network address, reuse the existing sac-etcd-token, and set ETCD_INITIAL_CLUSTER_STATE=existing. The detailed multi-node guide is in the upstream etcd documentation at https://etcd.io/docs/, and the SecureAnyCloud recipe is fully compatible with it.

Enabling TLS

The image ships with http:// on the client port. To enable TLS:

• Provision a server certificate and private key, for example from your own internal CA or from a public CA such as Let's Encrypt with a DNS challenge.
• Place them at /etc/etcd/server.crt and /etc/etcd/server.key (mode 0640, group etcd).
• Edit /etc/etcd/etcd.conf. Add the ETCD_CERT_FILE and ETCD_KEY_FILE lines, and change ETCD_LISTEN_CLIENT_URLS and ETCD_ADVERTISE_CLIENT_URLS to use https://.
• Restart the service with sudo systemctl restart etcd.service.

Your clients then connect to https://<your-public-host>:2379 and pass --cacert=<your-CA-bundle.pem> to etcdctl.

Updating

The image ships with etcd 3.5.30 from the official etcd-io GitHub release. Later versions in the etcd 3.5 line are drop-in compatible. To upgrade, download the new tarball from https://github.com/etcd-io/etcd/releases and verify the SHA-256. Then stop etcd.service, replace /usr/local/bin/etcd and /usr/local/bin/etcdctl, and restart.

The systemd unit, the first-boot scripts and the credentials file are all preserved across binary upgrades.

Screenshots

Support

24/7 technical support is included with this AMI. Email sales@secureanycloud.com or use the chat widget on https://secureanycloud.com/ for help with deployment, RBAC design, cluster expansion, performance tuning, or anything else related to etcd.

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com