Coroot Community Edition 1.29.2

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Coroot Community Edition 1.29.2 on Ubuntu 24.04 as the software version, choose your Region, and select Launch through EC2.

Choose m5.large (or larger as you add hosts), your key pair and a security group as described above. Metrics, traces, logs and profiles are stored on the root volume, so give it room. 50 GiB is a good start for a handful of hosts.

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=coroot}]'

First boot takes about half a minute after the instance is running. The first data shows in Coroot a minute or two later.

Connect over SSH as the ubuntu user. Software version and SSH user:

Coroot Community Edition 1.29.2 on Ubuntu 24.04: ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

The login message shows where the credentials are and how to run the self-test.

First boot sets the passwords, the API key and the certificate. Then ClickHouse, Prometheus, Coroot, the node agent and nginx start. Check the units and the first-boot marker:

systemctl is-active clickhouse-server prometheus coroot coroot-node-agent nginx coroot-firstboot
sudo test -f /var/lib/sac/data/coroot-firstboot.done && echo FIRSTBOOT_DONE

All six units are active and the first boot marker is present:

active active active active active active FIRSTBOOT_DONE

Only SSH and nginx listen on network ports. Coroot, Prometheus, ClickHouse and the node agent are on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:10300 users:(("coroot-node-age")) 127.0.0.1:8080 users:(("coroot")) 127.0.0.1:8123 users:(("clickhouse-serv")) 127.0.0.1:9000 users:(("clickhouse-serv")) 127.0.0.1:9090 users:(("prometheus")) [::1]:8123 users:(("clickhouse-serv")) [::1]:9000 users:(("clickhouse-serv")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

(The number of nginx entries follows the number of vCPUs.)

Read the credentials file as root:

sudo cat /var/lib/sac/data/coroot-credentials.txt

The file is readable by root only. It holds the site URL, the logins, the passwords and the API key:

# Coroot Community Edition 1.29.2 - generated on first boot by coroot-firstboot.service
# These credentials are unique to this instance; none was baked into the image. Keep this file secret. COROOT_URL=https://<public-dns-name>/ COROOT_USER=admin COROOT_PASSWORD=<COROOT_PASSWORD> COROOT_API_KEY=<COROOT_API_KEY> CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=<CLICKHOUSE_PASSWORD> PROMETHEUS_USER=coroot PROMETHEUS_PASSWORD=<PROMETHEUS_PASSWORD>
#
# Open COROOT_URL in a browser (self-signed certificate minted on this instance's first boot; port 80 redirects to # https) and sign in as COROOT_USER / COROOT_PASSWORD (change it under your user menu). This host's node agent already
# reports with COROOT_API_KEY; point node agents on other hosts at https://<public-dns-name>/ with the same key.
# ClickHouse (127.0.0.1:8123/9000) and Prometheus (127.0.0.1:9090) listen on loopback only; their passwords are here
# and in /etc/coroot/coroot.yaml.
# Self-test:
sudo coroot-selftest Change the served name:
sudo coroot-set-hostname <your-domain>
# User guide: https://secureanycloud.com/stacks/coroot-community-edition-1-29-2/

• COROOT_PASSWORD is your sign-in password for the web UI.
• COROOT_API_KEY is the key that node agents and OpenTelemetry SDKs send with their data. It is also the key for Coroot's Prometheus-compatible API.
• You do not need the ClickHouse and Prometheus passwords for normal use. Coroot uses them to reach its own databases.

Coroot is served under one name. At first boot, that name is the instance's public DNS name, or its public or private IP address when it has no DNS name. The certificate also names the IP addresses, and a request to an IP address is redirected to the name.

If the name changes, for example when you stop and start the instance without an Elastic IP, or if you want your own DNS name, set it with coroot-set-hostname (Step 11).

The health check answers without a login, and the instance's own address redirects to the served name:

curl -sk https://localhost/healthz
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost/

ok 301

Open COROOT_URL in your browser. The certificate is self-signed and generated on your instance, so accept the browser warning once, or install your own certificate (Step 11). Sign in with admin in the Email field and COROOT_PASSWORD.

Coroot opens the Applications list of the default project. The node agent on the instance has already found the services running on it. The Service Map shows which services talk to which, with the traffic between them.

Nodes lists every host that reports to Coroot, starting with this one. It shows each host's CPU, memory, network, availability zone and instance type.

Change the admin password from the user menu (bottom left, Admin > Change password). Add a user for each person under Settings > Organization.

To watch Coroot pick up an application with no instrumentation, start a small web server and a client that calls it once a second. Both run as throwaway systemd services. Then give the agent a minute and a half to report:

sudo systemd-run --quiet --unit=demo-web -p DynamicUser=yes python3 -m http.server 8000 --bind 127.0.0.1 --directory /usr/share/doc/sac-coroot
sudo systemd-run --quiet --unit=demo-client -p DynamicUser=yes /bin/sh -c 'while :; do curl -s -o /dev/null http://127.0.0.1:8000/; curl -s -o /dev/null http://127.0.0.1:8000/missing; sleep 1; done' sleep 90

demo-web and demo-client now show in Applications and on the Service Map. Open demo-client to see the requests it makes, their latency and the share that fail. The /missing page answers 404.

The same data is available from Coroot's Prometheus-compatible API, which takes the API key in the X-API-Key header. The key goes to curl in a file that only you can read, so it never appears on a command line:

( umask 077; printf 'header = "X-API-Key: %s"\n' "$(sudo sed -n 's/^COROOT_API_KEY=//p' /var/lib/sac/data/coroot-credentials.txt)" > ~/.coroot-api-key.curl ) HOST="$(sudo sed -n 's#^COROOT_URL=https://\([^/]*\)/$#\1#p' /var/lib/sac/data/coroot-credentials.txt)" curl -sk -K ~/.coroot-api-key.curl --resolve "${HOST}:443:127.0.0.1" -G "https://${HOST}/api/v1/query_range" \
  --data-urlencode 'query=sum by (status) (rate(container_http_requests_total{destination="127.0.0.1:8000"}[1m]))' \
  --data-urlencode "start=$(( $(date +%s) - 120 ))" --data-urlencode "end=$(date +%s)" --data-urlencode step=60 \
  | grep -o '"status":"[0-9]xx"'

"status":"2xx" "status":"4xx"

Without the key, nginx refuses the request:

HOST="$(sudo sed -n 's#^COROOT_URL=https://\([^/]*\)/$#\1#p' /var/lib/sac/data/coroot-credentials.txt)" curl -sk -o /dev/null -w '%{http_code}\n' --resolve "${HOST}:443:127.0.0.1" "https://${HOST}/api/v1/series"

401

When you are done, stop the demo. Coroot keeps its history until it ages out:

sudo systemctl stop demo-web demo-client rm -f ~/.coroot-api-key.curl

Every host that you want to see in Coroot runs the node agent and sends its data to this instance over HTTPS with the API key. On each host, complete the following three tasks.

1. Allow access. Allow the host to reach this instance on TCP 443 in the security group.

2. Trust this instance's certificate. The certificate is its own certificate authority. Copy /etc/ssl/coroot/coroot.crt from this instance to the host and add it to the host's trust store (on Ubuntu or Debian):

sudo install -m 0644 coroot.crt /usr/local/share/ca-certificates/coroot.crt
sudo update-ca-certificates

3. Install the agent. Use Coroot's installer, pointing it at COROOT_URL with COROOT_API_KEY. Pin the version this instance runs:

curl -sfL https://raw.githubusercontent.com/coroot/coroot-node-agent/main/install.sh \
  | COLLECTOR_ENDPOINT=https://<public-dns-name> API_KEY=<COROOT_API_KEY> sh -s -- --version 1.38.1

The host appears under Nodes a minute later, and its services join the application list and the service map. The Add nodes button on the Nodes page shows the same commands, filled in with this instance's address and key.

For Kubernetes, Docker and other installs, see the Coroot documentation and use the same endpoint and key. If you install a certificate from a public certificate authority (Step 11), task 2 is not needed.

Applications instrumented with OpenTelemetry can send traces and logs to Coroot over OTLP/HTTP on the same address. For example, set:

OTEL_EXPORTER_OTLP_ENDPOINT=https://<public-dns-name> OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_HEADERS=x-api-key=<COROOT_API_KEY>

The SDK posts to /v1/traces and /v1/logs. nginx refuses any request to these paths without a well-formed key, and Coroot refuses a wrong one.

The node agent loads eBPF programs into the kernel and reads other processes' network and file information, so it runs as root. Its systemd unit limits it to the capabilities below and nothing else. It has no ambient capabilities, no new privileges (NoNewPrivileges=yes) and a read-only system (ProtectSystem=strict) except its own write-ahead log directory. It cannot load kernel modules or reach the instance metadata service.

When the image was built, the agent was run in this sandbox with each capability removed in turn. The named function stopped working every time.

Capability, why the agent needs it, and what happens without it:

CAP_SYS_ADMIN Load the eBPF programs and maps (on this kernel it also covers CAP_BPF and CAP_PERFMON), and enter the host's namespaces the agent cannot start: failed to get uname: operation not permitted

CAP_SYS_PTRACE Read /proc/<pid>/ns, /proc/<pid>/fd and the memory maps of other users' processes to tie connections and files to services the agent cannot start: failed to get uname: open /proc/1/ns/uts: permission denied

CAP_DAC_READ_SEARCH Open the log files that services owned by other users write in their own private directories those services' logs are not collected

CAP_NET_ADMIN Read the kernel's connection tracking table over netlink, to see the real destination behind NAT and load balancer addresses netlink receive: operation not permitted

CAP_NET_RAW Open an ICMP socket to measure the network round trip time to the hosts your services talk to failed to open IPConn: listen ip4:icmp 0.0.0.0: socket: operation not permitted

Check the running agent's capability sets:

grep -E '^Cap(Eff|Bnd|Amb)|^NoNewPrivs' /proc/$(systemctl show -p MainPID --value coroot-node-agent)/status capsh --decode=0000000000283004

CapEff: 0000000000283004 CapBnd: 0000000000283004 CapAmb: 0000000000000000 NoNewPrivs: 1 0x0000000000283004=cap_dac_read_search,cap_net_admin,cap_net_raw,cap_sys_ptrace,cap_sys_admin

Coroot and Prometheus run as their own users with no capabilities at all. ClickHouse runs as clickhouse with only CAP_IPC_LOCK and CAP_SYS_NICE:

systemctl show -p User -p CapabilityBoundingSet -p NoNewPrivileges coroot prometheus clickhouse-server

CapabilityBoundingSet= User=coroot NoNewPrivileges=yes CapabilityBoundingSet= User=prometheus NoNewPrivileges=yes CapabilityBoundingSet=cap_ipc_lock cap_sys_nice User=clickhouse NoNewPrivileges=yes

The agent normally asks the instance metadata service which region, zone and instance type it runs in. Here that request is blocked. First boot writes those values to /etc/coroot-node-agent/agent.env instead, so the node still shows its availability zone and instance type in Coroot.

Data, where it is kept and how long it is kept:

Metrics Prometheus, /var/lib/prometheus 7 days (--storage.tsdb.retention.time=7d in /etc/systemd/system/prometheus.service)

Traces, logs and profiles ClickHouse, /var/lib/clickhouse 7 days (Coroot's default; set ttl under traces:, logs: and profiles: in /etc/coroot/coroot.yaml and restart coroot to change it)

Projects, users, settings and incidents Coroot, /var/lib/coroot until you delete them

To keep metrics longer, raise the retention and restart Prometheus. Grow the root volume to match:

sudo sed -i 's/--storage.tsdb.retention.time=7d/--storage.tsdb.retention.time=30d/' /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload && sudo systemctl restart prometheus

Point a DNS record at the instance. An Elastic IP keeps the address stable. Then set the name that Coroot is served as.

The command below creates a new self-signed certificate for the name. It updates nginx and the cross-site and Origin guards, updates COROOT_URL in the credentials file, and keeps the name across reboots:

sudo coroot-set-hostname <your-domain>

If the instance's address changes and you have no DNS name of your own, run the same command with the new public DNS name. Node agents on other hosts need the new certificate (Step 7) and the new endpoint.

To use a certificate from a public or internal certificate authority instead, for example one issued by Let's Encrypt for <your-domain>, copy it over the generated pair and reload nginx. Running coroot-set-hostname again replaces it with a self-signed one:

sudo install -m 0644 <your-domain>.crt /etc/ssl/coroot/coroot.crt
sudo install -m 0600 <your-domain>.key /etc/ssl/coroot/coroot.key
sudo nginx -t && sudo systemctl reload nginx

The shipped self-test verifies the following:

• The HTTPS redirect and the health check work.
• Every Coroot API call needs a session, and a wrong password is refused while the admin password works.
• The ingestion API refuses a missing or all-zero key and accepts this instance's key.
• The ClickHouse connection editor is closed.
• Node metrics reach Prometheus and traces reach ClickHouse, and both refuse a request without their password.
• Cross-site requests and unknown host names are refused.
• Coroot, Prometheus, ClickHouse and the agent listen on the loopback address only.
• The instance metadata service cannot be reached from inside any of the five services, while it can from your shell.

The self-test hands passwords to curl in root-only temporary files, never on a command line. Run it:

sudo coroot-selftest

SELFTEST-OK https 301 + healthz + UI shell with pinned CSP + /api/user 401 anonymous, 404 wrong password, set_admin_password 401, admin login 200 (Secure HttpOnly cookie) + project <project-id> status ok (prometheus, node agent) + /v1 API key 401 none/zero, 200 instance key + clickhouse-config/connect refused + node metrics in Prometheus, Coroot tables in ClickHouse (both refuse without password) + 403 cross-origin POST + 403 cross-site /stats + 444 unknown host + loopback-only coroot/prometheus/clickhouse/agent + metadata 000 inside 5 services (control 200)

The metadata block is a systemd setting on each service. The first-boot service is the one exception: it reads the instance's address, region and zone once, before the others start. Confirm the setting:

systemctl show -p IPAddressDeny clickhouse-server prometheus coroot coroot-node-agent nginx

IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32 IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32 IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32 IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32 IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32

A request for a host name that the instance does not serve gets no answer at all. nginx closes the connection, so curl exits with code 52:

curl -sk --http1.1 -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000 curl exit 52

The instance generated the certificate for its own names and addresses. Inspect it:

sudo openssl x509 -in /etc/ssl/coroot/coroot.crt -noout -subject -ext subjectAltName

subject=CN = <public-dns-name>, O = sac X509v3 Subject Alternative Name: DNS:<public-dns-name>, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost

Print the installed versions:

head -1 /usr/share/doc/sac-coroot/BUILD-INFO.txt /usr/local/bin/coroot-node-agent --version /usr/local/bin/prometheus --version 2>&1 | head -1 clickhouse-server --version nginx -v 2>&1

Coroot CE 1.29.2 (tag v1.29.2, commit b5e20cc1d1fe8d1ef8bfeecc5c8cc53348ce859d), Apache-2.0, edition Community Version: 1.38.1 prometheus, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d) ClickHouse server version 26.8.34.13 (official build). nginx version: nginx/1.24.0 (Ubuntu)

Path and purpose:

/usr/local/bin/coroot Coroot (server and web UI), run by coroot.service as the coroot user

/etc/coroot/coroot.yaml Coroot's settings: loopback listener, the Prometheus and ClickHouse connections, the project and its API key

/usr/local/bin/coroot-node-agent The node agent, run by coroot-node-agent.service

/etc/coroot-node-agent/agent.env The agent's endpoint, API key, region, zone and instance type (root only)

/usr/local/bin/prometheus Prometheus, run by prometheus.service as the prometheus user; settings in /etc/prometheus/

/etc/clickhouse-server/config.d/sac.xml ClickHouse settings: loopback listeners

/etc/nginx/sites-available/sac-coroot nginx site: HTTPS, the API key check, the login rate limit, proxy to 127.0.0.1:8080

/etc/ssl/coroot/ This instance's certificate and private key

/var/lib/sac/data/coroot-credentials.txt URL, logins, passwords and API key (root only)

/usr/share/doc/sac-coroot/ Build information, checksums, the licenses, the Go module lists and the agent's eBPF C sources

• Admin password: sign in and use the user menu (Admin > Change password). Then update or delete COROOT_PASSWORD in /var/lib/sac/data/coroot-credentials.txt (sudo nano /var/lib/sac/data/coroot-credentials.txt).
• API key: Coroot reads the project's keys from /etc/coroot/coroot.yaml (apiKeys under projects). Add a new key there and restart Coroot (sudo systemctl restart coroot). Move every agent and SDK to the new key. This host's agent reads it from /etc/coroot-node-agent/agent.env, so restart coroot-node-agent after editing. Then remove the old key and restart Coroot again.

• Keep port 443 open only to the people and the hosts that send data, and port 22 open only to your admin network. Never open 8080, 9090, 8123, 9000 or 10300.
• Use your own DNS name and a certificate from a certificate authority that your agents trust (Step 11).
• Keep the credentials file's values in a secrets store, give each person their own Coroot user, and change the API key when a host that held it is retired (Step 14). Logs and traces can contain personal or sensitive data, so treat the instance like the systems it watches.
• The instance applies Ubuntu security updates automatically. To upgrade Coroot, the agent and Prometheus, read the release notes, back up /var/lib/coroot and replace their binaries with newer upstream releases.