Zipkin 3.6.1

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Zipkin 3.6.1 on Ubuntu 24.04 as the software version and select your Region, then choose Launch through EC2.

Choose m5.large (8 GiB of memory; Zipkin's heap is capped at 1 GiB, see Step 8), your key pair, a subnet and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=zipkin}]'

Version SSH user

Zipkin 3.6.1 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot sets the host name and generates the certificate and the two logins. Then Zipkin and nginx start:

systemctl is-active zipkin-firstboot zipkin nginx
sudo test -f /var/lib/sac/data/zipkin-firstboot.done && echo FIRSTBOOT_DONE

All three units are active and the first boot marker is present:

active active active FIRSTBOOT_DONE

Only SSH and nginx listen on the network. Zipkin (the java process) listens on localhost only:

sudo ss -lntpH | awk '{print $4, $6}' | sed -E 's/users:\(\("([^"]+)".*/\1/' | grep -v '127.0.0.5' | sort -u

0.0.0.0:22 sshd 0.0.0.0:443 nginx 0.0.0.0:80 nginx [::]:22 sshd [::]:443 nginx [::]:80 nginx [::ffff:127.0.0.1]:19411 java

sudo cat /var/lib/sac/data/zipkin-credentials.txt

Only root can read the file. It holds the web interface URL with the admin login, and the span ingest URL with the spans login:

# Zipkin 3.6.1 - generated on first boot by zipkin-firstboot.service
# These logins are unique to this instance; none was baked into the image. Keep this file secret. ZIPKIN_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/zipkin/ ZIPKIN_USERNAME=admin ZIPKIN_PASSWORD=<ZIPKIN_PASSWORD> ZIPKIN_INGEST_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/ingest/api/v2/spans ZIPKIN_INGEST_USERNAME=spans ZIPKIN_INGEST_PASSWORD=<ZIPKIN_INGEST_PASSWORD>
#
# Open ZIPKIN_URL in a browser (self-signed certificate minted on this instance's first boot; port 80 redirects to # https) and sign in with ZIPKIN_USERNAME / ZIPKIN_PASSWORD: the UI and the query API (/api/v2/...). # Applications send spans (Zipkin JSON v2, POST) to ZIPKIN_INGEST_URL with HTTP basic auth ZIPKIN_INGEST_USERNAME / # ZIPKIN_INGEST_PASSWORD. Inside the VPC use the private address instead: https://<private-ip>/ingest/api/v2/spans # (the certificate names it). The ingest login cannot read traces; the admin login cannot post spans through /ingest/. # Traces are kept IN MEMORY (MEM_MAX_SPANS in /etc/zipkin/zipkin.env) and are lost when Zipkin restarts. # Self-test:
sudo zipkin-selftest New password:
sudo zipkin-rotate-password admin|spans
# Change the served name:
sudo zipkin-set-hostname <your-domain>
# User guide: https://secureanycloud.com/stacks/zipkin-3-6-1/

The URLs use the instance's public DNS name from EC2 instance metadata. Opening https://<public-ip>/ redirects you to that name.

The image includes a self-test. Through nginx, it checks that both logins are required and kept apart. It sends a span with the ingest login and reads it back with the admin login. It also checks the request guards, the service's privileges and the metadata block:

sudo zipkin-selftest

SELFTEST-OK https 301 + healthz + query API 401/429 without a login, with a wrong password and with the ingest login + UI + health UP (in-memory) + ingest 401/429 without a login, with the admin login and a wrong password + 403 browser ingest + span ingested (202) and queried back + 403 span POST without same-origin proof + 403 cross-origin POST + 444 unknown host + no CORS + zipkin uid zipkin, no capabilities, NoNewPrivs, listening 127.0.0.1:19411 only + metadata 000 inside zipkin and nginx (control 200)

The self-test's span appears in Zipkin under the service name sac-selftest.

Open the ZIPKIN_URL in your browser. The certificate is self-signed, so your browser asks you to confirm it the first time. You can also install your own certificate (Step 10). The browser then asks for a user name and password. Enter ZIPKIN_USERNAME and ZIPKIN_PASSWORD.

Find a trace lists recent traces. Choose a service name, and optionally a span name, tags, a minimum duration and a time range, then select Run query. Each row shows the root span, the number of spans, the services involved and the total duration:

Open a trace to see its spans on a timeline, with each span's service, duration, tags and remote endpoint:

Applications send spans to the ZIPKIN_INGEST_URL (/ingest/api/v2/spans, Zipkin JSON v2 or protobuf; /ingest/api/v1/spans takes the older v1 formats) with an HTTP POST and HTTP basic authentication, using the user spans and the ZIPKIN_INGEST_PASSWORD. Zipkin answers 202 when it accepts the spans.

This example sends a small trace from the instance itself, as a client in your VPC would. The password goes to curl on its standard input, never on the command line:

F=/var/lib/sac/data/zipkin-credentials.txt
URL=$(sudo grep '^ZIPKIN_INGEST_URL=' "$F" | cut -d= -f2-)
PASS=$(sudo grep '^ZIPKIN_INGEST_PASSWORD=' "$F" | cut -d= -f2-)
H=$(echo "$URL" | sed -E 's#^https://([^/]+)/.*#\1#')
T=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n');
NOW=$(( $(date +%s) * 1000000 ))
SPANS=$(printf '[{"traceId":"%s","id":"%s","name":"get /checkout","kind":"SERVER","timestamp":%s,"duration":42000,"localEndpoint":{"serviceName":"frontend"},"tags":{"http.method":"GET","http.path":"/checkout"}},{"traceId":"%s","parentId":"%s","id":"%s","name":"post /orders","kind":"CLIENT","timestamp":%s,"duration":30000,"localEndpoint":{"serviceName":"frontend"},"remoteEndpoint":{"serviceName":"orders"}},{"traceId":"%s","parentId":"%s","id":"%s","name":"post /orders","kind":"SERVER","timestamp":%s,"duration":28000,"localEndpoint":{"serviceName":"orders"}},{"traceId":"%s","parentId":"%s","id":"%s","name":"insert order","kind":"CLIENT","timestamp":%s,"duration":9000,"localEndpoint":{"serviceName":"orders"},"remoteEndpoint":{"serviceName":"postgres"},"tags":{"db.system":"postgresql"}}]' \
  "$T" "${T:0:16}" "$NOW" "$T" "${T:0:16}" "${T:16:16}" "$((NOW+5000))" "$T" "${T:0:16}" "${T:16:16}" "$((NOW+6000))" "$T" "${T:16:16}" "${T:8:16}" "$((NOW+12000))") printf 'user = "spans:%s"\n' "$PASS" | curl -sk -K - -o /dev/null -w 'span ingest: %{http_code}\n' \
  --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' --data "$SPANS" "$URL" echo "trace id: $T"

span ingest: 202 trace id: 981c22833bfde6db342bb9635cd88f26

Trust the instance's certificate. Clients check the certificate, so give them the instance's certificate (it is public, and only the key is secret), or install your own certificate (Step 10). Copy it to your machine with:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/ssl/zipkin/zipkin.crt zipkin.crt

The certificate names the public DNS name, the public IP address and the private IP address, so clients in the same VPC can send to https://<private-ip>/ingest/api/v2/spans and keep span traffic off the internet. Allow port 443 in the security group from those clients.

OpenTelemetry Collector. This is the usual way to connect OpenTelemetry-instrumented services. They export OTLP to a Collector, and the Collector's Zipkin exporter (in the otelcol-contrib distribution) sends the spans to this instance with the ingest login. The basicauth extension holds the credentials:

extensions: basicauth/zipkin: client_auth: username: spans password: <ZIPKIN_INGEST_PASSWORD> exporters: zipkin: endpoint: https://<private-ip>/ingest/api/v2/spans tls: ca_file: /etc/otelcol-contrib/zipkin.crt auth: authenticator: basicauth/zipkin service: extensions: [basicauth/zipkin] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [zipkin]

Zipkin reporters and other tools. Any reporter that can set an HTTP header can send directly. Add the header Authorization: Basic <base64 of spans:ZIPKIN_INGEST_PASSWORD> and point it at the ingest URL. A reporter that cannot set headers (or cannot be given the certificate) sends through a Collector as above, or through an OpenTelemetry Collector on the same host as the application.

Things to know:

• Use /ingest/, not /api/v2/spans. The /api/v2/spans route on the web interface's address needs the admin login and proof that the request comes from this instance's own web pages. This stops another website from posting spans with your browser's saved login. Span reporters send no such proof, so they get 403 there. Applications always use /ingest/.
• The ingest path takes POST only, refuses any request carrying browser headers (Origin, Referer or Sec-Fetch-*) with 403, and accepts up to 20 MB per request.
• A wrong or missing ingest password gets 401, and after repeated failures from one address it gets 429 until the minute has passed. Check the password in your application if you see either.

Dependencies draws the services that called each other in the chosen time range. Choose a time range and select Run query:

Scripts and dashboards can read traces through Zipkin's query API (/api/v2/...) with the admin login. This example lists the services and fetches the latest frontend trace:

F=/var/lib/sac/data/zipkin-credentials.txt
H=$(sudo sed -n 's#^ZIPKIN_URL=https://\([^/]*\)/.*#\1#p' "$F")
APASS=$(sudo grep '^ZIPKIN_PASSWORD=' "$F" | cut -d= -f2-) sleep 2 printf 'user = "admin:%s"\n' "$APASS" | curl -sk -K - --resolve "$H:443:127.0.0.1" "https://$H/api/v2/services"; echo printf 'user = "admin:%s"\n' "$APASS" | curl -sk -K - --resolve "$H:443:127.0.0.1" "https://$H/api/v2/traces?serviceName=frontend&limit=1" | python3 -c 'import json,sys; t=json.load(sys.stdin)[0]; print(len(t), "spans:", ", ".join(sorted({s["localEndpoint"]["serviceName"]+" "+s["name"] for s in t})))'

["sac-selftest","frontend","orders"] 4 spans: frontend get /checkout, frontend post /orders, orders insert order, orders post /orders

The ingest login gets 401 on the query API, so an application's credentials cannot be used to read traces.

Zipkin keeps traces in memory. That suits development, testing and short-term troubleshooting. It needs no database and is fast, but traces are lost when Zipkin or the instance restarts, and the oldest traces are dropped once the limit is reached. The settings are in /etc/zipkin/zipkin.env:

grep -E '^(STORAGE_TYPE|MEM_MAX_SPANS|ZIPKIN_JAVA_OPTS)=' /etc/zipkin/zipkin.env

STORAGE_TYPE=mem MEM_MAX_SPANS=200000 ZIPKIN_JAVA_OPTS=-Xms256m -Xmx1g -XX:+ExitOnOutOfMemoryError -XX:-UsePerfData -Dcom.linecorp.armeria.useJdkDnsResolver=true

• MEM_MAX_SPANS is how many spans Zipkin keeps (200,000 by default).
• -Xmx1g caps Zipkin's memory at 1 GiB. If Zipkin ever runs out of it, it exits and systemd starts it again, with the stored traces gone.

To keep more traces, raise both values together. For example, use MEM_MAX_SPANS=500000 with -Xmx2g on an instance with enough memory, then restart Zipkin:

sudo nano /etc/zipkin/zipkin.env
sudo systemctl restart zipkin

To give either login a new random password, run the helper. It updates nginx and the credentials file and prints nothing secret. Read the new password from the file (Step 4):

sudo zipkin-rotate-password admin
sudo zipkin-rotate-password spans

After rotating spans, update the password in every application or Collector that sends spans. Passwords are stored in nginx only as SHA-512 crypt hashes (/etc/nginx/zipkin-admin.htpasswd and /etc/nginx/zipkin-ingest.htpasswd).

• In your DNS, point <your-domain> (an A record) at the instance's public IP address. An Elastic IP keeps that address stable.
• Tell the instance to use the domain. This sets the name nginx accepts, the request guards and the URLs in the credentials file, and generates a new self-signed certificate for it:

sudo zipkin-set-hostname <your-domain>

• For a certificate your browser and clients trust, put the certificate chain for <your-domain> in /etc/ssl/zipkin/zipkin.crt and its private key in /etc/ssl/zipkin/zipkin.key (root-only, mode 600). Then run sudo systemctl reload nginx. Alternatively, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate that forwards to port 443 on the instance.

Without a login, the web interface, the query API and span ingest are refused. A span post from a browser page is refused. A request for any other host name gets no answer, because nginx closes the connection. Zipkin's own ports are closed to the network:

H=$(sudo sed -n 's#^ZIPKIN_URL=https://\([^/]*\)/.*#\1#p' /var/lib/sac/data/zipkin-credentials.txt)
curl -sk -o /dev/null -w 'UI without a login: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/zipkin/"
curl -sk -o /dev/null -w 'query API without a login: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/api/v2/services"
curl -sk -o /dev/null -w 'span POST without a login: %{http_code}\n' --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' --data '[]' "https://$H/ingest/api/v2/spans"
curl -sk -o /dev/null -w 'span POST from a browser page: %{http_code}\n' --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' -H 'Origin: https://evil.example' --data '[]' "https://$H/ingest/api/v2/spans"
curl -sk -o /dev/null -w 'unknown host name: %{http_code}\n' --resolve evil.example:443:127.0.0.1 https://evil.example/ || true for p in 9411 19411; do if timeout 3 bash -c "exec 3<>/dev/tcp/$(hostname -I | awk '{print $1}')/$p" 2>/dev/null; then echo "port $p: open"; else echo "port $p: closed"; fi; done

UI without a login: 401 query API without a login: 401 span POST without a login: 401 span POST from a browser page: 403 unknown host name: 000 port 9411: closed port 19411: closed

(If you have just made several failed logins from the instance, the first three show 429 instead of 401 for up to a minute.)

nginx and Zipkin are blocked from the instance metadata service:

for u in nginx zipkin; do printf '%s ' "$u"; systemctl show -p IPAddressDeny --value "$u" | tr ' ' '\n' | sort | tr '\n' ' '; echo; done

nginx 168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 zipkin 168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128

Zipkin runs as the zipkin user with no Linux capabilities and cannot gain any:

P=$(systemctl show -p MainPID --value zipkin) ps -o user= -p "$P" grep -E '^(CapEff|CapBnd|CapAmb|NoNewPrivs)' /proc/"$P"/status

zipkin CapEff: 0000000000000000 CapBnd: 0000000000000000 CapAmb: 0000000000000000 NoNewPrivs: 1

The credentials file and the certificate key are root-only. nginx can read the password hashes but not change them:

sudo stat -c '%a %U:%G %n' /var/lib/sac/data/zipkin-credentials.txt /etc/nginx/zipkin-admin.htpasswd /etc/nginx/zipkin-ingest.htpasswd /etc/ssl/zipkin/zipkin.key /etc/zipkin/zipkin.env /var/lib/zipkin

600 root:root /var/lib/sac/data/zipkin-credentials.txt 640 root:www-data /etc/nginx/zipkin-admin.htpasswd 640 root:www-data /etc/nginx/zipkin-ingest.htpasswd 600 root:root /etc/ssl/zipkin/zipkin.key 644 root:root /etc/zipkin/zipkin.env 700 zipkin:zipkin /var/lib/zipkin

Other notes:

• Who can reach it. Open ports 80 and 443 in the security group only to the networks your users and applications work from. Spans can contain request paths, tags and error messages, so treat the admin login like any other access to your application data.
• Headers. nginx sends a strict Content Security Policy, HSTS, X-Content-Type-Options: nosniff and X-Frame-Options: DENY. It removes Zipkin's Access-Control-Allow-Origin: *, because the web interface is served from the same address.
• Client addresses. nginx passes Zipkin the address of the connection, not one a client claims in X-Forwarded-For, and never passes your login on to Zipkin.
• The build. /usr/share/doc/sac-zipkin/BUILD-INFO.txt records the release and how the jar was rebuilt. jar-rebuild-plan.txt lists each replaced library with its checksum, and bundled-jars.txt lists every library in the shipped jar.
• Logs. nginx logs are in /var/log/nginx (without query strings). Zipkin logs to the journal (journalctl -u zipkin).

Ubuntu security updates, including the OpenJDK runtime, install automatically. To apply all updates now:

sudo apt-get update && sudo apt-get upgrade -y

New Zipkin releases arrive as new versions of this listing, built and checked the same way. Because traces are kept in memory, there is nothing to migrate. Launch the new version, point your applications or Collector at it, and stop the old instance. Read the Zipkin release notes before you move.