Grafana Tempo 3

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 as the software version, choose your Region, then choose Launch through EC2.

Choose m5.large (or larger for heavy trace volumes), your key pair and a security group as described in Prerequisites. Traces are stored on the root volume under /var/lib/tempo, so give it room for your traffic and retention.

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=tempo}]'

First boot finishes a few seconds after the instance is running. Grafana is ready about half a minute later.

Connect over SSH as the ubuntu user, which is the SSH user for Grafana Tempo 3 Tracing Stack on Ubuntu 24.04:

ssh -i <your-key.pem> ubuntu@<public-ip>

systemctl is-active tempo alloy grafana-server nginx tempo-stack-firstboot
sudo test -f /var/lib/sac/data/tempo-stack-firstboot.done && echo FIRSTBOOT_DONE

active active active active active FIRSTBOOT_DONE

Only SSH and nginx listen on network ports. Tempo, Alloy and Grafana are on the loopback address:

sudo ss -lntpH | awk '{match($6, /"[^"]+"/); print $4, substr($6, RSTART + 1, RLENGTH - 2)}' | sort -u

0.0.0.0:22 sshd 0.0.0.0:443 nginx 0.0.0.0:80 nginx 127.0.0.1:12345 alloy 127.0.0.1:3000 grafana 127.0.0.1:3200 tempo 127.0.0.1:4317 alloy 127.0.0.1:4318 alloy 127.0.0.1:4327 tempo 127.0.0.1:9096 tempo 127.0.0.53%lo:53 systemd-resolve 127.0.0.54:53 systemd-resolve [::]:22 sshd [::]:443 nginx [::]:80 nginx

Tempo's own readiness check, on the loopback address:

curl -s http://127.0.0.1:3200/ready; echo

ready

sudo cat /var/lib/sac/data/tempo-stack-credentials.txt

Only root can read the file. It looks like this, with this instance's own passwords in place of the <...> markers:

# Grafana Tempo 3 Tracing Stack (Tempo 3.1.1, Grafana OSS 13.3.0, Alloy 1.21.3) - generated on first boot by
# tempo-stack-firstboot.service. Every password here is unique to this instance; none was baked into the image.
# Keep this file secret. GRAFANA_URL=https://ec2-203-0-113-10.compute-1.amazonaws.com/ GRAFANA_ADMIN_USER=admin GRAFANA_ADMIN_PASSWORD=<GRAFANA_ADMIN_PASSWORD>
#
# Send traces (OTLP/HTTP, protobuf or JSON) from other machines to this URL with HTTP Basic auth: OTLP_HTTP_TRACES_URL=https://ec2-203-0-113-10.compute-1.amazonaws.com/otlp/v1/traces OTLP_USER=otlp OTLP_PASSWORD=<OTLP_PASSWORD> OTLP_AUTHORIZATION_HEADER=Basic <base64 of otlp:OTLP_PASSWORD>
# On this instance (or through an SSH tunnel) Alloy also takes OTLP/gRPC on 127.0.0.1:4317 and OTLP/HTTP on
# 127.0.0.1:4318 without a login.
#
# Sign in to Grafana as admin with the password above (change it under your profile). The Tempo datasource is ready in
# Explore. New ingest password:
sudo tempo-stack-otlp-password Self-test:
sudo tempo-stack-selftest
# Change the served name:
sudo tempo-stack-set-hostname <your-domain>
# User guide: https://secureanycloud.com/stacks/grafana-tempo-3/

The URL uses the instance's public DNS name from the EC2 instance metadata, or its public IP address when it has no DNS name. The certificate also names the public and private IP addresses. Requests to an IP address are redirected to the name. To serve Grafana under your own DNS name, see Step 10.

Without a login, Grafana's API and the ingest endpoint refuse the request. The health check answers without one:

H=$(sudo sed -n 's#^GRAFANA_URL=https://\([^/]*\)/.*#\1#p' /var/lib/sac/data/tempo-stack-credentials.txt)
curl -sk --resolve "$H:443:127.0.0.1" -o /dev/null -w '%{http_code}\n' "https://$H/api/user"
curl -sk --resolve "$H:443:127.0.0.1" -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' --data '{"resourceSpans":[]}' "https://$H/otlp/v1/traces"
curl -sk --resolve "$H:443:127.0.0.1" "https://$H/healthz"; echo

401 401 ok

The shipped self-test sends a test span through the public ingest path (nginx, Alloy, Tempo). It reads the span back through Grafana's Tempo data source by trace ID and with TraceQL.

The self-test also checks the sign-in and ingest logins, the cross-site guards, the loopback-only listeners and the metadata block. Passwords reach curl in a root-only temporary file, never on a command line:

sudo tempo-stack-selftest

SELFTEST-OK https 301 + healthz + Grafana 13.3.0 OSS healthy + /api/user 401 anonymous and admin/admin, 200 per-instance admin + Tempo datasource provisioned + OTLP ingest 401 none/wrong/admin login, 405 GET, 403 browser, 200 ingest login + span baeedf65 read back by trace ID and TraceQL through Grafana + 403 cross-origin login + 403 cross-site datasource proxy + 444 unknown host + loopback-only grafana/alloy/tempo + metadata 000 inside 4 services (control 200)

Applications send traces with OTLP/HTTP to https://<host>/otlp/v1/traces with the otlp login from the credentials file.

To see it work on the instance, send an example trace for an online shop. It is a POST /checkout request in a service called shop-frontend. That service calls cart.total and payment.charge in a shop-payments service, which runs a database query in shop-db.

The shell function otlp passes the Authorization header from the credentials file to curl on standard input, so it never appears on a command line:

otlp() {
sudo sed -n 's/^OTLP_AUTHORIZATION_HEADER=\(.*\)/header = "Authorization: \1"/p' /var/lib/sac/data/tempo-stack-credentials.txt | curl -sk -K - "$@"; }
H=$(sudo sed -n 's#^GRAFANA_URL=https://\([^/]*\)/.*#\1#p' /var/lib/sac/data/tempo-stack-credentials.txt)
TID=$(openssl rand -hex 16); echo "$TID" > /tmp/demo-trace-id python3 - "$TID" > /tmp/demo-trace.json <<'EOF' import json, os, sys, time tid = sys.argv[1] now = time.time_ns() def span(name, sid, parent, start_ms, dur_ms, kind=2, attrs=()): s = {"traceId": tid, "spanId": sid, "name": name, "kind": kind, "startTimeUnixNano": str(now - 900_000_000 + start_ms * 1_000_000), "endTimeUnixNano": str(now - 900_000_000 + (start_ms + dur_ms) * 1_000_000), "attributes": [{"key": k, "value": {"stringValue": v}} for k, v in attrs]} if parent: s["parentSpanId"] = parent return s ids = [os.urandom(8).hex() for _ in range(4)] def res(svc, spans): return {"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": svc}}]}, "scopeSpans": [{"scope": {"name": "guide-example"}, "spans": spans}]} print(json.dumps({"resourceSpans": [ res("shop-frontend", [span("POST /checkout", ids[0], None, 0, 420, 2, [("http.request.method", "POST"), ("http.route", "/checkout")]), span("cart.total", ids[1], ids[0], 15, 60, 1)]), res("shop-payments", [span("payment.charge", ids[2], ids[0], 90, 300, 2, [("payment.provider", "example")])]), res("shop-db", [span("SELECT orders", ids[3], ids[2], 120, 180, 3, [("db.system", "postgresql")])]), ]})) EOF otlp --resolve "$H:443:127.0.0.1" -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' --data-binary @/tmp/demo-trace.json "https://$H/otlp/v1/traces"

200

Read the trace back from Tempo by its trace ID, on the loopback address. Spans are readable within a few seconds:

sleep 5 curl -s "http://127.0.0.1:3200/api/v2/traces/$(cat /tmp/demo-trace-id)" | python3 -c ' import json, sys d = json.load(sys.stdin) for b in d["trace"]["resourceSpans"]: svc = [a["value"]["stringValue"] for a in b["resource"]["attributes"] if a["key"] == "service.name"][0] for ss in b["scopeSpans"]: for s in ss["spans"]: print(svc, "|", s["name"])' | sort

shop-db | SELECT orders shop-frontend | POST /checkout shop-frontend | cart.total shop-payments | payment.charge

TraceQL search finds a trace once Tempo has written it to a block, usually within a minute. This query asks Tempo for traces from shop-frontend that took longer than 100 ms:

for i in $(seq 1 20); do
R=$(curl -sG http://127.0.0.1:3200/api/search --data-urlencode 'q={ resource.service.name = "shop-frontend" && duration > 100ms }' --data-urlencode limit=5)
echo "$R" | grep -q "$(cat /tmp/demo-trace-id | sed 's/^0*//')" && break; sleep 5 done
echo "$R" | python3 -c 'import json,sys; [print(t["rootServiceName"], "|", t["rootTraceName"], "|", t["durationMs"], "ms") for t in json.load(sys.stdin)["traces"]]' | sort -u

shop-frontend | POST /checkout | 420 ms

Open the GRAFANA_URL from the credentials file in your browser. Your browser warns once about the self-signed certificate. Accept it, or install your own certificate (Step 10). Then sign in as admin with the GRAFANA_ADMIN_PASSWORD:

Grafana's brute-force protection is on. After five failed sign-ins, the account is locked for a few minutes. nginx also limits how fast sign-ins can be tried.

Open Explore (the compass icon) and keep the Tempo data source. Choose the Search or TraceQL query type and run a query such as { resource.service.name = "shop-frontend" }. Each row is one trace with its root service, root span and duration:

Click a trace ID to open the trace view. The waterfall shows every span with its service and timing. Here, the checkout spends most of its time in payment.charge, and most of that in the SELECT orders database query:

The Tempo data source is provisioned from /etc/grafana/provisioning/datasources/sac-tempo.yaml and points at Tempo on http://127.0.0.1:3200. It is the default data source. Because it is provisioned from a file, it is read-only in the UI, and Grafana re-applies it on every start:

Streaming search is switched off in the data source. Grafana streams TraceQL results over a WebSocket, which fails silently when you reach Grafana through a different name or a proxy. Searches return the same results without streaming.

Point your OpenTelemetry SDKs, collectors or agents at the ingest URL with the otlp login. Both OTLP/HTTP encodings (protobuf and JSON) are accepted.

Copy the instance's certificate (/etc/ssl/tempo-stack/tempo-stack.crt) to the senders so they can verify it. Alternatively, install a certificate that your systems already trust (Step 10).

Any OpenTelemetry SDK uses environment variables. The header value is OTLP_AUTHORIZATION_HEADER from the credentials file, with the space written as %20:

OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<host>/otlp/v1/traces OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic%20<base64 of otlp:password> OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE=/etc/otel/tempo-stack.crt OTEL_SERVICE_NAME=my-service

An OpenTelemetry Collector or Grafana Alloy on another machine:

otelcol.auth.basic "tempo_stack" { username = "otlp" password = sys.env("TEMPO_STACK_OTLP_PASSWORD") } otelcol.exporter.otlphttp "tempo_stack" { client { endpoint = "https://<host>/otlp" auth = otelcol.auth.basic.tempo_stack.handler tls { ca_file = "/etc/alloy/tempo-stack.crt" } } }

OTLP/gRPC. Alloy's gRPC receiver has no login, so it stays on the loopback address. From your workstation, forward a local port over SSH and point the exporter at localhost:4317. The traffic is plaintext, inside the SSH tunnel:

ssh -i <your-key.pem> -N -L 4317:127.0.0.1:4317 ubuntu@<public-ip>

Applications on the instance itself can send to 127.0.0.1:4317 (gRPC) or http://127.0.0.1:4318/v1/traces (HTTP) without a login.

Keep the ingest password in a secrets store, never in source code. To issue a new password, run sudo tempo-stack-otlp-password and update your senders. The old password stops working at once.

Tempo runs as one process (-target=all) with local storage. The write-ahead log and the blocks are under /var/lib/tempo. Its configuration is /etc/tempo/tempo.yaml:

sudo grep -A6 '^storage:' /etc/tempo/tempo.yaml
sudo du -sh /var/lib/tempo

storage: trace: backend: local wal: path: /var/lib/tempo/wal local: path: /var/lib/tempo/blocks 3.7M /var/lib/tempo

By default, blocks are kept for 14 days (336h, Tempo's default). To change the retention, add a compaction block under backend_worker in /etc/tempo/tempo.yaml. This example keeps blocks for seven days:

backend_worker: compaction: block_retention: 168h

Then run sudo systemctl restart tempo. Keep every address in the file on 127.0.0.1.

For long retention or high volumes, Tempo can store blocks in Amazon S3 instead. Use backend: s3 with an s3: block giving bucket, endpoint and region. See the Tempo storage documentation.

tempo.service is blocked from the instance metadata service, so it cannot use an instance role. Give it an access key for a bucket-only IAM user through a root-only environment file. Run sudo systemctl edit tempo, then set EnvironmentFile= with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

If you reach S3 through a VPC interface endpoint with private addresses, add the endpoint's subnet to the egress allow-list (Step 11).

Point a DNS record at the instance. An Elastic IP keeps the address stable.

Then set the name that Grafana is served as. This command mints a new self-signed certificate for the name and updates nginx, the cross-site guards, Grafana's root_url and the URLs in the credentials file. It keeps the name across reboots:

sudo tempo-stack-set-hostname <your-domain>

To use a certificate from a public or internal certificate authority instead, copy it over the generated pair and reload nginx. An example is a certificate issued by Let's Encrypt for <your-domain>. Running tempo-stack-set-hostname again replaces it with a self-signed certificate:

sudo install -m 0644 <your-domain>.crt /etc/ssl/tempo-stack/tempo-stack.crt
sudo install -m 0600 <your-domain>.key /etc/ssl/tempo-stack/tempo-stack.key
sudo nginx -t && sudo systemctl reload nginx

The tempo, alloy and grafana accounts may connect to public internet addresses. They may not connect to the instance metadata service, link-local, private (RFC 1918), carrier-grade NAT or reserved addresses, or your VPC's own address ranges, and not over IPv6.

This stops a compromised or misconfigured service from probing your private network. nginx and your own shell are not affected. Check the rules:

sudo iptables -S SAC-TEMPO-STACK-EGRESS | head -5 sudo iptables -S OUTPUT | grep -c SAC

-N SAC-TEMPO-STACK-EGRESS -A SAC-TEMPO-STACK-EGRESS -d 127.0.0.0/8 -o lo -j RETURN -A SAC-TEMPO-STACK-EGRESS -d 169.254.0.0/16 -p tcp -j REJECT --reject-with tcp-reset -A SAC-TEMPO-STACK-EGRESS -d 169.254.0.0/16 -j REJECT --reject-with icmp-admin-prohibited -A SAC-TEMPO-STACK-EGRESS -d 100.64.0.0/10 -p tcp -j REJECT --reject-with tcp-reset 4

If Tempo or Grafana must reach something on a private address, such as an S3 VPC endpoint or a Prometheus or alert receiver in your VPC, add one CIDR per line to /etc/sac/tempo-stack-egress-allow. Then reload the rules with sudo systemctl reload sac-tempo-stack-egress. Metadata and link-local addresses can never be allowed.

The metadata block is a systemd setting on each service:

systemctl show -p IPAddressDeny tempo alloy grafana-server nginx

IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128

A request for a host name the instance does not serve gets no answer at all. nginx closes the connection, so curl exits with code 52:

curl -sk -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000 curl exit 52

The certificate was generated on this instance for its own names:

sudo openssl x509 -in /etc/ssl/tempo-stack/tempo-stack.crt -noout -subject -ext subjectAltName

subject=CN = ec2-203-0-113-10.compute-1.amazonaws.com, O = sac X509v3 Subject Alternative Name: DNS:ec2-203-0-113-10.compute-1.amazonaws.com, IP Address:203.0.113.10, IP Address:172.31.20.15, DNS:ip-172-31-20-15.ec2.internal, IP Address:127.0.0.1, DNS:localhost

Grafana runs the open-source edition and loads its bundled plugins, the Tempo data source among them, from the root-owned package directory. The grafana account cannot change plugin code, and no other plugins are installed:

sudo journalctl -u grafana-server --no-pager -o cat | grep -F 'msg="Path Plugins"' | tail -1 | grep -o 'path=.*'

path="[/var/lib/grafana/plugins /usr/share/grafana/data/plugins-bundled]"

/usr/local/bin/tempo --version 2>&1 | head -1 /usr/local/bin/alloy --version 2>&1 | head -1 dpkg-query -W -f='grafana ${Version}\n' grafana nginx -v 2>&1

tempo, version v3.1.1 (branch: HEAD, revision: 9db7df7) alloy, version v1.21.3 (branch: HEAD, revision: f2108cd) grafana 13.3.0 nginx version: nginx/1.24.0 (Ubuntu)

These are the main paths on the instance:

• /etc/tempo/tempo.yaml: Tempo's configuration (loopback listeners, local storage, no usage report)
• /var/lib/tempo: trace storage (write-ahead log, live store and blocks)
• /etc/alloy/config.alloy: Alloy's pipeline (OTLP receiver, batch, export to Tempo)
• /etc/grafana/grafana.ini: Grafana's settings (127.0.0.1:3000, no sign-up, no anonymous access, outbound features off)
• /etc/grafana/provisioning/datasources/sac-tempo.yaml: the provisioned Tempo data source
• /etc/nginx/tempo-stack-otlp.htpasswd: the ingest login (otlp, bcrypt hash)
• /etc/ssl/tempo-stack/: this instance's certificate and private key
• /var/lib/sac/data/tempo-stack-credentials.txt: URLs, logins and passwords (root only)
• /etc/sac/tempo-stack-egress-allow: private networks the services may reach (empty by default)
• /usr/share/doc/sac-tempo-stack/: the licences, SOURCE-OFFER.txt and the go.mod and go.sum files the binaries were built from

Grafana administrator: sign in, open your profile (the avatar at the top right) and choose Change password. Then update or delete GRAFANA_ADMIN_PASSWORD in the credentials file (sudo nano /var/lib/sac/data/tempo-stack-credentials.txt). Add other users under Administration > Users and access.

Trace ingest: sudo tempo-stack-otlp-password replaces the otlp password and writes the new password and Authorization header to the credentials file. nginx picks it up at once. Update every sender.

• Keep port 443 open only to the people who use Grafana and the applications that send traces. Keep port 22 open only to your admin network.
• Use your own DNS name and a certificate that your senders trust (Step 10).
• Traces can contain URLs, user IDs and query text. Treat them like application logs, and keep the Grafana and ingest passwords in a secrets store.
• The instance applies Ubuntu security updates automatically, including Grafana updates from Grafana's repository within the 13.3 line.
• A Grafana package upgrade copies its bundled plugins to /var/lib/grafana/plugins-bundled again. This image does not use that copy, because it reads the packaged one, and you can delete it.
• To update Tempo or Alloy, back up /var/lib/tempo first. Then replace /usr/local/bin/tempo or /usr/local/bin/alloy with a newer build after reading its release notes, and restart the service.