Grafana Tempo 3 tracing stack

• In the Azure Portal, search the Marketplace for Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 LTS by SecureAnyCloud.
• Select Create.
• Choose your subscription, resource group and region.
• Set the size to Standard_B2ms or larger.
• Set the authentication type to SSH public key and provide your key. The administrator username is azureuser.
• On the Networking tab, allow inbound 22, 80 and 443.
• Select Review + create, then Create.

Accept the Marketplace terms once per subscription, then create the virtual machine.

# Accept the Marketplace terms once per subscription, before the first deployment.
az vm image terms accept --publisher sac --offer tempo-grafana-stack-ubuntu-24-04 --plan default
az group create --name tempo-stack-rg --location eastus
az vm create \
  --resource-group tempo-stack-rg \
  --name tempo-stack-vm \
  --image sac1647283583153:tempo-grafana-stack-ubuntu-24-04:default:latest \
  --size Standard_B2ms \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
az vm open-port --resource-group tempo-stack-rg --name tempo-stack-vm --port 22 --priority 1001
az vm open-port --resource-group tempo-stack-rg --name tempo-stack-vm --port 80 --priority 1002
az vm open-port --resource-group tempo-stack-rg --name tempo-stack-vm --port 443 --priority 1003

Connect to the virtual machine as azureuser.

ssh azureuser@<vm-public-ip>

First boot takes about 40 seconds to generate this machine's secrets. Wait for the sentinel before you go further.

sudo test -f /var/lib/sac/data/tempo-grafana-stack-firstboot.done && echo "first boot complete"

Confirm that all four components are active and that the ports reachable from outside are exactly 22, 80 and 443.

for u in tempo alloy grafana-server nginx tempo-grafana-stack-firstboot; do printf '%-34s %s\n' "$u" "$(systemctl is-active $u)" done

sudo /usr/local/sbin/tempo-grafana-stack-health.sh

Tempo (3200, 4327), Alloy (4317, 4318, 12345) and Grafana (3000) are all bound to 127.0.0.1. This is deliberate. Tempo and Alloy have no authentication mechanism at all, so exposing them would publish an unauthenticated write and read path to your trace data.

Every secret is generated on your own virtual machine at first boot and written to a root-only file. Read it.

sudo cat /root/tempo-grafana-stack-credentials.txt

The file records the Grafana URL and administrator password, the OTLP ingest endpoint with its own user and password, and the SSH-tunnel command for OTLP gRPC.

grafana.address_source tells you how the machine worked out its own public address. On Azure, the instance metadata service returns an empty response for a Standard-SKU public IP. The appliance then falls back to the load-balancer metadata view, which does report it. If that also returns nothing (a machine with no public IP), the file records the private address and says so.

Run the self-test that ships with the image. It exercises the real product path instead of only checking that processes are running. It sends a span into Alloy over OTLP gRPC and again over OTLP HTTP, and it confirms that Tempo stored each one. It then reads each span back by trace ID through Grafana's provisioned Tempo datasource, the same path the browser uses.

sudo /usr/local/sbin/tempo-grafana-stack-selftest.sh

The self-test begins with a control. It queries a trace ID that was never sent and requires the query to return nothing. Without that control, a query path that answered for any input would look like a working round trip.

A new appliance has nothing to look at yet. Run the generator to send a realistic seven-span checkout trace across five services, so the waterfall and the service graph have something meaningful in them.

sudo python3 /usr/local/share/sac/tempo-grafana-stack/emit-demo-trace.py 15

The generator posts OTLP/JSON to Alloy's HTTP receiver exactly as an instrumented application would. It exercises the real ingest path and does not write to Tempo directly.

Open https://<vm-public-ip>/ in a browser and sign in as admin with the grafana.password value from Step 5.

The certificate is self-signed and generated for your machine, so the browser warns on the first visit. Step 14 replaces it with your own.

Grafana keeps its brute-force protection enabled. Five consecutive failed sign-ins block the account for a few minutes, and during that window even the correct password is refused. If you are locked out, wait instead of resetting anything.

Go to Explore and choose the Tempo datasource. It is already selected because it is the default. Pick the TraceQL query type and run this query.

{ resource.service.name = "frontend" }

A span is retrievable by trace ID within about two seconds, but becomes searchable only after Tempo's live store cuts and completes a block. With this image's settings, that takes roughly a minute. If a trace you just sent is not in the search results yet, wait and run the query again. Nothing is wrong.

You can confirm the same query outside the browser, straight through Grafana's datasource proxy.

NOW=$(date +%s)
curl -sG -u "admin:<GRAFANA_PASSWORD>" \
  "http://127.0.0.1:3000/api/datasources/proxy/uid/sac-tempo/api/search" \
  --data-urlencode 'q={ resource.service.name = "frontend" }' \
  --data-urlencode "start=$((NOW-3600))" --data-urlencode "end=$((NOW+120))" \
  --data-urlencode 'limit=5' | head -c 400

Select any trace from the search results. Grafana draws the full waterfall: every span, its parent, its service and its duration.

This is the whole product working together. The trace entered through Alloy's OTLP receiver, Tempo stored it, and Grafana fetched it back through the provisioned datasource. That is four components on one path.

To fetch the same trace as JSON, run this command.

TID=$(sudo python3 /usr/local/share/sac/tempo-grafana-stack/emit-demo-trace.py 1 | awk '/DEMO_TRACE/{print $2}') sleep 5 curl -s "http://127.0.0.1:3200/api/traces/${TID}" | head -c 300

Go to Connections → Data sources → Tempo. It is already configured against Tempo's loopback API, and it is the default datasource.

The datasource is provisioned from a file, not from the database. View the file.

sudo cat /etc/grafana/provisioning/datasources/sac-tempo.yaml

Because the datasource is file-provisioned, Grafana re-applies it on every start. It cannot drift, and it survives a database reset.

Streaming search is deliberately turned off. Grafana's streaming TraceQL results travel over a WebSocket. Grafana rejects that WebSocket when the browser's origin does not match the root_url recorded at first boot. This happens whenever you reach the machine by a different hostname, through a reverse proxy, or over an SSH tunnel. The failure is silent, and the search panel shows nothing. The non-streaming path has no such dependency.

There are two supported paths. They differ because Alloy has no authentication of its own.

OTLP/HTTP over TLS works from anywhere. nginx terminates TLS and requires HTTP basic authentication, using the otlp.user and otlp.password values from Step 5.

curl -ks -o /dev/null -w 'ingest: HTTP %{http_code}\n' -X POST \
  -u "<OTLP_USER>:<OTLP_PASSWORD>" \
  -H 'Content-Type: application/json' \
  --data '{"resourceSpans":[]}' \
  https://127.0.0.1/otlp/v1/traces

For an OpenTelemetry SDK, set this variable.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<vm-public-ip>/otlp/v1/traces OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic <base64 of user:password>

nginx refuses an anonymous or wrong-password push.

curl -ks -o /dev/null -w 'anonymous push: HTTP %{http_code}\n' -X POST \
  -H 'Content-Type: application/json' --data '{"resourceSpans":[]}' \
  https://127.0.0.1/otlp/v1/traces

OTLP/gRPC works over an SSH tunnel. Alloy's gRPC receiver is unauthenticated, so it stays on loopback. Run this command on your own machine, then point your exporter at 127.0.0.1:4317 locally.

ssh -L 4317:127.0.0.1:4317 azureuser@<vm-public-ip>

If you need unauthenticated gRPC ingest from a private network, first restrict it with a network security group rule scoped to that subnet, then change the bind address. Never expose it to the internet.

This image runs Tempo as a single process against local disk. That suits one machine and does not suit a fleet. These are the limits.

• Capacity is this machine's disk. There is no object storage behind it, so retention is bounded by free space.
• There is no replication. If the disk is lost, the traces are lost.
• There is no query fan-out. One process serves ingest, compaction and queries.

To move to object storage, change the backend in /etc/tempo/tempo.yaml from local to azure (or s3) and restart Tempo.

To move beyond one machine, run Tempo's components separately with -target=distributor, -target=ingester and so on. This also introduces the Kafka-backed ingest path that Tempo 3 uses in its microservices mode.

grep -A6 '^storage:' /etc/tempo/tempo.yaml

Tempo 3 keeps state in several directories under /var/lib/tempo: the write-ahead log and blocks, plus the live store and the backend scheduler. This image pins all of them under that one root.

sudo ls -1 /var/lib/tempo

The shipped certificate is self-signed and unique to this machine. Replace it with your own. First, list the current files.

sudo ls -l /etc/ssl/sac/

Copy your certificate to /etc/ssl/sac/tempo-grafana-stack.crt and your key to /etc/ssl/sac/tempo-grafana-stack.key (mode 0600, owned by root). Then check the configuration and reload nginx.

sudo nginx -t

If you also change the hostname that users connect to, update root_url in /etc/grafana/grafana.ini to match. Otherwise Grafana refuses Grafana Live features for that origin.

Check the status of all four components.

systemctl --no-pager --lines=0 status tempo alloy grafana-server nginx

View the logs for each component. These commands filter to the startup lines instead of dumping the raw journal, because Tempo is chatty at idle.

sudo journalctl -u tempo --no-pager | grep -E 'Starting Tempo|server listening' | tail -3

sudo journalctl -u alloy --no-pager | grep -E 'server now listening|finished node evaluation' | tail -3

no jobs found in Tempo's log is expected and harmless. Tempo 3's backend worker polls its scheduler for compaction work. On a quiet appliance there is none, so the worker logs error calling scheduler ... no jobs found and backs off. This is not a fault, and the worker retries on its own.

Alloy's own diagnostics interface is on loopback at 127.0.0.1:12345. Reach it over an SSH tunnel, in the same way as the gRPC receiver.

Traces do not appear in search. Confirm first that they reached Tempo by trace ID. Search needs a completed block, which takes about a minute.

curl -s "http://127.0.0.1:3200/api/search/tag/service.name/values" | head -c 300

The ingest endpoint returns 401. The credential is the otlp.user and otlp.password pair from Step 5, not the Grafana login.

The ingest endpoint returns 400. The body is not valid OTLP JSON. {"resourceSpans":[]} is the smallest valid payload.

Grafana refuses the correct password. Five consecutive failures block the account for a few minutes. Wait for the window to expire.

Nothing is listening on 443. nginx and Grafana are both gated on the first-boot marker, and neither starts before this machine's secrets exist. Check the marker and the first-boot log.

sudo test -f /var/lib/sac/data/tempo-grafana-stack-ready && echo "bootstrap marker present"

sudo journalctl -u tempo-grafana-stack-firstboot --no-pager | tail -20

• Tempo, Alloy and Grafana all bind to loopback. Only nginx (443), its redirect (80) and SSH (22) are reachable from outside.
• The image contains no Grafana database, so no administrator account, password hash or salt is shared between machines. Your machine creates its own at first boot.
• The TLS key pair, the Grafana secret_key and the OTLP credential are all generated on your machine and never baked into the image.
• The default admin/admin login, a blank password, common guesses and the build-time password are all refused.
• Credentials live in /root/tempo-grafana-stack-credentials.txt, mode 0600, readable only by root.
• Unattended security upgrades are enabled. The four components are pinned to their minor lines, so security patches flow while a major version change does not happen unattended.

Grafana Tempo and Grafana Community Edition are licensed under AGPL-3.0. Grafana Alloy is licensed under Apache-2.0, and nginx under the 2-clause BSD licence. This image ships the AGPL-licensed open-source Grafana build, never Grafana Enterprise. Check the installed versions and licences with this command.

for p in tempo grafana alloy nginx; do printf '%-8s %s\n' "$p" "$(dpkg-query -W -f='${Version}' $p)"; done

SecureAnyCloud provides 24/7 support for the image itself, covering deployment, first boot, the pre-wired configuration and the Azure integration.

Deploy on Azure

Find Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 LTS by SecureAnyCloud in the Azure Marketplace.

Need Help?

Contact SecureAnyCloud support through the Azure Marketplace listing.

Deploy on Azure

Launch Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com