SecureAnyCloud
Deployment guide

VictoriaTraces

Overview

VictoriaTraces is a distributed tracing database from the VictoriaMetrics project. It receives OpenTelemetry spans over OTLP, stores them compactly on disk with no external database behind it, and serves them back through Jaeger-compatible query APIs and a built-in web UI.

A traditional tracing stack needs a collector, a storage backend such as Cassandra or Elasticsearch, and a separate query service. VictoriaTraces is a single static Go binary that does the storage and the querying itself.

The SecureAnyCloud image installs the pinned VictoriaTraces 0.9.4 binary under systemd, then locks it down for a marketplace appliance. VictoriaTraces has no built-in authentication of any kind. That matters more for a tracing backend than for most services. An open ingest endpoint lets anyone write or poison your trace store, and an open query endpoint exposes every span in it. Spans routinely carry hostnames, URL paths, user identifiers, SQL statements and the topology of your services.

For that reason, the binary is bound to loopback 127.0.0.1:10428, where it is never reachable from outside the VM. An nginx reverse proxy on port 80 puts a per-VM HTTP Basic Auth gate (user admin, unique password generated on first boot) in front of both the ingest path and the query paths. The only unauthenticated route is a static /healthz used by load balancer probes. All stored spans live on a dedicated Azure data disk mounted at /var/lib/victoria-traces.

What you get:

  • VictoriaTraces 0.9.4 installed as a single Go binary, running as the victoriatraces systemd service under a dedicated non-login account.
  • OTLP over HTTP span ingestion at /insert/opentelemetry/v1/traces, accepting protobuf or JSON.
  • Jaeger-compatible query APIs under /select/jaeger/api/ and the built-in web UI at /select/vmui/.
  • Per-VM HTTP Basic Auth (user admin) that protects the web UI, the ingest endpoint and every query API. The password is unique, generated on first boot, and there is no default login.
  • An unauthenticated /healthz endpoint for Azure Load Balancer health probes.
  • A dedicated Azure data disk mounted at /var/lib/victoria-traces that holds all span data, with a 7-day default retention and an 80 percent disk usage cap you can tune.
  • A shipped self test at /usr/local/sbin/victoriatraces-selftest.sh that proves the auth gate and a full ingest-to-query round trip.
  • An Ubuntu 24.04 LTS base, patched at build time, with unattended security upgrades enabled.
  • 24/7 SecureAnyCloud support.

Project maturity

VictoriaTraces is a young component of the VictoriaMetrics family. Factor that in before you make it the system of record for production tracing.

The project's first release was in July 2025, and it is under active development. Upstream has not yet declared a generally available release. Its own roadmap lists items that must be completed before GA, including "finalize the data structure and commit to backward compatibility", and the version numbering is still 0.x. Upstream does not describe the product as beta or experimental, but it is explicitly pre-GA by its own statement.

The practical consequence is that the on-disk span format is not yet covered by a backward compatibility commitment. A future major upgrade may require re-ingesting your spans instead of upgrading in place. Traces are usually short-lived, high-volume and reproducible, which makes this far less painful than it would be for a metrics or billing store. It is still a real consideration.

Two individual features are separately marked experimental upstream and are deliberately not enabled in this image: the Jaeger service dependency graph API and the Grafana Tempo query APIs.

VictoriaTraces and VictoriaMetrics are trademarks of their respective owners. This image is provided by SecureAnyCloud and is not affiliated with, endorsed by, or sponsored by VictoriaMetrics.

Prerequisites

  • An Azure subscription and permission to create a VM.
  • An SSH key pair (ssh-keygen -t ed25519).
  • Inbound access to port 22 (SSH) and port 80 (the web UI, ingest and query APIs) on the VM, and optionally 443 if you add TLS.
  • A recommended size of Standard_B2s or larger.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure portal, choose Create a resource, search the Marketplace for VictoriaTraces by SecureAnyCloud, and select Create.

On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab, then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:victoriatraces-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name victoriatraces \
  --image sac1647283583153:victoriatraces-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name victoriatraces --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Two units make up the appliance. victoriatraces.service is the tracing database itself, bound to loopback. nginx.service is the authenticating reverse proxy, and it is the only service listening on a public interface.

systemctl is-active victoriatraces.service nginx.service

Both services should report active. Confirm the listener layout and the dedicated data disk at the same time:

ss -tlnp | grep -E '127.0.0.1:10428 |:80 ' findmnt -no SOURCE,TARGET,FSTYPE,SIZE /var/lib/victoria-traces

VictoriaTraces appears on 127.0.0.1:10428 only. It has no public listener, which is deliberate.

Step 5 - Retrieve your per-VM password

Every VM generates its own web password on first boot. Nothing is baked into the image, so there is no default login to change. Read the credentials file:

sudo cat /root/victoriatraces-credentials.txt

The file is 0600 root:root. It contains the instance URL, the OTLP endpoint, the username admin and the generated password. It also records the pinned runtime configuration.

Step 6 - Understand the security gate

The unauthenticated health endpoint is open so that Azure Load Balancer probes work. nginx serves it directly and it never touches VictoriaTraces, so it exposes nothing about your spans.

curl -s http://localhost/healthz

Everything else requires the per-VM credentials. Verify this yourself, because it is the single most important property of the appliance. The following commands prove that both halves are closed to anonymous callers and that only the correct password opens them:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
echo "unauth ingest : $(curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' --data '{}' http://127.0.0.1/insert/opentelemetry/v1/traces)"
echo "unauth query : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/select/jaeger/api/services)"
echo "unauth web UI : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/select/vmui/)"
echo "wrong pw : $(curl -s -o /dev/null -w '%{http_code}' -u admin:wrong-pw http://127.0.0.1/select/jaeger/api/services)"
echo "correct pw : $(curl -s -o /dev/null -w '%{http_code}' -u admin:$PW http://127.0.0.1/select/vmui/)"

The first four return 401 and the last returns 200.

A browser that visits the VM without credentials meets the same gate before any part of the interface loads.

Step 7 - Send your first spans over OTLP

The ingest endpoint is /insert/opentelemetry/v1/traces, and it speaks standard OTLP over HTTP. Set Content-Type: application/json to send OTLP/JSON. Set application/x-protobuf to send the binary encoding that most collectors use by default.

This command sends a single span that describes a checkout operation in a service called guide-demo:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
TID=$(openssl rand -hex 16);
SID=$(openssl rand -hex 8);
NOW=$(date +%s)
curl -s -o /dev/null -w 'ingest -> HTTP %{http_code}\n' -u admin:$PW \
  -H 'Content-Type: application/json' --data-binary "{ \"resourceSpans\": [{ \"resource\": {\"attributes\": [{\"key\": \"service.name\", \"value\": {\"stringValue\": \"guide-demo\"}}]}, \"scopeSpans\": [{\"spans\": [{ \"traceId\": \"$TID\", \"spanId\": \"$SID\", \"name\": \"checkout\", \"kind\": 2, \"startTimeUnixNano\": \"${NOW}000000000\", \"endTimeUnixNano\": \"$((NOW+1))000000000\", \"attributes\": [{\"key\": \"http.response.status_code\", \"value\": {\"intValue\": \"200\"}}] }]}] }] }" http://127.0.0.1/insert/opentelemetry/v1/traces echo "$TID" | sudo tee /tmp/guide-trace-id >/dev/null

A 200 means the span was accepted.

Step 8 - Query your spans back

Spans are not queryable the instant they are accepted. VictoriaTraces buffers incoming spans and flushes them to the searchable store on an interval, typically ten to twenty seconds. Any automation you write should poll instead of assuming the span is immediately visible.

List the services that have reported spans:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt) for i in $(seq 1 30); do
BODY=$(curl -s -u admin:$PW http://127.0.0.1/select/jaeger/api/services) case "$BODY" in *guide-demo*) echo "$BODY"; break ;; esac sleep 3 done

Now fetch the trace itself. The Jaeger API takes start and end as Unix timestamps in microseconds, which is a common source of empty results:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
NOW=$(date +%s);
S=$(( (NOW - 3600) * 1000000 ));
E=$(( (NOW + 3600) * 1000000 )) for i in $(seq 1 30); do
BODY=$(curl -s -u admin:$PW "http://127.0.0.1/select/jaeger/api/traces?service=guide-demo&start=$S&end=$E&limit=5") case "$BODY" in *checkout*) break ;; esac sleep 3 done echo "$BODY" | python3 -m json.tool | head -25

List the operations a given service has recorded. This shows you what is worth filtering on:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
curl -s -u admin:$PW http://127.0.0.1/select/jaeger/api/services/guide-demo/operations

To fetch one specific trace by id, use the single trace endpoint:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
TID=$(sudo cat /tmp/guide-trace-id)
curl -s -u admin:$PW "http://127.0.0.1/select/jaeger/api/traces/$TID" | python3 -m json.tool | head -20

Step 9 - Explore traces in the web UI

Open http://<vm-public-ip>/select/vmui/ in a browser and sign in as admin with the password from Step 5. The Traces Explorer shows the spans arriving over time, with a breakdown by operation and service underneath.

The Trace query box filters spans. Because the service name field contains dots and a colon, quote the whole field name. To see only the spans from one service:

"resource_attr:service.name":"checkout-service"

Press Execute Query. The chart, the totals and the results below all narrow to that service.

To filter by operation as well, add the span name. This query shows only the place-order spans from checkout-service:

"resource_attr:service.name":"checkout-service" AND name:"place-order"

The Table tab lays the matching spans out as rows, which is the quickest way to scan timings across many spans.

The JSON tab shows each span in full, including its duration and every attribute your instrumentation attached. Use it to confirm that the attributes you expect, such as HTTP status codes or database statements, are arriving.

Step 10 - Point an OpenTelemetry Collector at this VM

In production, you will normally send spans through an OpenTelemetry Collector instead of with curl. Add this VM as an OTLP HTTP exporter in your collector configuration. The auth extension supplies the same Basic Auth credentials the API expects:

extensions: basicauth/victoriatraces: client_auth: username: admin password: <VICTORIATRACES_PASSWORD> exporters: otlphttp/victoriatraces: traces_endpoint: http://<vm-public-ip>/insert/opentelemetry/v1/traces auth: authenticator: basicauth/victoriatraces encoding: proto service: extensions: [basicauth/victoriatraces] pipelines: traces: receivers: [otlp] exporters: [otlphttp/victoriatraces]

To point an instrumented application straight at the VM, with no collector in between, set the standard OpenTelemetry SDK environment variables. Every OpenTelemetry SDK understands them:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<vm-public-ip>/insert/opentelemetry/v1/traces OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic <base64-of-admin:password> OTEL_SERVICE_NAME=my-application

Generate the header value for the Authorization header with:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt) printf 'Authorization=Basic %s\n' "$(printf 'admin:%s' "$PW" | base64 -w0)"

Everything travels over plain HTTP by default. Before you send real production spans across a public network, put the VM behind a private network, a VPN or a TLS-terminating load balancer. See Step 13.

Step 11 - Tune retention and size the data disk

Span volume is much higher than log or metric volume, so retention is the setting you are most likely to change. The image ships two limits that work together:

  • -retentionPeriod=7d drops spans older than seven days.
  • -retention.maxDiskUsagePercent=80 caps the store at 80 percent of the data disk, dropping the oldest partitions first.

The second limit is a safety net. Whatever retention period you choose, a sudden burst of spans cannot fill the volume and wedge the service. Check the current values and how much of the disk is in use:

systemctl show -p ExecStart --value victoriatraces.service | grep -oE '\-retentionPeriod=[^ ]+|\-retention.maxDiskUsagePercent=[^ ]+' df -h /var/lib/victoria-traces du -sh /var/lib/victoria-traces 2>/dev/null || true

To change the retention period, edit the systemd unit and restart the service:

sudo systemctl edit --full victoriatraces.service
# change -retentionPeriod=7d to e.g. 30d
sudo systemctl daemon-reload
sudo systemctl restart victoriatraces.service

Sizing the disk. Storage need scales with spans per second and retention, not with the number of services. As a planning starting point, VictoriaTraces stores a typical span in well under a kilobyte after compression, so the need is roughly spans per second x 86400 x retention days x span size. At 100 spans per second and 7 days, that is on the order of a few gigabytes, which the 32 GiB default disk holds comfortably.

If you raise retention to 30 days or push several hundred spans per second, grow the data disk in the Azure portal and then extend the filesystem. Watch the actual growth rate over your first few days with du -sh, and size from your own measured numbers instead of an estimate.

Step 12 - Verify the appliance at any time

The image ships a self test that re-proves everything this guide has covered. It checks that the health probe is open and that unauthenticated ingest and unauthenticated queries are both rejected. It checks that a wrong password is rejected. It also checks that a real span sent over OTLP comes back through the Jaeger query API with its operation name and attributes intact. Run it:

sudo /usr/local/sbin/victoriatraces-selftest.sh

On success, the test prints a single OK- line. Otherwise, it prints a SELFTEST-FAIL: line that names the failing check. It is a useful smoke test after you change the nginx configuration, rotate the password or resize the disk.

A note on the spans it leaves behind. The image itself ships with a completely empty trace store. However, this self test runs once automatically during first boot to prove that the appliance works before you rely on it, and each run ingests one real span. The first time you open the UI, you will see a service named sac-selftest-<timestamp> alongside your own. Another appears each time you run the self test by hand.

These spans are harmless. They are the proof that the round trip worked, and they disappear on their own once they pass the retention period. To start from a completely clean store, stop the service, clear the storage directory and start it again, before you point any application at the VM:

sudo systemctl stop victoriatraces
sudo find /var/lib/victoria-traces -mindepth 1 -maxdepth 1 ! -name lost+found -exec rm -rf {} +
sudo systemctl start victoriatraces

Step 13 - Production hardening

  • Add TLS. Spans and credentials travel in clear text over HTTP. Put the VM behind an Azure Application Gateway or Load Balancer that terminates TLS, or add a certificate to nginx directly and open port 443. The image does not ship a certificate because it cannot know your domain.
  • Restrict the network. Tracing backends should stay private to your own infrastructure. Scope the network security group for port 80 to the subnets your applications and collectors run in, instead of leaving it open to the internet.
  • Rotate the password. Replace the generated password by writing a new bcrypt entry. Then update the credentials note so the self test and your collectors stay in sync:
sudo htpasswd -B /etc/nginx/.victoriatraces.htpasswd admin
sudo systemctl reload nginx
  • Back up what matters. Spans are usually reproducible and short-lived, so most teams do not back up a trace store. If yours is an exception, snapshot the data disk instead of copying files out from under a running service.
  • Keep the OS patched. Unattended security upgrades are enabled by default. Leave them on.

Troubleshooting

A query returns an empty result immediately after ingesting. This is the flush interval, not an error. Spans become searchable roughly ten to twenty seconds after they are accepted. Poll as shown in Step 8 instead of querying once.

A Jaeger query returns nothing even after waiting. Check the start and end parameters. They are Unix timestamps in microseconds, not milliseconds or seconds. A window in the wrong unit silently matches nothing.

The UI query box returns no results for a service you know exists. Quote the whole field name: "resource_attr:service.name":"checkout-service". Leaving it unquoted is a parse error, because the field name itself contains a colon and dots.

Your collector gets 401. The credentials cover the ingest path as well as the query paths. Confirm the exporter is sending Basic Auth and that the password matches /root/victoriatraces-credentials.txt.

There is a sac-selftest-... service you did not create. That is the first-boot self test described in Step 12, not stray build data. See that step for what it is and how to clear it.

The service will not start. Check the journal and confirm that the data disk is mounted, because VictoriaTraces will not start if its storage path is missing:

systemctl status victoriatraces.service --no-pager | tail -20 mountpoint /var/lib/victoria-traces && echo "data disk mounted"

Support

SecureAnyCloud provides 24/7 support for this image. Contact sales@secureanycloud.com.

Deploy on Azure

Launch VictoriaTraces on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure