VictoriaTraces 0.13.0

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick VictoriaTraces 0.13.0 on Ubuntu 24.04 as the software version, select your Region, and choose Launch through EC2. Select m5.large, your key pair and a security group as above. The AMI brings its own 32 GiB data volume for the spans. Increase its size on the launch page if you expect a high span rate (Step 9).

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=victoriatraces}]'

Version SSH user

VictoriaTraces 0.13.0 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

The login message shows the web UI address, the OTLP ingest address and where the password is.

First boot takes a few seconds once the instance is up. It creates the certificate and the password, then starts VictoriaTraces and nginx:

systemctl is-active victoriatraces nginx
sudo test -f /var/lib/sac/data/victoriatraces-firstboot.done && echo FIRSTBOOT_DONE /usr/local/bin/victoria-traces-prod -version

active active FIRSTBOOT_DONE victoria-traces-20261105-063140-tags-v0.13.0-0-gf0295d1f7-sac-pure-compress-v1.20.2

The build record names the source archive and its checksum, the toolchain, the one dependency SecureAnyCloud raised and the checksum of the binary, next to the Apache-2.0 licence:

grep -E '^(VictoriaTraces|source|toolchain|dependency raise|binary)' /usr/share/doc/victoriatraces/BUILD-INFO.txt | cut -c1-150 head -3 /usr/share/doc/victoriatraces/LICENSE | sed '/^\s*$/d'

VictoriaTraces 0.13.0 (sac source rebuild) source https://github.com/VictoriaMetrics/VictoriaTraces/archive/refs/tags/v0.13.0.tar.gz sha256 3cc632ac7eb018082b3ddfe217fd65479405614f17bf5ad75e8bc toolchain go1.27.3 linux/amd64 (sha256 65efbc80fdc1fa2cd2631cad5b481777a68e8725f34e89f50fc6ac714e379b40); CGO_ENABLED=0 GOAMD64=v1 -trimpath -buildvcs dependency raise (d-320, govulncheck GO-2026-6865): github.com/klauspost/compress@v1.20.2 -> go.mod sha256 91437c6783a8d4f82c4c67f4f441dc83cb7e8bf6059 binary /usr/local/bin/victoria-traces-prod sha256 9d8eda317d1de6116875225427c7ea757eac8a20d8e500344818282b3f38737a (reproducible); go version -m: go-b Apache License Version 2.0, January 2004 http://www.apache.org/licenses/

Only SSH and nginx listen on the network. VictoriaTraces listens on its Unix socket, which belongs to the victoriatraces user and the www-data group that nginx runs as:

sudo ss -Hlntp | awk '{print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort
sudo ls -l /run/victoriatraces/

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx")) total 0 srwxrwx--- 1 victoriatraces www-data 0 Nov 7 16:52 victoriatraces.sock

The addresses and the user name are in the root-only credentials file:

sudo grep -E '^VICTORIATRACES_(URL|OTLP_ENDPOINT|USER)=' /root/victoriatraces-credentials.txt

VICTORIATRACES_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/select/vmui/ VICTORIATRACES_OTLP_ENDPOINT=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/insert/opentelemetry VICTORIATRACES_USER=admin

The same file also holds:

VICTORIATRACES_TLS_SHA256=<the SHA-256 fingerprint of this instance's certificate> VICTORIATRACES_PASS=<40 random hexadecimal characters>

• VICTORIATRACES_URL is the trace explorer.
• VICTORIATRACES_OTLP_ENDPOINT is the OpenTelemetry base endpoint. Spans go to it with /v1/traces appended.
• VICTORIATRACES_USER / VICTORIATRACES_PASS sign in to everything. Store the password in your password manager or secrets store, because it is unique to this instance. Read it with sudo cat /root/victoriatraces-credentials.txt.
• VICTORIATRACES_TLS_SHA256 lets you check the certificate that your browser or collector sees.

The addresses use the instance's public DNS name from cloud-init, or its public or private IPv4 address when it has no public name. The certificate names that host plus the instance's public and private addresses:

openssl x509 -in /etc/ssl/victoriatraces/victoriatraces.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = sac X509v3 Subject Alternative Name: DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, IP Address:<private-ip>, IP Address:127.0.0.1 notAfter=<date about 27 months after first boot>

The image includes an end-to-end test. It checks VictoriaTraces through its socket and confirms that every route refuses requests without the password or with a wrong one. It also confirms that the maintenance endpoints and a cross-site ingest are refused. Then it sends an OTLP span over HTTPS with the password and reads it back through the Jaeger API:

sudo victoriatraces-selftest

VictoriaTraces self-test passed: health via the socket; build 0.13.0; every route 401 without / with a wrong credential; /internal and /debug 403; cross-origin ingest 403; OTLP trace <trace-id> stored and read back over pinned HTTPS with its operation and attribute; web UI served VICTORIATRACES_SELFTEST_OK

Each run stores one small span under a service named sac-selftest-<time>, which ages out with the retention period.

Without the password, every path is refused. A wrong password is also refused, and port 80 redirects to HTTPS. 127.0.0.1:80/healthz answers on the instance itself, without a login, for local monitoring. The following commands read the password from the credentials file into a shell variable and hand it to curl through a file descriptor, so it never appears in the process list or your shell history:

H=$(sudo sed -n 's#^VICTORIATRACES_URL=https://\([^/]*\)/.*#\1#p' /root/victoriatraces-credentials.txt)
PW=$(sudo sed -n 's/^VICTORIATRACES_PASS=//p' /root/victoriatraces-credentials.txt) CA=/etc/ssl/victoriatraces/victoriatraces.crt
IP=$(hostname -I | cut -d' ' -f1)
curl -s -o /dev/null -w 'http:// -> HTTP %{http_code}\n' --resolve "$H:80:$IP" "http://$H/select/vmui/"
curl -s -o /dev/null -w 'no password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" "https://$H/select/vmui/"
curl -s -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "admin:%s"\n' "$(openssl rand -hex 20)") "https://$H/select/vmui/"
curl -s -o /dev/null -w 'admin + password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "admin:%s"\n' "$PW") "https://$H/select/vmui/"
curl -s -o /dev/null -w 'OTLP ingest, no password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' -d '{}' "https://$H/insert/opentelemetry/v1/traces"
curl -s -w ' <- loopback /healthz HTTP %{http_code}\n' http://127.0.0.1/healthz

http:// -> HTTP 301 no password -> HTTP 401 wrong password -> HTTP 401 admin + password -> HTTP 200 OTLP ingest, no password -> HTTP 401 OK <- loopback /healthz HTTP 200

--resolve sends the request to this instance while keeping the real host name, so the certificate check passes. From another machine, drop --resolve and use the address directly.

OpenTelemetry SDKs and collectors send spans to VICTORIATRACES_OTLP_ENDPOINT + /v1/traces as protobuf or JSON. The following block builds one trace by hand, with three spans from three services in the OTLP JSON format. It sends the trace, then reads it back with the Jaeger API:

H=$(sudo sed -n 's#^VICTORIATRACES_URL=https://\([^/]*\)/.*#\1#p' /root/victoriatraces-credentials.txt)
PW=$(sudo sed -n 's/^VICTORIATRACES_PASS=//p' /root/victoriatraces-credentials.txt) CA=/etc/ssl/victoriatraces/victoriatraces.crt
TID=$(openssl rand -hex 16);
NOW=$(date +%s) span() {
# service span-id parent-id name start-offset-ms duration-ms printf '{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"%s"}}]},"scopeSpans":[{"scope":{"name":"guide"},"spans":[{"traceId":"%s","spanId":"%s","parentSpanId":"%s","name":"%s","kind":2,"startTimeUnixNano":"%s","endTimeUnixNano":"%s","attributes":[{"key":"http.route","value":{"stringValue":"/api/checkout"}}],"status":{}}]}]}' \
  "$1" "$TID" "$2" "$3" "$4" "$(( NOW*1000000000 + $5*1000000 ))" "$(( NOW*1000000000 + ($5+$6)*1000000 ))" } printf '{"resourceSpans":[%s,%s,%s]}' \
  "$(span shop-frontend 1111111111111111 '' 'POST /api/checkout' 0 180)" \
  "$(span checkout-api 2222222222222222 1111111111111111 'create order' 15 140)" \
  "$(span orders-db 3333333333333333 2222222222222222 'INSERT orders' 40 60)" | curl -s -o /dev/null -w 'OTLP ingest -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" \
  -K <(printf 'user = "admin:%s"\n' "$PW") -H 'Content-Type: application/json' --data-binary @- \
  "https://$H/insert/opentelemetry/v1/traces" for i in $(seq 1 30); do
T=$(curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "admin:%s"\n' "$PW") "https://$H/select/jaeger/api/traces/$TID") [ "$(echo "$T" | jq '.data[0].spans | length' 2>/dev/null)" = 3 ] && break; sleep 2 done echo "$T" | jq -r '.data[0] as $t | $t.spans | sort_by(.startTime)[] | "\($t.processes[.processID].serviceName) \(.operationName) \(.duration/1000) ms"' curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "admin:%s"\n' "$PW") "https://$H/select/jaeger/api/services" | jq -c '.data | map(select(startswith("sac") | not)) | sort'

OTLP ingest -> HTTP 200 shop-frontend POST /api/checkout 180 ms checkout-api create order 140 ms orders-db INSERT orders 60 ms ["checkout-api","orders-db","shop-frontend"]

New spans become searchable within a few seconds. The Jaeger query API is at /select/jaeger/api/ (services, operations, trace search and trace by ID) and the Tempo API is at /select/tempo/api/. Both need the same password. The VictoriaTraces documentation describes every endpoint and the LogsQL query language that the explorer uses.

Open VICTORIATRACES_URL from Step 4 (https://<your-host>/select/vmui/) in a browser. The certificate is self-signed, so the browser warns once. Compare the fingerprint it shows with VICTORIATRACES_TLS_SHA256 and accept it, or install your own certificate (Step 10). Sign in as admin with the password from the credentials file.

The Trace Explorer lists the traces in the selected time range with their root service, operation, duration, span count and error count, above a duration scatter chart. Use the filters on the left to narrow the list by service, operation, duration, span kind and tags:

The search box takes a LogsQL query over span fields. The query status_code:2 finds spans that ended with an error:

Click a trace to open its preview, which shows the services involved, the critical span and every error:

View full trace shows the span waterfall with the fields of the selected span:

Any OpenTelemetry SDK, agent or collector that exports OTLP over HTTP works. Give it the endpoint, the user admin, the password, and the certificate of the instance so it can verify the connection. Copy the certificate to each host first. It is public, not a secret:

scp -i <your-key.pem> ubuntu@<victoriatraces-host>:/etc/ssl/victoriatraces/victoriatraces.crt ./victoriatraces.crt

OpenTelemetry SDKs (Java, Python, Node.js, Go, .NET and others) read the standard environment variables. The Authorization value is Basic followed by the base64 encoding of admin:<password>:

OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<victoriatraces-host>/insert/opentelemetry/v1/traces OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic%20<base64 of admin:password> OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE=/etc/ssl/certs/victoriatraces.crt OTEL_SERVICE_NAME=<your-service-name>

OpenTelemetry Collector (config.yaml), forwarding the spans it receives:

extensions: basicauth/victoriatraces: client_auth: username: admin password: ${env:VICTORIATRACES_PASS} exporters: otlphttp/victoriatraces: traces_endpoint: https://<victoriatraces-host>/insert/opentelemetry/v1/traces auth: authenticator: basicauth/victoriatraces tls: ca_file: /etc/otelcol/victoriatraces.crt service: extensions: [basicauth/victoriatraces] pipelines: traces: receivers: [otlp] exporters: [otlphttp/victoriatraces]

Keep the password in the environment of the collector or in a root-only file, never in a world-readable configuration. SDKs and collectors send no browser Origin header, so the cross-site request guard of the image does not affect them. OTLP over gRPC (port 4317) is not served. Use OTLP over HTTP, or run a collector next to your applications that receives gRPC and exports over HTTP as above.

Grafana: add a Jaeger data source with the URL https://<victoriatraces-host>/select/jaeger. Turn on Basic auth with the user admin and the password, and add the instance certificate under With CA Cert (or a CA-issued certificate, Step 10).

VictoriaTraces keeps all span data on the dedicated EBS volume mounted at /var/lib/victoria-traces, listed in /etc/fstab by its file system UUID:

sudo du -sh /var/lib/victoria-traces df -h /var/lib/victoria-traces | tail -1 | awk '{print $2" total, "$4" free on "$6}' grep victoria-traces /etc/fstab | sed -E 's/UUID=[0-9a-f-]+/UUID=<uuid>/' systemctl show -p ExecStart victoriatraces.service | tr ' ' '\n' | grep -E '^-(retention|storage)'

380K /var/lib/victoria-traces 32G total, 30G free on /var/lib/victoria-traces UUID=<uuid> /var/lib/victoria-traces ext4 defaults,nofail 0 2 -storageDataPath=/var/lib/victoria-traces -retentionPeriod=7d -retention.maxDiskUsagePercent=80

Spans older than 7 days are deleted automatically, and VictoriaTraces stops accepting new spans before the volume is more than 80% full, so the disk never fills. To keep spans longer, change -retentionPeriod (for example -retentionPeriod=30d) in /etc/systemd/system/victoriatraces.service. Then run sudo systemctl daemon-reload && sudo systemctl restart victoriatraces. To grow the volume, modify it in the EC2 console (Volumes, Modify volume), then extend the file system online:

sudo resize2fs "$(findmnt -no SOURCE /var/lib/victoria-traces)"

Take EBS snapshots of the data volume for backups.

Your own DNS name. The served host name is fixed by configuration, never taken from a request. To serve VictoriaTraces under your own name, point the name at the instance and run the helper. The helper records the name, creates a new certificate for it, updates the credentials file and reloads nginx:

sudo victoriatraces-set-hostname <your-domain>

Run the same command after a stop and start if the public DNS name of the instance changed (an Elastic IP avoids that), because a request for a name the instance does not serve is dropped. To use a certificate from your own CA or a public CA instead, replace /etc/ssl/victoriatraces/victoriatraces.crt (full chain) and /etc/ssl/victoriatraces/victoriatraces.key (mode 0600, owner root). Then run sudo nginx -t && sudo systemctl reload nginx.

To change the password, run the following command on the instance and type the new password (at least 16 characters, no colon) at the prompt. The password is read from the terminal, never from the command line, and nginx stores only its bcrypt hash:

# Type <new-password> at the prompt; nothing is echoed. read -rs -p 'New VictoriaTraces password: ' P; echo printf '%s\n' "$P" | sudo victoriatraces-set-password admin; unset P

Then update every SDK and collector that sends spans. Once you have stored the password somewhere safe, you can delete the VICTORIATRACES_PASS line from /root/victoriatraces-credentials.txt.

Every request on port 443 also passes the access list in /etc/nginx/sac/victoriatraces-access.conf. By default, the list is allow all; (every request still needs the password). To allow only your own networks, replace it with your addresses and a final deny all;:

printf 'allow <your-mgmt-cidr>;\nallow 127.0.0.1;\ndeny all;\n' | sudo tee /etc/nginx/sac/victoriatraces-access.conf
sudo nginx -t && sudo systemctl reload nginx

VictoriaTraces runs as its own unprivileged user, may only open Unix sockets, and sees a read-only system. Like nginx and the first boot service, it is denied the instance metadata endpoint:

systemctl show victoriatraces -p User -p NoNewPrivileges -p ProtectSystem -p ProtectHome -p RestrictAddressFamilies -p IPAddressDeny

IPAddressDeny=fd00:ec2::254/128 168.63.129.16/32 ::ffff:169.254.169.254/128 169.254.169.254/32 User=victoriatraces ProtectHome=yes ProtectSystem=strict NoNewPrivileges=yes RestrictAddressFamilies=AF_UNIX

The storage maintenance and profiling endpoints are refused even with the password. An ingest that carries a foreign browser Origin is also refused, so a web page cannot write spans through a signed-in browser:

H=$(sudo sed -n 's#^VICTORIATRACES_URL=https://\([^/]*\)/.*#\1#p' /root/victoriatraces-credentials.txt)
PW=$(sudo sed -n 's/^VICTORIATRACES_PASS=//p' /root/victoriatraces-credentials.txt) CA=/etc/ssl/victoriatraces/victoriatraces.crt for p in /internal/force_merge /internal/force_flush /debug/pprof/; do curl -s -o /dev/null -w "$p with the password -> HTTP %{http_code}\n" --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "admin:%s"\n' "$PW") "https://$H$p" done curl -s -o /dev/null -w 'cross-origin ingest with the password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "admin:%s"\n' "$PW") -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{"resourceSpans":[]}' "https://$H/insert/opentelemetry/v1/traces"

/internal/force_merge with the password -> HTTP 403 /internal/force_flush with the password -> HTTP 403 /debug/pprof/ with the password -> HTTP 403 cross-origin ingest with the password -> HTTP 403

SSH accepts keys only. Passwords, root logins and X11 forwarding are switched off in /etc/ssh/sshd_config.d/00-sac.conf, which sorts before every other SSH setting:

sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|x11forwarding|pubkeyauthentication) '

permitrootlogin no pubkeyauthentication yes passwordauthentication no kbdinteractiveauthentication no x11forwarding no

Other points:

• VictoriaTraces has a single account. Anyone with the password can send spans and read every trace, so give each team its own instance if they must not see each other's traces.
• Span deletion is switched off (-delete.enable is not set), and CORS is off, so no other web site can read from VictoriaTraces through a browser.
• nginx logs the real client address of every request. It ignores X-Forwarded-For headers sent by clients, and it never passes the Authorization header to VictoriaTraces.
• The AMI requires IMDSv2. Restrict port 443 to the networks your users, applications and collectors use.