Grafana Pyroscope 2.2.3

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Grafana Pyroscope 2.2.3 on Ubuntu 24.04 as the software version and choose your Region. Then choose Launch through EC2.

Choose m5.large, your key pair and a security group as described above. Profiles are stored on the root volume under /var/lib/pyroscope/data, so give it room for your retention period (31 days by default).

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=pyroscope}]'

First boot takes a few seconds after the instance is running. Pyroscope reports ready about a minute after it starts.

For Grafana Pyroscope 2.2.3 on Ubuntu 24.04, the SSH user is ubuntu. Connect with your key pair:

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot sets the password and the certificate. Then nginx and Pyroscope start:

systemctl is-active pyroscope nginx pyroscope-firstboot
sudo test -f /var/lib/sac/data/pyroscope-firstboot.done && echo FIRSTBOOT_DONE

All three units are active, and the first-boot marker is present:

active active active FIRSTBOOT_DONE

Only SSH and nginx listen on network ports. Everything Pyroscope opens is on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:4040 users:(("pyroscope")) 127.0.0.1:7946 users:(("pyroscope")) 127.0.0.1:9095 users:(("pyroscope")) 127.0.0.1:9099 users:(("pyroscope")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

Check Pyroscope's own readiness endpoint on the loopback address:

curl -s http://127.0.0.1:4040/ready; echo

ready

sudo cat /root/grafana-pyroscope-credentials.txt

The file is readable by root only:

# Grafana Pyroscope 2.2.3 on Ubuntu 24.04 (sac) - generated on this instance's first boot. # This password is unique to this instance; none was baked into the image. Keep this file secret. PYROSCOPE_URL=https://<public-ip>/ PYROSCOPE_USER=admin PYROSCOPE_PASSWORD=<PYROSCOPE_PASSWORD> # Web UI: https://<host>/ (HTTP basic auth with the user and password above) # Ingest: agents and SDKs push to https://<host>/ with the SAME basic-auth user and password, e.g. Grafana # Alloy: pyroscope.write "sac" { endpoint { url = "https://<host>" # basic_auth { username = "admin" password_file = "/etc/alloy/pyroscope.pass" } } } # Health: https://<host>/healthz (no login; answers "ok") # The certificate is self signed (minted on this instance): accept the browser warning once, give agents # the certificate (/etc/nginx/tls/pyroscope.crt), or install your own. Serve a DNS name instead: # sudo pyroscope-set-hostname <your-dns-name> # Pyroscope listens on 127.0.0.1 only; nginx on 443 is the only way in. Data: /var/lib/pyroscope/data # Self-test:
sudo /usr/local/sbin/pyroscope-selftest.sh
# User guide: https://secureanycloud.com/stacks/grafana-pyroscope-2-2-3/

The URL uses the instance's public IP address from the EC2 instance metadata. The certificate also names the public DNS name and the private IP address, so https://<public-dns-name>/ and https://<private-ip>/ work too.

If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it. To serve Pyroscope under your own DNS name, see Step 9.

A request without the password gets nginx's 401. The health check answers without a password:

curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/
curl -sk https://127.0.0.1/healthz

401 ok

Every way of sending profiles (the Pyroscope SDKs, Grafana Alloy, or a plain HTTP upload) goes to https://<host>/ with the user admin and the password from the credentials file.

To see it work on the instance, upload an example CPU profile for a service called shop-api. Use the folded stack format, with one call stack per line followed by its sample count. Upload it six times, once for each 10-second window of the last minute.

The small shell function pyro passes the password from the credentials file to curl on standard input, so the password never appears on a command line:

pyro() {
sudo sed -n 's/^PYROSCOPE_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/grafana-pyroscope-credentials.txt | curl -sgk -K - "$@"; } cat > /tmp/shop-api.folded <<'EOF' main;net/http.(*conn).serve;shop/api.(*Server).Checkout;shop/db.(*Pool).Query;database/sql.(*DB).QueryContext 910 main;net/http.(*conn).serve;shop/api.(*Server).Checkout;shop/cart.Total;shop/pricing.ApplyDiscounts 420 main;net/http.(*conn).serve;shop/api.(*Server).Checkout;encoding/json.Marshal 260 main;net/http.(*conn).serve;shop/api.(*Server).Search;shop/search.Rank;sort.Slice 640 main;net/http.(*conn).serve;shop/api.(*Server).Search;shop/db.(*Pool).Query;database/sql.(*DB).QueryContext 380 main;net/http.(*conn).serve;shop/api.(*Server).Health 15 runtime.gcBgMarkWorker;runtime.gcDrain;runtime.scanobject 190 EOF
NOW=$(date +%s) for i in 0 1 2 3 4 5; do
FROM=$((NOW - 60 + i * 10)) pyro -o /dev/null -w '%{http_code} ' --data-binary @/tmp/shop-api.folded \
  "https://127.0.0.1/ingest?name=shop-api.cpu{env=demo}&from=${FROM}&until=$((FROM + 10))&format=folded&sampleRate=100&spyName=gospy" done; echo

Each upload is accepted:

200 200 200 200 200 200

The same upload without the password is refused by nginx before it reaches Pyroscope:

curl -sk -o /dev/null -w '%{http_code}\n' --data-binary @/tmp/shop-api.folded "https://127.0.0.1/ingest?name=intruder.cpu&format=folded"

401

Ask Pyroscope which services it has data for. Then read the flamegraph back through the query API:

pyro() {
sudo sed -n 's/^PYROSCOPE_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/grafana-pyroscope-credentials.txt | curl -sgk -K - "$@"; }
NOWMS=$(( $(date +%s) * 1000 )) pyro -X POST -H 'Content-Type: application/json' \
  -d "{\"name\":\"service_name\",\"start\":$((NOWMS - 3600000)),\"end\":${NOWMS}}" \
  https://127.0.0.1/querier.v1.QuerierService/LabelValues; echo

{"names":["shop-api"]}

pyro() {
sudo sed -n 's/^PYROSCOPE_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/grafana-pyroscope-credentials.txt | curl -sgk -K - "$@"; } pyro -G https://127.0.0.1/pyroscope/render \
  --data-urlencode 'query=process_cpu:cpu:nanoseconds:cpu:nanoseconds{service_name="shop-api"}' \
  --data-urlencode 'from=now-15m' --data-urlencode 'until=now' --data-urlencode 'format=json' \
  | jq -r '.flamebearer as $f | "numTicks=\($f.numTicks)", ($f.names[1:] | sort | .[])'

numTicks=168900000000 database/sql.(*DB).QueryContext encoding/json.Marshal main net/http.(*conn).serve runtime.gcBgMarkWorker runtime.gcDrain runtime.scanobject shop/api.(*Server).Checkout shop/api.(*Server).Health shop/api.(*Server).Search shop/cart.Total shop/db.(*Pool).Query shop/pricing.ApplyDiscounts shop/search.Rank sort.Slice

In the browser, open the URL from the credentials file. Your browser warns once about the self-signed certificate. Accept it (or install your own certificate, Step 9), then sign in as admin with the password from the credentials file.

The explore view opens on a service. The timeline at the top shows CPU use over the selected time range. Below it, the top functions table and the flamegraph show where the time went. Here database/sql.(*DB).QueryContext is the widest leaf, so database queries are where shop-api spends most of its CPU.

Switch the view to Flame Graph to see the call tree on its own. Click a frame to zoom into it, and use Search to highlight a function.

The service menu at the top lists every service that has sent profiles. It also lists each service's profile types (CPU, memory allocations, goroutines, locks and so on, depending on what the agent sends).

The example data is yours to delete. It ages out with the retention period. For a clean store right away, stop Pyroscope, empty /var/lib/pyroscope/data and start Pyroscope again.

Point your SDKs or Alloy at https://<host>/, using the private IP or your DNS name, which the certificate names. Use the user admin and the password from the credentials file.

Copy the instance's certificate (/etc/nginx/tls/pyroscope.crt) to the senders so they can verify it. Alternatively, install a certificate that your systems already trust (Step 9).

Grafana Alloy (for example, with pyroscope.ebpf or pyroscope.scrape feeding pyroscope.write.sac.receiver):

pyroscope.write "sac" { endpoint { url = "https://<host>" basic_auth { username = "admin" password_file = "/etc/alloy/pyroscope.pass" } tls_config { ca_file = "/etc/alloy/pyroscope.crt" } } }

Go with github.com/grafana/pyroscope-go:

pyroscope.Start(pyroscope.Config{ ApplicationName: "my-service", ServerAddress: "https://<host>", BasicAuthUser: "admin", BasicAuthPassword: os.Getenv("PYROSCOPE_PASSWORD"), })

Java with the Pyroscope Java agent: set PYROSCOPE_SERVER_ADDRESS=https://<host>, PYROSCOPE_BASIC_AUTH_USER=admin and PYROSCOPE_BASIC_AUTH_PASSWORD in the application's environment. Add the instance's certificate to the JVM trust store.

Keep the password in a secrets store or in a file that only the sender can read, and never in source code. The Pyroscope documentation covers every language SDK and the Alloy components.

To look at the profiles from Grafana, add a Grafana Pyroscope data source with the URL https://<host>. Use basic authentication with the same user and password, and set the instance's certificate as the CA certificate.

Pyroscope runs in single-binary mode: one process with every component, storing profiles on the local disk under /var/lib/pyroscope/data, with no separate database or object store. It keeps profiles for 31 days.

The settings are command-line flags in /etc/systemd/system/pyroscope.service. Change them with a drop-in so that your change survives. For example, set a 14-day retention:

sudo systemctl edit pyroscope
# in the editor, add: [Service] ExecStart= ExecStart=/usr/local/bin/pyroscope <all the flags from pyroscope.service> -retention-period=336h

Then run sudo systemctl restart pyroscope. Keep the 127.0.0.1 address flags as they are.

For long retention or a scale-out cluster, Pyroscope can store profiles in Amazon S3 instead (-storage.backend=s3 with -storage.s3.bucket-name, -storage.s3.endpoint and -storage.s3.region). See the object storage documentation.

pyroscope.service is blocked from the instance metadata service, so it cannot use an instance role. Give it an access key for a bucket-only IAM user through a root-only environment file (EnvironmentFile= in the drop-in, with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY).

Point a DNS record at the instance. An Elastic IP keeps the address stable. Then set the name that Pyroscope is served as.

This command mints a new self-signed certificate for the name. It updates nginx, the list of origins that the web UI may post from, and the URL in /root/grafana-pyroscope-credentials.txt. It also keeps the name across reboots:

sudo pyroscope-set-hostname <your-domain>

To use a certificate from a public or internal certificate authority instead, copy it over the generated pair and reload nginx. An example is a certificate issued by Let's Encrypt for <your-domain>. Running pyroscope-set-hostname again replaces it with a self-signed one:

sudo install -m 0644 <your-domain>.crt /etc/nginx/tls/pyroscope.crt
sudo install -m 0600 <your-domain>.key /etc/nginx/tls/pyroscope.key
sudo nginx -t && sudo systemctl reload nginx

The shipped self-test checks the following:

• The exact set of listening ports, and that every Pyroscope port is on the loopback address
• That an unknown host name is dropped
• That nginx refuses requests without a password, with admin/admin or with a wrong password, while this instance's password is accepted
• That a cross-site post is refused
• That the instance metadata service cannot be reached from inside the pyroscope and nginx services, while it can be reached from your shell

The password is handed to curl in a root-only temporary file, never on a command line:

sudo /usr/local/sbin/pyroscope-selftest.sh

selftest OK: listening tcp lo:4040 lo:53 lo:7946 lo:9095 lo:9099 pub:22 pub:443 pub:80 | pyroscope listeners 127.0.0.1:4040 127.0.0.1:7946 127.0.0.1:9095 127.0.0.1:9099 (loopback only); healthz ok, unknown Host dropped (444); anonymous UI+ingest 401, admin/admin and wrong passwords 401 (nginx auth), per-instance password 200 (UI + query API), cross-origin POST 403; instance metadata blocked inside pyroscope and nginx (000), reachable from the host shell (200)

Add --ingest to also push a tiny profile for a service called sac-selftest and read it back.

The metadata block is a systemd setting on each service. Check it with:

systemctl show -p IPAddressDeny pyroscope nginx pyroscope-firstboot

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128 IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128 IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

A request for a host name that the instance does not serve gets no answer at all. nginx closes the connection, so curl exits with code 52:

curl -sk -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000 curl exit 52

The certificate was generated on this instance for its own addresses:

sudo openssl x509 -in /etc/nginx/tls/pyroscope.crt -noout -subject -ext subjectAltName

subject=CN = <public-ip>, O = sac X509v3 Subject Alternative Name: IP Address:<public-ip>, DNS:<public-dns-name>, IP Address:<private-ip>, DNS:<private-dns-name>, IP Address:127.0.0.1, DNS:localhost

/usr/local/bin/pyroscope --version 2>/dev/null | head -1 nginx -v 2>&1

pyroscope, version 2.2.3 (branch: HEAD, revision: a13c2d994) nginx version: nginx/1.24.0 (Ubuntu)

The key paths on the instance are:

• /usr/local/bin/pyroscope: the Pyroscope binary, run by pyroscope.service as the pyroscope user
• /etc/systemd/system/pyroscope.service: Pyroscope's settings (command-line flags) for loopback addresses, storage and retention
• /var/lib/pyroscope/data: profile storage (segments, blocks and the metadata store)
• /etc/nginx/sites-available/sac-pyroscope: the nginx site, with HTTPS, basic authentication and a proxy to 127.0.0.1:4040
• /etc/nginx/pyroscope.htpasswd: the basic authentication user and password hash (SHA-512 crypt)
• /etc/nginx/tls/: this instance's certificate and private key
• /root/grafana-pyroscope-credentials.txt: the URL, user and password (root only)
• /usr/local/share/pyroscope/LICENSE: the GNU Affero General Public License v3 that Pyroscope is released under

Run these lines in your SSH session. The new password is read at a prompt and reaches openssl on standard input, so it never appears on a command line or in your shell history:

read -rsp 'New password: ' NEWPASS; echo printf 'admin:%s\n' "$(printf '%s\n' "$NEWPASS" | openssl passwd -6 -stdin)" | sudo tee /etc/nginx/pyroscope.htpasswd >/dev/null unset NEWPASS

nginx reads the file on every request, so the new password works at once. Update it on every agent and SDK that sends profiles.

Then update or delete PYROSCOPE_PASSWORD in /root/grafana-pyroscope-credentials.txt (sudo nano /root/grafana-pyroscope-credentials.txt). To give each team or agent its own login, append another user:hash line in the same format.

• Keep port 443 open only to the networks that need the web UI and the agents that send profiles, and port 22 open only to your admin network. Never open 4040, 9095, 7946 or 9099.
• Use your own DNS name and a certificate from a certificate authority that your agents trust (Step 9).
• Keep the password in a secrets store and change it when people leave (Step 12). Profiles reveal function names and code structure, so treat them like source code.
• The instance applies Ubuntu security updates automatically. To update Pyroscope itself, read the release notes and back up /var/lib/pyroscope/data first. Then replace /usr/local/bin/pyroscope with a newer upstream release after checking its checksum against the release's checksums.txt, and restart the service.