Grafana Loki 3.7.9

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Select Grafana Loki 3.7.9 on Ubuntu 24.04 as the software version, choose your Region, and select Launch through EC2. Choose m5.large, your key pair, and a security group as described above. Logs are stored on the root volume under /var/lib/loki, so give it room for the log volume you expect (Step 10).

To launch with the AWS CLI instead, run the following command and replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=loki}]'

Connect over SSH as the user shown below.

Version SSH user

Grafana Loki 3.7.9 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate and the password, then nginx starts in front of Loki:

systemctl is-active loki nginx loki-firstboot
sudo test -f /var/lib/sac/data/loki-firstboot.done && echo FIRSTBOOT_DONE /usr/local/bin/loki -version | head -1

active active active FIRSTBOOT_DONE loki, version 3.7.9 (branch: release-3.7.x, revision: d3b4a2f7)

Only SSH and nginx listen on public addresses. Loki listens on loopback:

sudo ss -Hlntp | awk '{print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:3100 users:(("loki")) 127.0.0.1:9096 users:(("loki")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

The endpoints are in the root-only credentials file:

sudo grep -E '^LOKI_(URL|PUSH_URL|QUERY_URL|USER|CA_CERT)=' /root/loki-credentials.txt

LOKI_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ LOKI_PUSH_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/push LOKI_QUERY_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/query_range LOKI_USER=loki LOKI_CA_CERT=/etc/ssl/loki/loki.crt

The password is the LOKI_PASSWORD line of the same file: sudo grep '^LOKI_PASSWORD=' /root/loki-credentials.txt. Store it in your password manager or secrets store. It is unique to this instance.

The URL uses the instance's public DNS name from EC2 instance metadata (IMDSv2), or its public or private IPv4 address when it has no public name. The certificate names that host plus the instance's public and private IP addresses, so agents can use any of them:

openssl x509 -in /etc/ssl/loki/loki.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = sac X509v3 Subject Alternative Name: DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost notAfter=<date about 27 months after first boot>

Without the password, every path is refused. A wrong password is refused too, and port 80 redirects to HTTPS. With the password, /ready answers. The following commands read the password from the credentials file into a shell variable and pass it to curl through a file descriptor, so it never appears in the process list or your shell history:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s -o /dev/null -w 'http:// -> %{http_code} %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/ready" curl -s -o /dev/null -w 'no login -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" "https://$H/ready" curl -s -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -u loki:not-the-password "https://$H/ready" curl -s -w ' <- HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/ready"

http:// -> 301 https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ready no login -> HTTP 401 wrong password -> HTTP 401 ready <- HTTP 200

The --resolve option sends the request to this instance while keeping the real host name, so the certificate check passes. From another machine, omit --resolve and use the URL directly.

Loki's push API accepts JSON streams: a set of labels and a list of [timestamp in nanoseconds, line] pairs. Push one line, then read it back with a LogQL query:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt
TS=$(date +%s%N)
curl -s -o /dev/null -w 'push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" \
  -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Content-Type: application/json' \
  -d "{\"streams\":[{\"stream\":{\"job\":\"guide-test\",\"host\":\"$(hostname)\"},\"values\":[[\"$TS\",\"hello from the sac Loki guide\"]]}]}" \
  "https://$H/loki/api/v1/push" sleep 2
curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -G \
  --data-urlencode 'query={job="guide-test"}' --data-urlencode 'limit=5' \
  "https://$H/loki/api/v1/query_range" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"]); [print(s["stream"]["job"], v[1]) for s in d["data"]["result"] for v in s["values"]]'

push -> HTTP 204 success guide-test hello from the sac Loki guide

List the labels Loki has seen and the values of one label:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/labels"; echo curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/label/job/values"; echo

{"status":"success","data":["host","job","service_name"]} {"status":"success","data":["guide-test"]}

Any agent that speaks the Loki push API works. Give it the push URL, the user loki, the password, and the instance's certificate so that it can verify the connection. Copy the certificate to each agent host first. The certificate is public, not a secret:

scp -i <your-key.pem> ubuntu@<loki-host>:/etc/ssl/loki/loki.crt ./loki.crt

Grafana Alloy (/etc/alloy/config.alloy on the agent host), shipping the systemd journal:

loki.source.journal "system" { forward_to = [loki.write.sac.receiver] labels = { job = "systemd-journal", host = constants.hostname } } loki.write "sac" { endpoint { url = "https://<loki-host>/loki/api/v1/push" basic_auth { username = "loki" password = "<LOKI_PASSWORD>" } tls_config { ca_file = "/etc/alloy/loki.crt" } } }

Fluent Bit ([OUTPUT] section):

[OUTPUT] Name loki Match * Host <loki-host> Port 443 tls On tls.verify On tls.ca_file /etc/fluent-bit/loki.crt http_user loki http_passwd <LOKI_PASSWORD> labels job=fluent-bit, host=${HOSTNAME}

Keep the password in the agent's own secret store or a root-only file, not in a world-readable configuration. Agents send no browser Origin header, so the image's cross-site request guard does not affect them.

Loki has no user interface of its own, and Grafana is the usual way to explore it. In Grafana, open Connections, Data sources, Add data source, Loki, and set:

• URL: https://<loki-host>/ (the LOKI_URL from Step 4)
• Authentication: Basic authentication, user loki, password LOKI_PASSWORD
• TLS settings: turn on Add self-signed certificate and paste the contents of /etc/ssl/loki/loki.crt as the CA certificate (or install a CA-issued certificate, Step 9)

Choose Save & test, then use Explore with a query such as {job="systemd-journal"} |= "error".

The served host name is fixed by configuration and never taken from a request. To serve Loki under your own DNS name, point the name at the instance and run the helper as root. The helper records the name, regenerates the self-signed certificate for it, updates the credentials file and reloads nginx:

sudo loki-set-hostname <your-dns-name>

If the instance's public DNS name changes after a stop and start, run the same command again, because a request for a name the instance does not serve is dropped. An Elastic IP avoids the change.

To use a certificate from your own CA or a public CA instead, replace /etc/ssl/loki/loki.crt (full chain) and /etc/ssl/loki/loki.key (mode 0600, owner root), then run sudo nginx -t && sudo systemctl reload nginx. Alternatively, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate, and let the load balancer forward to port 443.

Loki writes chunks, the write-ahead log and the TSDB index under /var/lib/loki on the root volume:

sudo du -sh /var/lib/loki df -h /var/lib/loki | tail -1 | awk '{print $2" total, "$4" free on "$6}'

84K /var/lib/loki 19G total, 15G free on /

By default Loki keeps logs until you delete them. To expire old logs automatically, add a retention period to /etc/loki/loki.yaml, for example 31 days, and restart Loki with sudo systemctl restart loki:

compactor: working_directory: /var/lib/loki/compactor retention_enabled: true delete_request_store: filesystem limits_config: retention_period: 744h

As your log volume grows, grow the root EBS volume or attach a dedicated volume mounted at /var/lib/loki. For larger or multi-node deployments, Loki can also store chunks in Amazon S3. See the Loki storage documentation.

The Loki service is sandboxed and may talk only to loopback, so it cannot reach instance metadata or any other address:

systemctl show loki -p User -p IPAddressAllow -p IPAddressDeny -p ProtectSystem -p NoNewPrivileges

IPAddressAllow=127.0.0.0/8 ::1/128 IPAddressDeny=0.0.0.0/0 ::/0 User=loki ProtectSystem=strict NoNewPrivileges=yes

A state-changing request that carries a foreign browser Origin is refused, even with the right password, so a web page cannot push or tail through a signed-in browser:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s -o /dev/null -w 'Origin evil.example push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{"streams":[]}' "https://$H/loki/api/v1/push"

Origin evil.example push -> HTTP 403

Keep these points in mind:

• Loki runs single-tenant (auth_enabled: false), so anyone with the password can push and read every stream. Give each team its own instance if they must not see each other's logs.
• nginx logs the real client address of every request and ignores X-Forwarded-For headers sent by clients.
• The AMI requires IMDSv2. Restrict port 443 to the networks your agents and Grafana use, and never open ports 3100 or 9096.
• The Loki 3.7.9 binary is Grafana's official release. Its Go modules include three advisories that have no upstream fix: GO-2026-5932 (golang.org/x/crypto/openpgp) and GO-2022-0646 and GO-2022-0635 (the AWS SDK's S3 client-side encryption package). Running govulncheck against the v3.7.9 source shows that Loki never calls any of them, so they are not reachable from this server.
• To rotate the password, generate a new one and write its hash to /etc/nginx/loki.htpasswd as loki:<hash> (openssl passwd -6). Then update /root/loki-credentials.txt, reload nginx and update your agents.