Grafana Loki 3.7.8

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Select Grafana Loki 3.7.8 on Ubuntu 24.04 as the software version, choose your Region, and select Launch through EC2.

Choose m5.large, your key pair and a security group as described above. Logs are stored on the root volume under /var/lib/loki, so give the volume room for the log volume you expect (Step 10).

Alternatively, launch with the AWS CLI. Replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=loki}]'

Version SSH user

Grafana Loki 3.7.8 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate and the password, and then nginx starts in front of Loki:

systemctl is-active loki nginx loki-firstboot
sudo test -f /var/lib/sac/data/loki-firstboot.done && echo FIRSTBOOT_DONE /usr/local/bin/loki -version | head -1

active active active FIRSTBOOT_DONE loki, version 3.7.8 (branch: release-3.7.x, revision: 09e6ce2f)

Only SSH and nginx listen on public addresses. Loki listens on loopback:

sudo ss -Hlntp | awk '{print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:3100 users:(("loki")) 127.0.0.1:9096 users:(("loki")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

The endpoints are in the root-only credentials file:

sudo grep -E '^LOKI_(URL|PUSH_URL|QUERY_URL|USER|CA_CERT)=' /root/loki-credentials.txt

LOKI_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ LOKI_PUSH_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/push LOKI_QUERY_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/query_range LOKI_USER=loki LOKI_CA_CERT=/etc/ssl/loki/loki.crt

The password is the LOKI_PASSWORD line of the same file: sudo grep '^LOKI_PASSWORD=' /root/loki-credentials.txt. Store it in your password manager or secrets store. It is unique to this instance.

The URL uses the instance's public DNS name from EC2 instance metadata (IMDSv2), or its public or private IPv4 address when it has no public name. The certificate names that host plus the instance's public and private IP addresses, so agents can use any of them:

openssl x509 -in /etc/ssl/loki/loki.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = sac X509v3 Subject Alternative Name: DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost notAfter=<date about 27 months after first boot>

Without the password, every path is refused. A wrong password is also refused, and port 80 redirects to HTTPS. With the correct password, /ready answers.

The commands below read the password from the credentials file into a shell variable and pass it to curl through a file descriptor, so it never appears in the process list or your shell history:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s -o /dev/null -w 'http:// -> %{http_code} %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/ready" curl -s -o /dev/null -w 'no login -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" "https://$H/ready" curl -s -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -u loki:not-the-password "https://$H/ready" curl -s -w ' <- HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/ready"

http:// -> 301 https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ready no login -> HTTP 401 wrong password -> HTTP 401 ready <- HTTP 200

--resolve sends the request to this instance while keeping the real host name, so the certificate check passes. From another machine, omit --resolve and use the URL directly.

Loki's push API takes JSON streams: a set of labels and a list of [timestamp in nanoseconds, line] pairs. Push one line, and then read it back with a LogQL query:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt
TS=$(date +%s%N)
curl -s -o /dev/null -w 'push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" \
  -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Content-Type: application/json' \
  -d "{\"streams\":[{\"stream\":{\"job\":\"guide-test\",\"host\":\"$(hostname)\"},\"values\":[[\"$TS\",\"hello from the sac Loki guide\"]]}]}" \
  "https://$H/loki/api/v1/push" sleep 2
curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -G \
  --data-urlencode 'query={job="guide-test"}' --data-urlencode 'limit=5' \
  "https://$H/loki/api/v1/query_range" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"]); [print(s["stream"]["job"], v[1]) for s in d["data"]["result"] for v in s["values"]]'

push -> HTTP 204 success guide-test hello from the sac Loki guide

List the labels Loki has seen and the values of one label:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/labels"; echo curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/label/job/values"; echo

{"status":"success","data":["host","job","service_name"]} {"status":"success","data":["guide-test"]}

Any agent that speaks the Loki push API works. Give it the push URL, the user loki, the password, and the instance's certificate so that it can verify the connection.

First, copy the certificate to each agent host. The certificate is public, not a secret:

scp -i <your-key.pem> ubuntu@<loki-host>:/etc/ssl/loki/loki.crt ./loki.crt

Grafana Alloy (/etc/alloy/config.alloy on the agent host), shipping the systemd journal:

loki.source.journal "system" { forward_to = [loki.write.sac.receiver] labels = { job = "systemd-journal", host = constants.hostname } } loki.write "sac" { endpoint { url = "https://<loki-host>/loki/api/v1/push" basic_auth { username = "loki" password = "<LOKI_PASSWORD>" } tls_config { ca_file = "/etc/alloy/loki.crt" } } }

Fluent Bit ([OUTPUT] section):

[OUTPUT] Name loki Match * Host <loki-host> Port 443 tls On tls.verify On tls.ca_file /etc/fluent-bit/loki.crt http_user loki http_passwd <LOKI_PASSWORD> labels job=fluent-bit, host=${HOSTNAME}

Keep the password in the agent's own secret store or a root-only file, rather than in a world-readable configuration. Agents send no browser Origin header, so the image's cross-site request guard does not affect them.

Loki has no user interface of its own, and Grafana is the usual way to explore it. In Grafana, open Connections, Data sources, Add data source, Loki, and set the following:

• URL: https://<loki-host>/ (the LOKI_URL from Step 4)
• Authentication: Basic authentication, user loki, password LOKI_PASSWORD
• TLS settings: turn on Add self-signed certificate and paste the contents of /etc/ssl/loki/loki.crt as the CA certificate (or install a CA-issued certificate, Step 9)

Click Save & test, and then use Explore with a query such as {job="systemd-journal"} |= "error".

The served host name is fixed by configuration and is never taken from a request. To serve Loki under your own DNS name, point the name at the instance and run the helper as root. The helper records the name, regenerates the self-signed certificate for it, updates the credentials file and reloads nginx:

sudo loki-set-hostname <your-dns-name>

If the instance's public DNS name changes after a stop and start, run the same command again, because a request for a name that the instance does not serve is dropped. An Elastic IP avoids this.

To use a certificate from your own CA or a public CA instead, replace /etc/ssl/loki/loki.crt (full chain) and /etc/ssl/loki/loki.key (mode 0600, owner root). Then run sudo nginx -t && sudo systemctl reload nginx.

Alternatively, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate and let the load balancer forward to port 443.

Loki writes chunks, the write-ahead log and the TSDB index under /var/lib/loki on the root volume:

sudo du -sh /var/lib/loki df -h /var/lib/loki | tail -1 | awk '{print $2" total, "$4" free on "$6}'

84K /var/lib/loki 19G total, 15G free on /

By default, Loki keeps logs until you delete them. To expire old logs automatically, add a retention period to /etc/loki/loki.yaml, for example 31 days. Then restart Loki with sudo systemctl restart loki:

compactor: working_directory: /var/lib/loki/compactor retention_enabled: true delete_request_store: filesystem limits_config: retention_period: 744h

Grow the root EBS volume (or attach a dedicated volume mounted at /var/lib/loki) as your log volume grows. For larger or multi-node deployments, Loki can also store chunks in Amazon S3. See the Loki storage documentation.

The Loki service is sandboxed and may talk only to loopback, so it cannot reach instance metadata or any other address:

systemctl show loki -p User -p IPAddressAllow -p IPAddressDeny -p ProtectSystem -p NoNewPrivileges

IPAddressAllow=127.0.0.0/8 ::1/128 IPAddressDeny=0.0.0.0/0 ::/0 User=loki ProtectSystem=strict NoNewPrivileges=yes

A state-changing request that carries a foreign browser Origin is refused, even with the right password. A web page therefore cannot push or tail through a signed-in browser:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s -o /dev/null -w 'Origin evil.example push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{"streams":[]}' "https://$H/loki/api/v1/push"

Origin evil.example push -> HTTP 403

Other points:

• Loki runs single-tenant (auth_enabled: false), so anyone with the password can push and read every stream. Give each team its own instance if they must not see each other's logs.
• nginx logs the real client address of every request and ignores X-Forwarded-For headers sent by clients.
• The AMI requires IMDSv2. Restrict port 443 to the networks that your agents and Grafana use, and do not open ports 3100 or 9096.
• To rotate the password, generate a new one, write its hash to /etc/nginx/loki.htpasswd as loki:<hash> (openssl passwd -6), update /root/loki-credentials.txt, reload nginx, and update your agents.