Grafana Loki 3.7.1

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Grafana Loki 3.7.1 on Ubuntu 24.04 as the software version and select your Region. Then choose Launch through EC2.

Choose m5.large, your key pair and a security group as described in Prerequisites. Logs are stored on the root volume under /var/lib/loki, so size the volume for the log volume you expect (Step 10).

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=loki}]'

Connect as the ubuntu user, the SSH user for Grafana Loki 3.7.1 on Ubuntu 24.04:

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate and the password, then starts nginx in front of Loki. Confirm that the services are active and that first boot finished:

systemctl is-active loki nginx loki-firstboot
sudo test -f /var/lib/sac/data/loki-firstboot.done && echo FIRSTBOOT_DONE /usr/local/bin/loki -version | head -1

active active active FIRSTBOOT_DONE loki, version 3.7.1 (branch: release-3.7.x, revision: 2c8fff22)

Confirm that only SSH and nginx listen on public addresses and that Loki listens on loopback:

sudo ss -Hlntp | awk '{print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:3100 users:(("loki")) 127.0.0.1:9096 users:(("loki")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

Your endpoints are in the root-only credentials file:

sudo grep -E '^LOKI_(URL|PUSH_URL|QUERY_URL|USER|CA_CERT)=' /root/loki-credentials.txt

LOKI_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ LOKI_PUSH_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/push LOKI_QUERY_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/query_range LOKI_USER=loki LOKI_CA_CERT=/etc/ssl/loki/loki.crt

The password is the LOKI_PASSWORD line of the same file: sudo grep '^LOKI_PASSWORD=' /root/loki-credentials.txt. Store it in your password manager or secrets store. It is unique to this instance.

The URL uses the instance's public DNS name from EC2 instance metadata (IMDSv2). If the instance has no public name, it uses the public or private IPv4 address. The certificate names that host plus the instance's public and private IP addresses, so agents can use any of them:

openssl x509 -in /etc/ssl/loki/loki.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = sac X509v3 Subject Alternative Name: DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost notAfter=<date about 27 months after first boot>

Without the password, every path is refused. A wrong password is refused, and port 80 redirects to HTTPS. With the right password, /ready answers.

The commands below read the password from the credentials file into a shell variable and pass it to curl through a file descriptor, so it never appears in the process list or your shell history:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s -o /dev/null -w 'http:// -> %{http_code} %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/ready" curl -s -o /dev/null -w 'no login -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" "https://$H/ready" curl -s -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -u loki:not-the-password "https://$H/ready" curl -s -w ' <- HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/ready"

http:// -> 301 https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ready no login -> HTTP 401 wrong password -> HTTP 401 ready <- HTTP 200

--resolve sends the request to this instance while keeping the real host name, so the certificate check passes. From another machine, drop --resolve and use the URL directly.

Loki's push API takes JSON streams: a set of labels and a list of [timestamp in nanoseconds, line] pairs. Push one line, then read it back with a LogQL query:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt
TS=$(date +%s%N)
curl -s -o /dev/null -w 'push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" \
  -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Content-Type: application/json' \
  -d "{\"streams\":[{\"stream\":{\"job\":\"guide-test\",\"host\":\"$(hostname)\"},\"values\":[[\"$TS\",\"hello from the sac Loki guide\"]]}]}" \
  "https://$H/loki/api/v1/push" sleep 2
curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -G \
  --data-urlencode 'query={job="guide-test"}' --data-urlencode 'limit=5' \
  "https://$H/loki/api/v1/query_range" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"]); [print(s["stream"]["job"], v[1]) for s in d["data"]["result"] for v in s["values"]]'

push -> HTTP 204 success guide-test hello from the sac Loki guide

List the labels Loki has seen and the values of one label:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/labels"; echo curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/label/job/values"; echo

{"status":"success","data":["host","job","service_name"]} {"status":"success","data":["guide-test"]}

Any agent that speaks the Loki push API works. Give it the push URL, the user loki, the password and the instance's certificate, so it can verify the connection.

Copy the certificate to each agent host first. The certificate is public, not a secret:

scp -i <your-key.pem> ubuntu@<loki-host>:/etc/ssl/loki/loki.crt ./loki.crt

Grafana Alloy (/etc/alloy/config.alloy on the agent host), shipping the systemd journal:

loki.source.journal "system" { forward_to = [loki.write.sac.receiver] labels = { job = "systemd-journal", host = constants.hostname } } loki.write "sac" { endpoint { url = "https://<loki-host>/loki/api/v1/push" basic_auth { username = "loki" password = "<LOKI_PASSWORD>" } tls_config { ca_file = "/etc/alloy/loki.crt" } } }

Fluent Bit ([OUTPUT] section):

[OUTPUT] Name loki Match * Host <loki-host> Port 443 tls On tls.verify On tls.ca_file /etc/fluent-bit/loki.crt http_user loki http_passwd <LOKI_PASSWORD> labels job=fluent-bit, host=${HOSTNAME}

Keep the password in the agent's own secret store or a root-only file, not in a world-readable configuration. Agents send no browser Origin header, so the image's cross-site request guard does not affect them.

Loki has no user interface of its own, and Grafana is the usual way to explore it. In Grafana, open Connections, Data sources, Add data source, Loki, and set:

• URL: https://<loki-host>/ (the LOKI_URL from Step 4)
• Authentication: Basic authentication, user loki, password LOKI_PASSWORD
• TLS settings: turn on Add self-signed certificate and paste the contents of /etc/ssl/loki/loki.crt as the CA certificate. If you installed a CA-issued certificate (Step 9), you do not need this

Choose Save & test, then open Explore and run a query such as {job="systemd-journal"} |= "error".

The served host name is fixed by configuration and never taken from a request. To serve Loki under your own DNS name, point the name at the instance and run the helper as root. It records the name, regenerates the self-signed certificate for it, updates the credentials file and reloads nginx:

sudo loki-set-hostname <your-dns-name>

If the instance's public DNS name changes after a stop and start, run the same command again, because a request for a name the instance does not serve is dropped. An Elastic IP avoids this.

To use a certificate from your own CA or a public CA instead, replace /etc/ssl/loki/loki.crt (full chain) and /etc/ssl/loki/loki.key (mode 0600, owner root). Then run sudo nginx -t && sudo systemctl reload nginx.

Alternatively, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate and let the load balancer forward to port 443.

Loki writes chunks, the write-ahead log and the TSDB index under /var/lib/loki on the root volume. Check how much space they use:

sudo du -sh /var/lib/loki df -h /var/lib/loki | tail -1 | awk '{print $2" total, "$4" free on "$6}'

84K /var/lib/loki 19G total, 15G free on /

By default, Loki keeps logs until you delete them. To expire old logs automatically, add a retention period to /etc/loki/loki.yaml, for example 31 days. Then restart Loki with sudo systemctl restart loki:

compactor: working_directory: /var/lib/loki/compactor retention_enabled: true delete_request_store: filesystem limits_config: retention_period: 744h

As your log volume grows, grow the root EBS volume or attach a dedicated volume mounted at /var/lib/loki. For larger or multi-node deployments, Loki can also store chunks in Amazon S3. See the Loki storage documentation.

The Loki service is sandboxed and may talk only to loopback, so it cannot reach instance metadata or any other address:

systemctl show loki -p User -p IPAddressAllow -p IPAddressDeny -p ProtectSystem -p NoNewPrivileges

IPAddressAllow=127.0.0.0/8 ::1/128 IPAddressDeny=0.0.0.0/0 ::/0 User=loki ProtectSystem=strict NoNewPrivileges=yes

A state-changing request that carries a foreign browser Origin is refused, even with the right password, so a web page cannot push or tail through a signed-in browser:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt) CA=/etc/ssl/loki/loki.crt curl -s -o /dev/null -w 'Origin evil.example push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{"streams":[]}' "https://$H/loki/api/v1/push"

Origin evil.example push -> HTTP 403

Keep these points in mind:

• Single tenant. Loki runs with auth_enabled: false, so anyone with the password can push and read every stream. Give each team its own instance if they must not see each other's logs.
• Real client addresses. nginx logs the real client address of every request and ignores X-Forwarded-For headers sent by clients.
• Restricted network access. The AMI requires IMDSv2. Restrict port 443 to the networks your agents and Grafana use, and never open 3100 or 9096.
• Password rotation. Generate a new password and write its hash to /etc/nginx/loki.htpasswd as loki:<hash> (openssl passwd -6). Update /root/loki-credentials.txt, reload nginx and update your agents.