bazel-remote 2.6.3

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick bazel-remote 2.6.3 on Ubuntu 24.04 as the software version, choose your Region, then choose Launch through EC2.

Choose m5.large, your key pair, and a security group as described above. The AMI carries the 20 GiB cache volume. On the launch page, set it to the size your cache should be, and first boot uses all of it. You can also grow it later (Step 9).

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=bazel-remote}]'

Version SSH user

bazel-remote 2.6.3 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate, sizes the cache and generates the password. Then bazel-remote and nginx start:

systemctl is-active bazel-remote nginx
sudo test -f /var/lib/sac/data/bazel-remote-firstboot.done && echo FIRSTBOOT_DONE

active active FIRSTBOOT_DONE

Only SSH and nginx listen on the network. bazel-remote's HTTP (8080) and gRPC (9092) listeners are on loopback behind nginx, and the DNS resolver listens on loopback only:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | sed -E 's/pid=[0-9]+,fd=[0-9]+/.../g' | sort

0.0.0.0:22 users:(("sshd",...),("systemd",...)) 0.0.0.0:443 users:(("nginx",...),("nginx",...),("nginx",...)) 127.0.0.1:8080 users:(("bazel-remote",...)) 127.0.0.1:9092 users:(("bazel-remote",...)) 127.0.0.53%lo:53 users:(("systemd-resolve",...)) 127.0.0.54:53 users:(("systemd-resolve",...)) [::]:22 users:(("sshd",...),("systemd",...)) [::]:443 users:(("nginx",...),("nginx",...),("nginx",...))

Only root can read the credentials file. The commands below print it with the password hidden. Run sudo cat /root/bazel-remote-info.txt to show the password itself.

sudo sed -E 's/^(BAZEL_REMOTE_PASSWORD=).*/\1<hidden>/' /root/bazel-remote-info.txt

# bazel-remote 2.6.3 - generated on first boot by bazel-remote-firstboot.service
# This password is unique to this instance; none was baked into the image. bazel-remote keeps only a bcrypt hash
# of it in /etc/bazel-remote/htpasswd. Every request (reads too) needs it, over HTTPS and gRPC. Keep this file secret. BAZEL_REMOTE_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ BAZEL_REMOTE_GRPC_URL=grpcs://ec2-<public-ip-dashed>.compute-1.amazonaws.com BAZEL_REMOTE_USER=bazel BAZEL_REMOTE_PASSWORD=<hidden> BAZEL_REMOTE_CA_CERT=/etc/nginx/tls/bazel-remote.crt BAZEL_REMOTE_CERT_SHA256=BB:33:29:65:3B:D3:AB:36:57:4C:5D:DF:C1:9D:0F:19:85:D5:F6:32:1E:9C:D9:D8:58:93:47:32:C2:14:AF:B9 BAZEL_REMOTE_MAX_SIZE_GIB=15
#
# On a Bazel client, with a copy of the certificate above (bazel-remote.crt), in a .bazelrc you keep private:
# build --remote_cache=grpcs://ec2-<public-ip-dashed>.compute-1.amazonaws.com
# build "--remote_header=Authorization=Basic <base64 of bazel:<BAZEL_REMOTE_PASSWORD>>"
# build --tls_certificate=/path/to/bazel-remote.crt
# Change the password:
sudo bazel-remote-set-password
# Add a user:
sudo bazel-remote-set-password --user <name>
# Change the name it serves as:
sudo bazel-remote-set-hostname <your-dns-name>
# User guide: https://secureanycloud.com/stacks/bazel-remote-2-6-3/

The URLs use the instance's public DNS name from EC2 instance metadata, which resolves to the private address inside your VPC. The certificate also names the public and private IP addresses, so grpcs://<private-ip> works too.

BAZEL_REMOTE_CERT_SHA256 is the certificate's fingerprint. Compare it against the copy you give your build machines. To serve the cache under your own DNS name, see Step 8.

The HTTP cache stores a file under its SHA-256 at /cas/<sha256>. Upload a small file and read it back. The password is handed to curl on standard input, so it never appears in the process list.

sudo bash -c ' P=$(grep "^BAZEL_REMOTE_PASSWORD=" /root/bazel-remote-info.txt | cut -d= -f2-) auth() { printf "user = \"bazel:%s\"\n" "$P"; } C=/etc/nginx/tls/bazel-remote.crt echo "hello from the sac bazel-remote guide" > /tmp/blob.txt H=$(sha256sum /tmp/blob.txt | cut -d" " -f1) auth | curl -K - -s --cacert $C -o /dev/null -w "upload PUT /cas/<sha256> -> HTTP %{http_code}\n" -X PUT --data-binary @/tmp/blob.txt https://127.0.0.1/cas/$H auth | curl -K - -s --cacert $C https://127.0.0.1/cas/$H rm -f /tmp/blob.txt'

upload PUT /cas/<sha256> -> HTTP 200 hello from the sac bazel-remote guide

The /status endpoint also needs the password. It reports the cache's size and the running release:

sudo bash -c 'P=$(grep "^BAZEL_REMOTE_PASSWORD=" /root/bazel-remote-info.txt | cut -d= -f2-); printf "user = \"bazel:%s\"\n" "$P" | curl -K - -s --cacert /etc/nginx/tls/bazel-remote.crt https://127.0.0.1/status' | python3 -m json.tool

{ "CurrSize": 110592, "UncompressedSize": 106496, "ReservedSize": 0, "MaxSize": 16106127360, "NumFiles": 11, "ServerTime": 1792449164, "GitCommit": "985cc081b1d5299e9d651ab01a605362f6bb170c", "GitTags": "v2.6.3", "NumGoroutines": 11 }

MaxSize is the cache cap in bytes (15 GiB on the default 20 GiB volume).

On each workstation and CI runner, copy the server's certificate from the instance. Compare its fingerprint with BAZEL_REMOTE_CERT_SHA256 from Step 4:

# on your workstation scp -i <your-key.pem> ubuntu@<public-ip>:/etc/nginx/tls/bazel-remote.crt ~/bazel-remote.crt openssl x509 -noout -fingerprint -sha256 -in ~/bazel-remote.crt

Bazel sends the password as an Authorization header. Build the header value from the password in Step 4. Put the cache settings in a .bazelrc that only you can read, never in the repository's own .bazelrc. Use your home directory's ~/.bazelrc, or a CI secret file passed with --bazelrc.

Use the instance address that your build machines can reach: the public IP as shown here, the private IP, or the DNS name from Step 4. All three are in the certificate.

Then build as usual. The first build runs the actions and uploads their results. After bazel clean, or on another machine, the same actions come from the cache:

# on your workstation
B64=$(printf 'bazel:%s' '<cache-password>' | base64 | tr -d '\n')
cat >> ~/.bazelrc <<EOF build --remote_cache=grpcs://<public-ip> build "--remote_header=Authorization=Basic ${B64}" build --tls_certificate=${HOME}/bazel-remote.crt EOF chmod 600 ~/.bazelrc bazel build //hello bazel clean bazel build //hello

Keep the quotes around the --remote_header line in the file, because the header value contains a space. The output below is from a small genrule target built against the instance:

INFO: Analyzed target //hello:hello (6 packages loaded, 8 targets configured). INFO: Found 1 target... Target //hello:hello up-to-date: bazel-bin/hello/hello.txt INFO: Elapsed time: 11.910s, Critical Path: 0.16s INFO: 2 processes: 1 internal, 1 linux-sandbox. INFO: Build completed successfully, 2 total actions INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes. INFO: Analyzed target //hello:hello (6 packages loaded, 8 targets configured). INFO: Found 1 target... Target //hello:hello up-to-date: bazel-bin/hello/hello.txt INFO: Elapsed time: 1.340s, Critical Path: 0.19s INFO: 2 processes: 1 remote cache hit, 1 internal. INFO: Build completed successfully, 2 total actions

1 remote cache hit is the cache at work. If the password is wrong, Bazel prints a Remote Cache: UNAUTHENTICATED warning and builds everything locally. A typo never breaks a build, it only makes it slower.

The HTTP cache works too, with the user and password in the URL instead of a header: build --remote_cache=https://bazel:<cache-password>@<public-dns-name> with the same --tls_certificate line. Prefer the gRPC form. Bazel uses fewer connections with it, and the password is not part of a URL that tools may print.

Requests without the password, or with a wrong one, are refused. The health check stays open for load balancers. A request for a host name the instance does not serve is dropped without an answer, and nothing answers on port 80:

C=/etc/nginx/tls/bazel-remote.crt curl -s --cacert $C -o /dev/null -w 'no login -> HTTP %{http_code}\n' https://127.0.0.1/status curl -s --cacert $C -K <(printf 'user = "bazel:not-the-password"\n') -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' https://127.0.0.1/status curl -s --cacert $C -o /dev/null -w 'health check -> HTTP %{http_code}\n' https://127.0.0.1/healthz curl -s --cacert $C -H 'Host: evil.example' -o /dev/null -w 'unknown host -> %{http_code}\n' https://127.0.0.1/healthz || true curl -s -o /dev/null -w 'plain http :80 -> %{http_code}\n' --connect-timeout 3 http://127.0.0.1/ || true

no login -> HTTP 401 wrong password -> HTTP 401 health check -> HTTP 200 unknown host -> 000 plain http :80 -> 000

000 means no answer: nginx closes the connection for a host name it does not serve, and nothing listens on port 80.

The gRPC cache refuses a call without the password with the gRPC status UNAUTHENTICATED (16):

printf '\0\0\0\0\0' > /tmp/caps.req curl -s --http2 --cacert /etc/nginx/tls/bazel-remote.crt -o /dev/null -D - -X POST -H 'content-type: application/grpc' -H 'te: trailers' \
  --data-binary @/tmp/caps.req https://127.0.0.1/build.bazel.remote.execution.v2.Capabilities/GetCapabilities | grep -i '^grpc-' rm -f /tmp/caps.req

grpc-status: 16 grpc-message: no authentication metadata found

bazel-remote runs as the unprivileged bazel-remote user with Basic authentication on and its listeners on loopback. bazel-remote, nginx and the first-boot service cannot reach the instance metadata service:

ps -o user:14=,args= -C bazel-remote systemctl show bazel-remote nginx bazel-remote-firstboot -p Id -p IPAddressDeny

bazel-remote /opt/bazel-remote/bazel-remote --dir /var/lib/bazel-remote/cache --http_address 127.0.0.1:8080 --grpc_address 127.0.0.1:9092 --htpasswd_file /etc/bazel-remote/htpasswd IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 Id=bazel-remote.service IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 Id=nginx.service IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 Id=bazel-remote-firstboot.service

bazel-remote logs at start that authentication is required for every request and that the metrics endpoint is off:

sudo journalctl -u bazel-remote -b --no-pager -o cat | grep -E 'built with|Authentication|Access mode|Endpoint metrics|Starting (HTTP|gRPC)' | sed -E 's/^[0-9/]+ [0-9:]+ //' | tail -6

bazel-remote built with go1.27.2 from git commit 985cc081b1d5299e9d651ab01a605362f6bb170c v2.6.3. Authentication: basic Access mode: authentication required Endpoint metrics: disabled Starting gRPC server on address 127.0.0.1:9092 Starting HTTP server on address 127.0.0.1:8080

bazel-remote keeps only a bcrypt hash of the password. The plain text is in the root-only credentials file, and the TLS key is root-only. The certificate was generated on this instance and names its own addresses:

sudo sed -E 's/^(bazel:\$2y\$10\$).*/\1<bcrypt-hash>/' /etc/bazel-remote/htpasswd
sudo stat -c '%a %U:%G %n' /etc/bazel-remote/htpasswd /root/bazel-remote-info.txt /etc/nginx/tls/bazel-remote.key openssl x509 -noout -subject -ext subjectAltName -in /etc/nginx/tls/bazel-remote.crt

bazel:$2y$10$<bcrypt-hash> 640 root:bazel-remote /etc/bazel-remote/htpasswd 600 root:root /root/bazel-remote-info.txt 600 root:root /etc/nginx/tls/bazel-remote.key subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = sac X509v3 Subject Alternative Name: DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost

bazel-remote-set-password manages the login file. Run it with no arguments to generate a new random password for bazel, which it writes to /root/bazel-remote-info.txt.

To choose the password yourself, hand it over on standard input. printf is a shell builtin, so the password never appears in the process list. Start the line with a space so that most shells keep it out of your history.

Give each team or CI system its own user with --user <name>, and remove one with --delete <name>. Every user can read and write the whole cache. Passwords are 12 to 128 printable characters without spaces or :. bazel-remote reads the login file at start, so the helper restarts it:

sudo bazel-remote-set-password printf '%s\n' '<paste-a-password-for-ci>' | sudo bazel-remote-set-password --user ci --stdin sudo bazel-remote-set-password --delete ci

Point a DNS record at the instance, then set the name that the cache is served as. The command below mints a new self-signed certificate for the name (and the instance's IP addresses), updates nginx and the URLs in the credentials file, and keeps the name across reboots. Afterwards, give your build machines the new certificate:

sudo bazel-remote-set-hostname <your-domain>

To use a certificate from your own certificate authority instead, copy it over the generated pair and reload nginx. Running bazel-remote-set-hostname again replaces it with a self-signed one. Build machines that trust your authority no longer need --tls_certificate:

sudo install -m 0644 <your-domain>.crt /etc/nginx/tls/bazel-remote.crt
sudo install -m 0600 <your-domain>.key /etc/nginx/tls/bazel-remote.key
sudo nginx -t && sudo systemctl reload nginx

The cache lives on the separate EBS volume, owned by the bazel-remote service account and mounted without device files, set-uid programs or execution. The size cap, in GiB, is in /etc/bazel-remote/bazel-remote.env:

findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/bazel-remote df -h --output=size,used,avail,target /var/lib/bazel-remote
sudo stat -c '%a %U:%G %n' /var/lib/bazel-remote
sudo grep -v '^#' /etc/bazel-remote/bazel-remote.env

/dev/nvme1n1 ext4 rw,nosuid,nodev,noexec,relatime Size Used Avail Mounted on 20G 3.2M 19G /var/lib/bazel-remote 750 bazel-remote:bazel-remote /var/lib/bazel-remote BAZEL_REMOTE_MAX_SIZE=15

To grow the cache later, modify the volume's size in the EC2 console and grow the filesystem. Then raise the cap to about 80% of the new size (80 for a 100 GiB volume) and restart bazel-remote:

sudo resize2fs "$(findmnt -no SOURCE /var/lib/bazel-remote)"
sudo sed -i -E 's/^BAZEL_REMOTE_MAX_SIZE=.*/BAZEL_REMOTE_MAX_SIZE=80/' /etc/bazel-remote/bazel-remote.env
sudo systemctl restart bazel-remote

A build cache can always be rebuilt, so it needs no backup. If it is ever corrupted, stop bazel-remote, empty /var/lib/bazel-remote/cache and start it again.

You can add other documented BAZEL_REMOTE_* settings to the same file, for example BAZEL_REMOTE_MAX_SIZE_HARD_LIMIT. Settings that would turn off authentication, move a listener, or enable profiling or the metrics endpoint are refused, and bazel-remote does not start with them.

Component Path

bazel-remote release binary /opt/bazel-remote/bazel-remote

Licence /usr/share/doc/bazel-remote/LICENSE

Cache (EBS volume) /var/lib/bazel-remote/cache

Settings (cache size) /etc/bazel-remote/bazel-remote.env

Login file (bcrypt) /etc/bazel-remote/htpasswd

Service unit /etc/systemd/system/bazel-remote.service (loopback only)

nginx site /etc/nginx/sites-available/sac-bazel-remote

Certificate and key /etc/nginx/tls/

First boot script /usr/local/sbin/bazel-remote-firstboot.sh

Helpers /usr/local/sbin/bazel-remote-set-password, bazel-remote-set-hostname, bazel-remote-auth-guard

Credentials /root/bazel-remote-info.txt (mode 0600)

cat /opt/bazel-remote/VERSION sha256sum /opt/bazel-remote/bazel-remote nginx -v 2>&1

bazel-remote 2.6.3 (official release binary, Apache-2.0, sha256 65427624c288e29b17c7b0e189cf1d03ae1d8745354d9effc432f75c4361159a) 65427624c288e29b17c7b0e189cf1d03ae1d8745354d9effc432f75c4361159a /opt/bazel-remote/bazel-remote nginx version: nginx/1.24.0 (Ubuntu)

Logs: sudo journalctl -u bazel-remote for bazel-remote, /var/log/nginx/access.log for requests (with the real client address).

To move to a newer bazel-remote release, download the linux-amd64 binary from the bazel-remote releases page and check it against the SHA-256 that the release page publishes. Then replace /opt/bazel-remote/bazel-remote and run sudo systemctl restart bazel-remote. The cache in /var/lib/bazel-remote is kept.

• Allow port 22 in the security group from your own networks only, and port 443 only from your developers' and CI runners' networks.
• Give each team or CI system its own user (Step 8), and rotate a password if it may have been shared. Anyone with a password can write to the cache, and builds trust what the cache returns.
• Keep the cache settings in a private .bazelrc or a CI secret, never in the repository.
• Use your own DNS name and certificate (Step 8).
• Keep the operating system patched. Unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.