bazel-remote cache

Search the Marketplace for bazel-remote on Ubuntu 24.04 and choose your VM size.

Attach a network security group (NSG) that allows TCP 22 (SSH) from your management network. Also allow TCP 8080 (HTTP cache) and TCP 9092 (gRPC cache) from the build and CI machines that will use the cache.

Basic Auth is enforced, so the cache is safe to expose to your build fleet. Still, restrict the source ranges to networks you control.

RG="bazel-remote-prod"; LOCATION="eastus"; VM_NAME="bazel-remote-01" IMAGE_URN="sac1647283583153:bazel-remote-cache-ubuntu-24-04:default:latest" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create -g "$RG" --name br-vnet --address-prefix 10.90.0.0/16 --subnet-name br-subnet --subnet-prefix 10.90.1.0/24 az network nsg create -g "$RG" --name br-nsg az network nsg rule create -g "$RG" --nsg-name br-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name br-nsg --name allow-cache --priority 110 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 8080 9092 --access Allow --protocol Tcp
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:bazel-remote-cache-ubuntu-24-04:default:latest
az vm create -g "$RG" --name "$VM_NAME" --image "$IMAGE_URN" \
  --size Standard_B2s --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name br-vnet --subnet br-subnet --nsg br-nsg --public-ip-sku Standard

ssh azureuser@<vm-ip>

bazel-remote serves the HTTP cache on :8080 and the gRPC cache on :9092. Every request requires authentication, so an unauthenticated call to the status endpoint returns 401. Run:

sudo systemctl start bazel-remote 2>/dev/null || true
sudo systemctl is-active bazel-remote ss -tln | grep -E ':8080|:9092' echo "unauthenticated /status -> HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/status)"

The output shows the service active, both listening sockets, and HTTP 401 from the unauthenticated status call. The 401 confirms that authentication is enforced.

The username and password were generated on this VM's first boot and stored in a root-only file. Read them and keep them secret. Then confirm that the per-VM credential authenticates and that an unauthenticated request does not.

sudo cat /root/bazel-remote-info.txt
U=$(sudo grep '^BAZEL_REMOTE_USER=' /root/bazel-remote-info.txt | cut -d= -f2-)
P=$(sudo grep '^BAZEL_REMOTE_PASSWORD=' /root/bazel-remote-info.txt | cut -d= -f2-)
echo "unauthenticated -> HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/status)"
echo "authenticated -> HTTP $(curl -s -o /dev/null -w '%{http_code}' -u "$U:$P" http://127.0.0.1:8080/status)"

The info note holds BAZEL_REMOTE_USER, BAZEL_REMOTE_PASSWORD and the cache URL. It is owned root:root with mode 0600. The status endpoint returns 401 without the credential and 200 with it. Basic Auth is enforced identically on the gRPC interface.

The Content Addressable Storage keys every blob by the SHA-256 of its content. Store a blob with an authenticated PUT, then fetch it back and confirm that it matches byte for byte. Bazel does exactly this for every cached artifact.

U=$(sudo grep '^BAZEL_REMOTE_USER=' /root/bazel-remote-info.txt | cut -d= -f2-)
P=$(sudo grep '^BAZEL_REMOTE_PASSWORD=' /root/bazel-remote-info.txt | cut -d= -f2-) BLOB="hello from bazel-remote at $(date -u +%FT%TZ)"
HASH=$(printf '%s' "$BLOB" | sha256sum | awk '{print $1}') printf '%s' "$BLOB" | curl -s -u "$U:$P" -X PUT --data-binary @- "http://127.0.0.1:8080/cas/$HASH" -o /dev/null -w 'PUT /cas -> HTTP %{http_code}\n'
GOT=$(curl -s -u "$U:$P" "http://127.0.0.1:8080/cas/$HASH") [ "$GOT" = "$BLOB" ] && echo "GET /cas -> byte for byte match" || echo "GET /cas -> MISMATCH" echo "unauthenticated GET /cas -> HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/cas/$HASH)"

The PUT returns 200, and the authenticated GET returns the identical blob. An unauthenticated GET of the same key returns 401, so cached content is never served without the credential.

Add the cache to your project's .bazelrc, or pass it on the command line. Use the username and password from Step 5. Over HTTP, put the credentials in the URL:

# .bazelrc — HTTP remote cache (username:password from /root/bazel-remote-info.txt) build --remote_cache=http://<cache-username>:<cache-password>@<vm-ip>:8080
# optional: keep going if the cache is briefly unreachable build --remote_local_fallback

To use the gRPC interface instead, point Bazel at grpc://<vm-ip>:9092 and pass the same credentials as a Basic Auth header:

# .bazelrc — gRPC remote cache. Compute the header once:
# echo -n "<cache-username>:<cache-password>" | base64 build --remote_cache=grpc://<vm-ip>:9092 build --remote_header=authorization="Basic <base64-of-username:password>"

Run any build twice. The first build populates the cache. A clean second build (bazel clean then bazel build //...) pulls the results back from the cache instead of recompiling. Share the same cache across your CI fleet so that every machine benefits.

The cache size is capped by BAZEL_REMOTE_MAX_SIZE_GIB in /etc/bazel-remote/bazel-remote.env (default 10 GiB). Once the cap is reached, the least recently used entries are evicted. Inspect the current setting and disk usage:

grep -E 'MAX_SIZE|DIR' /etc/bazel-remote/bazel-remote.env du -sh /var/lib/bazel-remote/cache 2>/dev/null df -h /var/lib/bazel-remote

To grow the cache, raise BAZEL_REMOTE_MAX_SIZE_GIB and run sudo systemctl restart bazel-remote.

For large caches, attach a dedicated data disk and mount it at /var/lib/bazel-remote/cache, preserving bazel-remote ownership. Raise the cap to match the disk, but keep it comfortably below the disk size to leave room for in-flight evictions.

The cache enforces Basic Auth, but Basic Auth protects the credential on the wire only if the transport is encrypted. Restrict the NSG source ranges to networks you control, and enable TLS for traffic across untrusted networks. bazel-remote serves HTTPS and gRPCS natively when you supply a certificate and key:

# add to the service and restart (see the bazel-remote docs for --tls_cert_file / --tls_key_file)
# --tls_cert_file /etc/bazel-remote/tls/cert.pem --tls_key_file /etc/bazel-remote/tls/key.pem
# clients then use https://... and grpcs://... ; supply the CA to Bazel with --tls_certificate echo "See https://github.com/buchgr/bazel-remote for the TLS and mTLS flags."

With TLS enabled, point Bazel at https://<vm-ip>:8080 or grpcs://<vm-ip>:9092. Pass the certificate authority to Bazel with --tls_certificate.

• Open the product page in the AWS Marketplace. Choose Continue to Subscribe, then Continue to Configuration.
• Select the software version and the Region you want to launch in. Then choose Continue to Launch.
• Under Choose Action, select Launch through EC2.
• Review the instance type (the listing recommends m5.large), your key pair, and a security group that allows inbound TCP 22, 8080 and 9092 from your own hosts. Then launch.

Replace the AMI ID, key name, subnet and security group with your own values:

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type m5.large \
  --key-name your-key \
  --security-group-ids sg-xxxxxxxx \
  --subnet-id subnet-xxxxxxxx \
  --associate-public-ip-address \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=bazel-remote-cache}]'

Your security group should allow inbound TCP 22 from your management network. Allow inbound TCP 8080 and 9092 only from the CI and build hosts that will use the cache.

SSH in as the default login user for your operating system variant:

OS variant SSH login user

Ubuntu 24.04 ubuntu

ssh -i your-key.pem ubuntu@<public-ip>

The cache starts automatically after the first-boot service has created this instance's credential and written the bootstrap marker. Confirm that the service is active and listening on TCP 8080 (HTTP) and 9092 (gRPC):

sudo systemctl is-active bazel-remote
sudo ss -ltn | grep -E ':8080|:9092' grep -m1 -i 'remote build cache' <<< "$(/usr/local/bin/bazel-remote --help 2>&1)"

On first boot, a unique username and a long random password were generated for this instance and written to a root-owned file. Read them with:

sudo cat /root/bazel-remote-info.txt

The file records BAZEL_REMOTE_USER, BAZEL_REMOTE_PASSWORD, a ready-to-use BAZEL_REMOTE_URL (HTTP) and BAZEL_REMOTE_GRPC_URL (gRPC), all unique to your instance. Keep this credential safe. Every request to the cache, over HTTP or gRPC, must present it.

Add the cache to your project's .bazelrc. Use the per-instance username and password and your instance's public IP. Bazel accepts either the HTTP or the gRPC endpoint:

# HTTP Remote Cache build --remote_cache=http://your-user:your-password@<public-ip>:8080
# or gRPC Remote Cache build --remote_cache=grpc://your-user:your-password@<public-ip>:9092

You can verify the cache end to end on the instance itself. This command reads the per-instance credential and performs an authenticated, content-addressed PUT then GET round trip against the local HTTP endpoint. It also confirms that the store lives on the dedicated data volume:

U=$(sudo grep '^BAZEL_REMOTE_USER=' /root/bazel-remote-info.txt | cut -d= -f2-)
P=$(sudo grep '^BAZEL_REMOTE_PASSWORD=' /root/bazel-remote-info.txt | cut -d= -f2-)
BLOB=$(head -c 512 /dev/urandom | base64 -w0)
H=$(printf '%s' "$BLOB" | sha256sum | awk '{print $1}') printf '%s' "$BLOB" | curl -s -o /dev/null -w 'PUT -> HTTP %{http_code}\n' -u "$U:$P" -X PUT --data-binary @- "http://127.0.0.1:8080/cas/$H"
GOT=$(curl -s -u "$U:$P" "http://127.0.0.1:8080/cas/$H") [ "$GOT" = "$BLOB" ] && echo "GET -> 200 OK, retrieved bytes match stored bytes (cache hit)" df -h /var/lib/bazel-remote | tail -1

The cache has no anonymous access. An unauthenticated request to either the status endpoint or a CAS object is rejected with HTTP 401, and the same Basic Auth is enforced on the gRPC interface:

curl -s -o /dev/null -w 'unauthenticated /status -> HTTP %{http_code}\n' http://127.0.0.1:8080/status
curl -s -o /dev/null -w 'unauthenticated /cas -> HTTP %{http_code}\n' http://127.0.0.1:8080/cas/deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
sudo ss -Htln 'sport = :9092' && echo "gRPC listener live on :9092"

The cache store lives on a dedicated EBS data volume at /var/lib/bazel-remote/cache. The maximum on-disk cache size is capped by BAZEL_REMOTE_MAX_SIZE_GIB (default 10 GiB) in the environment file. When the cap is reached, bazel-remote evicts least-recently-used entries.

To grow the cache, raise the cap and restart the service:

sudo sed -i 's/^BAZEL_REMOTE_MAX_SIZE_GIB=.*/BAZEL_REMOTE_MAX_SIZE_GIB=40/' /etc/bazel-remote/bazel-remote.env
sudo systemctl restart bazel-remote

If you need more than the data volume's capacity, resize the EBS volume in the AWS console and grow the filesystem (sudo resize2fs /dev/nvme1n1). Then raise the cap to match.

• Restrict access. Scope the instance security group so only your own CI and build hosts can reach TCP 8080 and 9092.
• TLS. For encrypted transport, terminate TLS in front of the cache (for example with a reverse proxy such as nginx or Caddy), or run bazel-remote with its own --tls_cert_file and --tls_key_file. SecureAnyCloud support can help you set this up.
• Metrics. bazel-remote exposes Prometheus metrics and an authenticated status endpoint for monitoring cache hit rates and size.
• Updates. Each instance refreshes itself to the newest bazel-remote release on first boot. On a running instance, replace /usr/local/bin/bazel-remote with a newer release binary and restart the service.