Arroyo 0.15.0

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Arroyo 0.15.0 on Ubuntu 24.04 as the software version, select your Region, and choose Launch through EC2.

Choose m5.large (or larger for heavy pipelines), your key pair and the security group from the prerequisites.

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=arroyo}]'

First boot takes about a minute after the instance is running. The boot service generates the login, the API key and the certificate, and then Arroyo and nginx start.

Connect over SSH as the ubuntu user, which is the default user for Arroyo 0.15.0 on Ubuntu 24.04:

ssh -i <your-key.pem> ubuntu@<public-ip>

Confirm that the four services are active and that the first-boot marker exists:

systemctl is-active arroyo-metadata-names arroyo-firstboot arroyo nginx
sudo test -f /var/lib/sac/data/arroyo-firstboot.done && echo FIRSTBOOT_DONE

Expected output:

active active active active FIRSTBOOT_DONE

Only SSH and nginx listen on network addresses. Arroyo's admin service (5114), API (5115), controller (5116) and compiler (5117) listen on 127.0.0.1:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:5114 users:(("arroyo")) 127.0.0.1:5115 users:(("arroyo")) 127.0.0.1:5116 users:(("arroyo")) 127.0.0.1:5117 users:(("arroyo")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

While a pipeline runs, its worker process adds a few more listeners on random ports. These also bind to 127.0.0.1 only.

Read the credentials file as root. It holds the console URL, the admin user and password, the Arroyo API key and the SHA-256 fingerprint of this instance's certificate:

sudo cat /root/arroyo-credentials.txt

# Arroyo 0.15.0 on Ubuntu 24.04 - generated on first boot by arroyo-firstboot.service.
# The console login and the Arroyo API key are unique to this instance; none was baked into the image. Keep this file
# secret. Signing in to the console lets you run SQL pipelines and Python UDFs, which is code execution on this
# instance: share the login only with people you would give a shell to.
# nginx stores the password only as a bcrypt hash (/etc/nginx/arroyo-htpasswd). The console is served over HTTPS only
# (port 443; port 80 redirects). The certificate is self signed (minted on this instance): before you sign in, check
# that the SHA-256 fingerprint your browser shows matches ARROYO_TLS_CERT_SHA256 below - or install a CA-issued
# certificate with
#
sudo arroyo-install-cert <chain.pem> <key.pem>
# Change the served name:
sudo arroyo-set-hostname <dns-name> Re-verify:
sudo arroyo-selfcheck ARROYO_URL=https://<public-ip>/ ARROYO_HOST=<public-ip> ARROYO_ADMIN_USER=admin ARROYO_ADMIN_PASSWORD=<ARROYO_ADMIN_PASSWORD> ARROYO_API_KEY=<api-key> ARROYO_TLS_CERT_SHA256=<sha256-fingerprint> ARROYO_TLS_CERT_FILE=/etc/sac/arroyo-ui.pem
# REST API through nginx with the login:
sudo curl -K /root/arroyo-curl.conf --cacert /etc/sac/arroyo-ui.pem https://<public-ip>/api/v1/pipelines
# REST API on this instance with the key:
sudo curl -K /root/arroyo-api.curl http://127.0.0.1:5115/api/v1/pipelines
# User guide: https://secureanycloud.com/stacks/arroyo-0-15-0/

To print only the URL, the user and the fingerprint, without the password or the key:

sudo grep -E '^ARROYO_(URL|ADMIN_USER|TLS_CERT_SHA256)=' /root/arroyo-credentials.txt

ARROYO_URL=https://<public-ip>/ ARROYO_ADMIN_USER=admin ARROYO_TLS_CERT_SHA256=<sha256-fingerprint>

The host is the instance's public IP address from the EC2 instance metadata, or the private address when the instance has no public one. The certificate also names the public DNS name, the private IP address and the private DNS name.

If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it, unless you installed your own (Step 14).

To read the certificate's names and constraints from the instance:

openssl x509 -in /etc/sac/arroyo-ui.pem -noout -ext subjectAltName,basicConstraints,keyUsage,extendedKeyUsage

X509v3 Subject Alternative Name: IP Address:<public-ip>, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<private-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, IP Address:127.0.0.1, DNS:localhost X509v3 Basic Constraints: critical CA:FALSE X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication

Browse to the ARROYO_URL from Step 4. The certificate is self-signed and unique to this instance, so the browser warns about it the first time.

Check the certificate before you continue. Open the browser's certificate viewer for the page and compare its SHA-256 fingerprint with ARROYO_TLS_CERT_SHA256 from the credentials file. Continue only if they match, and only for this site.

Never add this certificate to an operating-system or browser trust store. For a certificate that browsers trust without a warning, install a CA-issued one (Step 14).

When the browser asks for a user name and password, sign in as admin with the password from the credentials file. To print just the password on the instance:

sudo awk -F= '$1=="ARROYO_ADMIN_PASSWORD" {print $2}' /root/arroyo-credentials.txt

Pipelines lists your pipelines with their state. Create Pipeline opens the SQL editor. Write a query, select Preview to run it against live data without saving it, and select Launch to start it as a pipeline. In this example, a Nexmark benchmark source (built into Arroyo) feeds a query that also calls a Python UDF (Step 8):

Each pipeline has its own page with the operator graph, outputs, checkpoints, the query, its UDFs and any errors:

Connections creates reusable connection profiles and tables for the external systems your pipelines read and write:

If a preview stays on Loading..., select Stop and then Preview again. The console sometimes subscribes to the output a moment before the job is ready.

Everything the console does goes through Arroyo's REST API (/api/v1/...). You can call it in two ways:

• Through nginx, with the login, from anywhere that can reach port 443. /root/arroyo-curl.conf holds the login as a curl config file, so the password never appears on a command line. nginx replaces your login with the API key on the way to Arroyo.
• On the instance, with the API key, directly on 127.0.0.1:5115. /root/arroyo-api.curl holds the key as a curl header line.

On the instance, the examples pin the instance's public name to 127.0.0.1 with --resolve. They therefore work whatever the security group allows, and they check the server against this instance's certificate:

H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem https://$H/api/v1/ping; echo curl -s -o /dev/null -w 'without the login: %{http_code}\n' --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem https://$H/api/v1/pipelines curl -s -o /dev/null -w 'direct, without the key: %{http_code}\n' http://127.0.0.1:5115/api/v1/pipelines
sudo curl -s -K /root/arroyo-api.curl http://127.0.0.1:5115/api/v1/connectors | python3 -c 'import json,sys; print(" ".join(sorted(c["id"] for c in json.load(sys.stdin)["data"])))'

"Pong" without the login: 401 direct, without the key: 401 blackhole confluent delta filesystem fluvio iceberg impulse kafka kinesis mqtt nats nexmark polling_http preview rabbitmq redis single_file_source sse stdout webhook websocket

From your workstation, copy the certificate (Step 4) and keep the login in a file that only you can read. Then call the API the same way:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/sac/arroyo-ui.pem arroyo.pem openssl x509 -in arroyo.pem -noout -fingerprint -sha256
# must match ARROYO_TLS_CERT_SHA256 ( umask 077; printf 'user = "admin:%s"\n' "<your-password>" > arroyo-login.conf )
curl -K arroyo-login.conf --cacert arroyo.pem https://<public-ip>/api/v1/pipelines

Some requests must carry the browser's same-origin signal: those that stream a pipeline's output, and those that make the server connect to another system (connection tests, autocomplete, query and UDF validation, previews). In scripts, add -H 'Sec-Fetch-Site: same-origin', as in Step 7.

The full API description is at /api/v1/api-docs/openapi.json, behind the same login.

This pipeline reads Arroyo's built-in Nexmark source, a generated stream of auctions, bids and people. It counts the bids per auction in 5-second tumbling windows. Create it through the API. The console's Create Pipeline does the same:

mkdir -p ~/arroyo-demo && cd ~/arroyo-demo cat > bids.sql <<'SQL' CREATE TABLE nexmark WITH (connector = 'nexmark', event_rate = '200'); SELECT bid.auction AS auction, count(*) AS bids FROM nexmark WHERE bid IS NOT NULL GROUP BY bid.auction, tumble(interval '5 seconds'); SQL python3 -c 'import json; print(json.dumps({"name": "nexmark-bids", "query": open("bids.sql").read(), "parallelism": 1}))' > pipeline.json
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @pipeline.json https://$H/api/v1/pipelines \
  | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["id"], p["name"])' | tee pipeline.id

pl_<id> nexmark-bids

Wait for its job to run:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt);
P=$(cut -d' ' -f1 pipeline.id) for i in $(seq 1 30); do
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem https://$H/api/v1/pipelines/$P/jobs \
  | python3 -c 'import json,sys; j=json.load(sys.stdin)["data"][0]; print(j["id"], j["state"])' > job.id grep -q Running job.id && break; sleep 2 done cat job.id

job_<id> Running

Read a few seconds of the pipeline's live output. The output is a server-sent events stream of record batches, which this command decodes:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt);
P=$(cut -d' ' -f1 pipeline.id);
J=$(cut -d' ' -f1 job.id) {
sudo timeout 15 curl -s -N -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -H 'Sec-Fetch-Site: same-origin' https://$H/api/v1/pipelines/$P/jobs/$J/output || true; } | python3 -c ' import json, sys n = 0 for line in sys.stdin: if line.startswith("data:") and n < 6: for r in json.loads(json.loads(line[5:])["batch"])[:3]: print("auction %(auction)s: %(bids)s bids" % r); n += 1'

auction 1632: 8 bids auction 1600: 403 bids auction 1558: 2 bids auction 1673: 5 bids auction 1700: 478 bids auction 1712: 5 bids

A pipeline with no sink of its own writes to the preview sink you just read. For production, end the query with an INSERT INTO a sink table, such as Kafka, a file system or object store, or a webhook. Step 10 shows Kafka.

Python user-defined functions work out of the box. The Arroyo build in this image embeds Python 3.12 from Ubuntu and needs no compiler.

A UDF is a Python function with type annotations and the @udf decorator. Register one for the whole cluster, where it then appears under UDFs in the editor's side panel, and use it in a query:

cd ~/arroyo-demo cat > price_band.py <<'PY' from arroyo_udf import udf @udf def price_band(price: int) -> str: if price < 1000: return "low" if price < 10000: return "medium" return "high" PY python3 -c 'import json; print(json.dumps({"prefix": "", "language": "python", "definition": open("price_band.py").read(), "description": "Bid price band"}))' > udf.json
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @udf.json https://$H/api/v1/udfs | python3 -c 'import json,sys; u=json.load(sys.stdin); print(u["name"], u["language"])' cat > bands.sql <<'SQL' CREATE TABLE nexmark WITH (connector = 'nexmark', event_rate = '200'); SELECT price_band(bid.price) AS band, count(*) AS bids FROM nexmark WHERE bid IS NOT NULL GROUP BY price_band(bid.price), tumble(interval '5 seconds'); SQL python3 -c 'import json; print(json.dumps({"name": "nexmark-price-bands", "query": open("bands.sql").read(), "parallelism": 1}))' > bands.json
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @bands.json https://$H/api/v1/pipelines \
  | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["id"], p["name"])' | tee bands.id

price_band python pl_<id> nexmark-price-bands

After a few seconds, the output shows the bid count per price band for each 5-second window:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt);
B=$(cut -d' ' -f1 bands.id) for i in $(seq 1 30); do
BJ=$(sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem https://$H/api/v1/pipelines/$B/jobs \
  | python3 -c 'import json,sys; j=json.load(sys.stdin)["data"][0]; print(j["id"] if j["state"] == "Running" else "")') [ -n "$BJ" ] && break; sleep 2 done {
sudo timeout 15 curl -s -N -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -H 'Sec-Fetch-Site: same-origin' https://$H/api/v1/pipelines/$B/jobs/$BJ/output || true; } | python3 -c ' import json, sys n = 0 for line in sys.stdin: if line.startswith("data:") and n < 6: for r in json.loads(json.loads(line[5:])["batch"]): print("%(band)s: %(bids)s bids" % r); n += 1'

high: 615 bids medium: 154 bids low: 151 bids medium: 155 bids low: 159 bids high: 606 bids

Security note. A UDF is arbitrary Python code that runs inside the pipeline's worker process on this instance. SQL connectors can read and write files and open connections to any address the instance can reach. Anyone with the console login can execute code on the instance, so treat the login like a shell login.

The image confines that code. It runs as the unprivileged arroyo user, may write only to /var/lib/arroyo, cannot read the credentials file, the TLS key or the login hash, and cannot reach the instance metadata service. Step 12 proves this from inside a running pipeline. The engine must be able to read the API key, so code in a pipeline can read it too. That grants nothing the login does not already grant.

Only the Python standard library is available to UDFs. Rust UDFs need a Rust toolchain, which the image does not ship.

Arroyo's compiler can download a toolchain at start-up if you set install-rustc = true and install-clang = true in /etc/arroyo/config.toml and restart Arroyo. That downloads and runs a compiler from the internet as the arroyo user, and Python UDFs cover most needs without it.

Stop a pipeline with a final checkpoint (or with "immediate"), then delete it:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt) for f in pipeline.id bands.id; do
P=$(cut -d' ' -f1 $f)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -X PATCH -H 'Content-Type: application/json' --data '{"stop": "checkpoint"}' https://$H/api/v1/pipelines/$P | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["name"], "stop:", p["stop"])' done sleep 15 for f in pipeline.id bands.id; do
P=$(cut -d' ' -f1 $f)
sudo curl -s -o /dev/null -w "delete $P: %{http_code}\n" -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem -X DELETE https://$H/api/v1/pipelines/$P done

nexmark-bids stop: checkpoint nexmark-price-bands stop: checkpoint delete pl_<id>: 200 delete pl_<id>: 200

{"stop": "none"} restarts a stopped pipeline from its last checkpoint. In the console, select Stop on the pipeline's page, and use the delete icon in the pipelines list.

Create a Kafka connection profile in the console (Connections, Create, Kafka), then create tables that use it. For a broker with a TLS listener, set the bootstrap servers and these Connection Properties (librdkafka options):

Property Value

security.protocol SSL (or SASL_SSL with SASL authentication)

ssl.ca.location a PEM file on the instance, readable by Arroyo, that holds the CA (or the broker certificate) to trust

Arroyo reads files under /etc and /var/lib/arroyo, so place the trust file in /etc/arroyo/kafka/.

If the broker uses a self-signed certificate, pin it for this client only. Copy the broker certificate to the instance and check its fingerprint against the one the broker administrator gives you. Then point ssl.ca.location at that copy. Never add a broker's self-signed certificate to the instance's system trust store.

sudo install -d -m 0755 /etc/arroyo/kafka
sudo install -m 0644 broker.pem /etc/arroyo/kafka/broker.pem openssl x509 -in /etc/arroyo/kafka/broker.pem -noout -fingerprint -sha256
# compare with the broker administrator's value

To create the same profile through the API, put your bootstrap servers in the angle brackets:

cat > kafka-profile.json <<'JSON' {"name": "kafka-tls", "connector": "kafka", "config": {"bootstrapServers": "<broker-1>:9093,<broker-2>:9093", "authentication": {}, "schemaRegistryEnum": {}, "connectionProperties": {"security.protocol": "SSL", "ssl.ca.location": "/etc/arroyo/kafka/broker.pem"}}} JSON
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/sac/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @kafka-profile.json https://$H/api/v1/connection_profiles

A query can then read a topic through the profile and write results back to another topic:

CREATE TABLE orders (order_id TEXT, amount DOUBLE) WITH (connector = 'kafka', connection_profile = 'kafka-tls', topic = 'orders', type = 'source', format = 'json'); CREATE TABLE order_totals (orders BIGINT, total DOUBLE) WITH (connector = 'kafka', connection_profile = 'kafka-tls', topic = 'order-totals', type = 'sink', format = 'json'); INSERT INTO order_totals SELECT count(*) AS orders, sum(amount) AS total FROM orders GROUP BY tumble(interval '1 minute');

Keep SASL passwords out of saved queries. Connection fields marked var-str accept {{ NAME }} template variables that Arroyo fills from its environment. Put the values in a root-only file and add it to the service with sudo systemctl edit arroyo ([Service] then EnvironmentFile=/etc/arroyo/secrets.env).

The instance metadata block (Step 13) means Arroyo cannot use the instance's IAM role. Amazon MSK IAM authentication and the Kinesis connector therefore need explicit credentials.

Arroyo keeps its metadata database (pipelines, connection profiles, UDFs) and the pipelines' checkpoints in /var/lib/arroyo, the only directory the engine may write:

sudo ls -l /var/lib/arroyo | awk 'NR>1 {print $1, $3, $4, $9}' grep -E '^(checkpoint-url|default-checkpoint-interval)' /etc/arroyo/config.toml

drwxr-x--- arroyo arroyo artifacts drwxr-x--- arroyo arroyo checkpoints -rw-r----- arroyo arroyo config.sqlite -rw-r----- arroyo arroyo config.sqlite-shm -rw-r----- arroyo arroyo config.sqlite-wal checkpoint-url = "/var/lib/arroyo/checkpoints" default-checkpoint-interval = "10s"

The directory starts empty on every new instance.

For large state, attach an EBS volume and stop Arroyo (sudo systemctl stop arroyo). Copy /var/lib/arroyo onto the volume, mount it on /var/lib/arroyo (owner arroyo, mode 0750), and start Arroyo again.

To keep checkpoints in Amazon S3 instead, set checkpoint-url = "s3://<bucket>/<prefix>" in /etc/arroyo/config.toml. Give Arroyo credentials through an EnvironmentFile as in Step 10 (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION). The engine cannot reach the instance's IAM role.

Run arroyo-selfcheck to re-prove the whole appliance with this instance's credentials. It checks:

• The units and the per-instance files, and the served certificate.
• The login: anonymous and wrong-password requests are refused, and the login is accepted.
• The API key on 127.0.0.1: missing and wrong keys are refused.
• The redirect, unknown host names, foreign origins, WebSocket upgrades, form posts, ; path parameters and encoded traversal.
• The exact listening set.

It then runs a real SQL pipeline and checks its output. From inside running pipelines, it also proves the sandbox. A Python UDF cannot read the credentials file, the TLS key, the login hash or nginx's key file, and cannot reach the instance metadata service, while it can reach a loopback address. An HTTP-source connector pointed at 169.254.169.254 fails, while the same connector reads a local test endpoint. The check cleans up after itself:

sudo arroyo-selfcheck

ok units active; per-instance credentials, login hash, API key and TLS key present with their permissions ok TLS: the served certificate is this instance's end-entity certificate (CA:FALSE, serverAuth; SHA-256 <first-bytes>...) ok front door: HTTPS only (80 -> 301), unknown Host dropped, login 401/401/200 on the console, API and output stream, foreign Origin 403, WebSocket 403, OPTIONS 405, form POST 415, ';' 400, encoded traversal 400, CORS stripped, basename header blanked; API key on 127.0.0.1:5115/:5114 401/401/200 ok public listeners 22/80/443 only; every Arroyo listener on 127.0.0.1 .. 400 live output records, 400 with transformed == counter*10+7 ok SQL pipeline: invalid query refused 400; impulse -> transform Running, live output verified; worker in arroyo's cgroup, every listener on loopback .. 25 UDF probe rows: uid=<arroyo-uid>;creds=EACCES;tlskey=EACCES;htpasswd=EACCES;bearer=EACCES;control=READ;imds=TIMEOUT;imds6=TIMEOUT;loopback=OPEN ok Python UDF (in a running pipeline, uid arroyo): credentials file, TLS key, login hash and bearer include unreadable (EACCES; config control READ); 169.254.169.254 unreachable (v4 and v4-mapped), loopback control OPEN ok HTTP-source connector (polling_http, in a running pipeline): http://169.254.169.254/ -> 'error sending request', 0 records; loopback control -> the nonce SELFCHECK OK

To view the engine's sandbox settings:

systemctl show -p User,NoNewPrivileges,ProtectSystem,ProtectHome,PrivateTmp,ReadWritePaths arroyo.service

User=arroyo ReadWritePaths=/var/lib/arroyo PrivateTmp=yes ProtectHome=yes ProtectSystem=strict NoNewPrivileges=yes

systemd IPAddressDeny rules deny the instance metadata service (169.254.169.254) to Arroyo (and so to every pipeline worker and UDF), to nginx and to the boot service. Neither pipeline code nor a request to the console can read the instance's IAM credentials.

The only service allowed to reach the metadata service is arroyo-metadata-names. It runs once at boot, reads the instance's host names and addresses (no credentials) for the certificate, and may reach nothing else:

for u in arroyo nginx arroyo-firstboot arroyo-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u.service)" "$(systemctl show -p IPAddressAllow --value $u.service)"; done

arroyo: deny=::ffff:168.63.129.16/128 169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32 ::ffff:169.254.169.254/128 allow= nginx: deny=169.254.169.254/32 fd00:ec2::254/128 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 168.63.129.16/32 allow= arroyo-firstboot: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 allow= arroyo-metadata-names: deny=::/0 0.0.0.0/0 allow=169.254.169.254/32 fd00:ec2::254/128

Pipelines can still reach your sources and sinks on the network and the internet. Only the metadata address is blocked. Connectors that would normally use the instance's IAM role (Kinesis, S3 checkpoints, MSK IAM) need explicit credentials (Step 10).

To serve the console under your own DNS name, point the name at the instance (an Elastic IP is recommended) and then set it. This writes the name into nginx and issues a new self-signed certificate that covers it:

sudo arroyo-set-hostname arroyo.example.com

To install a certificate from your CA (or Let's Encrypt), copy the full chain (server certificate first) and the unencrypted private key to the instance. Then run:

sudo arroyo-install-cert /path/to/fullchain.pem /path/to/privkey.pem arroyo.example.com

The boot-time renewal never replaces a certificate installed this way. To go back to a self-signed certificate, run sudo arroyo-set-hostname <name>. Both commands update the URL and fingerprint in /root/arroyo-credentials.txt.

The password is stored as a bcrypt hash in /etc/nginx/arroyo-htpasswd. To set a new one without putting it on a command line, let htpasswd prompt for it. Then update the curl config and the credentials file:

sudo htpasswd -B -C 10 /etc/nginx/arroyo-htpasswd admin
sudo nano /root/arroyo-curl.conf /root/arroyo-credentials.txt
sudo systemctl reload nginx

To rotate the API key, write a new 64-character hex key (for example from openssl rand -hex 32) into these places:

• The two api-key lines of /etc/arroyo/conf.d/90-sac-auth.toml
• /etc/nginx/arroyo-upstream-auth.conf
• /root/arroyo-api.curl
• The credentials file

Then run sudo systemctl restart arroyo and sudo systemctl reload nginx. Edit the files with an editor rather than sed, so the key never appears on a command line.

arroyo --version journalctl -u arroyo -b --no-pager -o cat | grep -E 'Starting (HTTP API|arroyo-controller|compiler)|cluster HTTP admin' | sed -E 's/^[0-9T:.-]+Z +//' | awk 'n < 4 {print; n++}'

arroyo 0.15.0 INFO arroyo_server_common: Starting cluster HTTP admin server on 127.0.0.1:5114 INFO arroyo_api: Starting HTTP API server on 127.0.0.1:5115 INFO arroyo_compiler_service: Starting compiler service at 127.0.0.1:5117 INFO arroyo_controller: Starting arroyo-controller on 127.0.0.1:5116

Path Contents

/usr/local/bin/arroyo the Arroyo 0.15.0 binary

/etc/arroyo/config.toml Arroyo configuration (no secrets)

/etc/arroyo/conf.d/90-sac-auth.toml this instance's API key (root:arroyo 0640)

/var/lib/arroyo metadata database, checkpoints, artifacts

/etc/nginx/sites-available/sac-arroyo the nginx front door

/root/arroyo-credentials.txt, /root/arroyo-curl.conf, /root/arroyo-api.curl login, API key, curl configs (root only)

/usr/share/doc/arroyo Arroyo's licences and the release provenance

Arroyo logs to the journal (journalctl -u arroyo), including every pipeline worker's log lines. nginx logs to /var/log/nginx/. Pipeline errors also appear on the pipeline's Errors tab and at /api/v1/pipelines/<id>/jobs/<job>/errors.

• Back up with Arroyo stopped. Run sudo systemctl stop arroyo, then back up /var/lib/arroyo to keep pipelines, connection profiles, UDFs and checkpoints. To keep the key and certificate, also back up /etc/arroyo/conf.d, /etc/sac and /etc/nginx/ssl. Keep your SQL and UDF source in version control as well.
• Updates. Ubuntu security updates install automatically (unattended-upgrades). Arroyo is a single binary in /usr/local/bin. To move to a newer Arroyo release, launch the newer version from the listing and recreate your pipelines there, exporting the queries from the console or the API.
• Sizing. Each running pipeline is a worker process with its own memory for state. m5.large suits development and modest streams. Use larger instances for high event rates or large windows.

• Restrict network access. Limit TCP 443 and 22 in the security group to your admin network. The console login grants code execution on the instance (Step 8), so share it only with people you would give a shell to.
• Use a trusted certificate. Serve the console under your own DNS name with a certificate from a trusted CA (Step 14). Until then, check the self-signed certificate's fingerprint (Step 4) before you sign in.
• Keep the loopback bindings and the API key. Do not change the bind-address settings or remove the key file. The engine refuses to start with a wildcard bind or without the key.
• Scope pipeline credentials. Give pipelines credentials of their own, limited to the data they need, rather than broad keys. The instance's IAM role is deliberately unreachable from the engine.
• End shared sessions. HTTP Basic authentication has no sign-out. Close the browser to end a session on a shared computer.