Conduit 0.21.2

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Select Conduit 0.21.2 on Ubuntu 24.04 as the software version, pick your Region, and choose Launch through EC2. Select m5.large as the instance type (or a larger one for heavy pipelines), then choose your key pair and the security group described in Prerequisites.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=conduit}]'

First boot takes about a minute after the instance is running. The boot service generates the login, the API token and the certificate, and then Conduit and nginx start.

Version SSH user

Conduit 0.21.2 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

Check that the services are active:

systemctl is-active conduit-metadata-names conduit-firstboot conduit nginx
sudo test -f /var/lib/sac/data/conduit-firstboot.done && echo FIRSTBOOT_DONE

Confirm that all four units are active and that the first-boot marker is present:

active active active active FIRSTBOOT_DONE

Only SSH and nginx listen on network addresses. Conduit's HTTP API (8080), its gRPC API (8084) and its internal connector utilities server (a random port) listen on 127.0.0.1:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:38483 users:(("conduit")) 127.0.0.1:8080 users:(("conduit")) 127.0.0.1:8084 users:(("conduit")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx"))

The credentials file is readable by root only. It holds the URL, the admin user and password, the Conduit API token and the SHA-256 fingerprint of this instance's certificate:

sudo cat /root/conduit-credentials.txt

# Conduit 0.21.2 on Ubuntu 24.04 - generated on first boot by conduit-firstboot.service.
# The UI login and the Conduit API token are unique to this instance; none was baked into the image. Keep this file
# secret. Whoever has either of them can create pipelines, which read from and write to the systems you connect and
# make outbound connections from this instance: share them only with people you would trust with those systems.
# nginx stores the password only as a bcrypt hash (/etc/nginx/conduit-htpasswd). The UI, Swagger UI (/openapi/) and
# REST API (/v1/) are served over HTTPS only (port 443; port 80 redirects). The certificate is self signed (minted on
# this instance): before you sign in, check that the SHA-256 fingerprint your browser shows matches
# CONDUIT_TLS_CERT_SHA256 below - or install a CA-issued certificate with
#
sudo conduit-install-cert <chain.pem> <key.pem>
# Change the served name:
sudo conduit-set-hostname <dns-name> Re-verify:
sudo conduit-selfcheck CONDUIT_URL=https://<public-ip>/ CONDUIT_HOST=<public-ip> CONDUIT_UI_USER=admin CONDUIT_UI_PASSWORD=<CONDUIT_UI_PASSWORD> CONDUIT_API_TOKEN=<api-token> CONDUIT_API_TOKEN_FILE=/etc/conduit/api-token CONDUIT_TLS_CERT_SHA256=<sha256-fingerprint> CONDUIT_TLS_CERT_FILE=/etc/sac/conduit-ui.pem
# REST API over HTTPS with the token (curl reads it from a file, never the command line):
#
sudo curl -K /root/conduit-api.curl --cacert /etc/sac/conduit-ui.pem --resolve <public-ip>:443:127.0.0.1 https://<public-ip>/v1/pipelines
# CLI (gRPC on 127.0.0.1:8084):
sudo conduit pipelines ls --api.token-file /etc/conduit/api-token
# User guide: https://secureanycloud.com/stacks/conduit-0-21-2/

Print the URL, the user, the token file and the fingerprint, without the password or the token:

sudo grep -E '^CONDUIT_(URL|UI_USER|API_TOKEN_FILE|TLS_CERT_SHA256)=' /root/conduit-credentials.txt

CONDUIT_URL=https://<public-ip>/ CONDUIT_UI_USER=admin CONDUIT_API_TOKEN_FILE=/etc/conduit/api-token CONDUIT_TLS_CERT_SHA256=<sha256-fingerprint>

The host is the instance's public IP address from the EC2 instance metadata (the private address when the instance has no public one). The certificate also names the public DNS name, the private IP address and the private DNS name. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it (unless you installed your own, Step 13).

Read the certificate's names and constraints from the instance:

openssl x509 -in /etc/sac/conduit-ui.pem -noout -ext subjectAltName,basicConstraints,keyUsage,extendedKeyUsage

X509v3 Subject Alternative Name: IP Address:<public-ip>, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<private-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, IP Address:127.0.0.1, DNS:localhost X509v3 Basic Constraints: critical CA:FALSE X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication

Open the CONDUIT_URL from Step 4 in your browser. The certificate is self-signed and unique to this instance, so the browser warns about it the first time. Check the certificate before you continue: open the browser's certificate viewer for the page and compare its SHA-256 fingerprint with CONDUIT_TLS_CERT_SHA256 from the credentials file. Continue only if they match, and only for this site. Never add this certificate to an operating-system or browser trust store. To get a certificate that browsers trust without a warning, install a CA-issued one (Step 13).

The browser then asks for a user name and password. Sign in as admin with the password from the credentials file. To print just the password on the instance:

sudo awk -F= '$1=="CONDUIT_UI_PASSWORD" {print $2}' /root/conduit-credentials.txt

The Pipelines page lists every pipeline with its state (screenshot at the top of this guide). Conduit 0.21.2's UI always shows a yellow banner that says No auth configured. Upstream renders it on every page whatever the server's settings, so it does not describe this instance, where every request needs the login or the API token (Step 6 shows the refusals). Dismiss it with ×.

Select a pipeline to see its sources, processors and destinations with their record rates, and its Live record flow, which streams the records passing through each stage:

View opens a record with its key, payload, metadata and how each stage changed it:

The Swagger UI at /openapi/ (behind the same login) describes the whole REST API and lets you call it from the browser with Try it out:

HTTP Basic authentication has no sign-out button: close the browser to end the session on a shared computer.

Every API call needs this instance's API token. There are three ways to call Conduit:

• The REST API over HTTPS, from anywhere that can reach port 443, with the token as a bearer token. /root/conduit-api.curl holds it as a curl header line, so the token never appears on a command line.
• The REST API on the instance, directly on http://127.0.0.1:8080, with the same token.
• The conduit command-line tool on the instance, which talks to the gRPC API on 127.0.0.1:8084 and reads the token from /etc/conduit/api-token (--api.token-file, so the token is never in the process list).

On the instance the examples pin the instance's public name to 127.0.0.1 with --resolve, so they work whatever the security group allows, and check the server against this instance's certificate:

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem https://$H/v1/pipelines \
  | python3 -c 'import json,sys; [print(p["id"], p["state"]["status"]) for p in json.load(sys.stdin)]' curl -s -o /dev/null -w 'over HTTPS without the login or the token: %{http_code}\n' --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem https://$H/v1/pipelines
sudo conduit pipelines ls --api.token-file /etc/conduit/api-token

demo-generator-to-log STATUS_RUNNING over HTTPS without the login or the token: 401 +-----------------------+---------+----------------------+----------------------+ | ID | STATE | CREATED | LAST_UPDATED | +-----------------------+---------+----------------------+----------------------+ | demo-generator-to-log | running | 2026-10-29T16:52:19Z | 2026-10-29T16:52:19Z | +-----------------------+---------+----------------------+----------------------+

Conduit 0.21 refuses requests itself, before any of its own code runs, when the token is missing or wrong, when the Host header is not a loopback name (a DNS-rebinding attack from a web page), when the request comes from another site's page (Origin), or when a change is not sent as JSON (a cross-site form post). Each refusal carries its error code:

why() { python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["message"], "/", d["details"][0]["reason"])'; } printf 'no token: '; curl -s http://127.0.0.1:8080/v1/pipelines | why printf 'foreign Host: ';
sudo curl -s -K /root/conduit-api.curl -H 'Host: rebind.example' http://127.0.0.1:8080/v1/pipelines | why printf 'foreign Origin: ';
sudo curl -s -K /root/conduit-api.curl -H 'Origin: https://evil.example' http://127.0.0.1:8080/v1/pipelines | why printf 'form-style POST: ';
sudo curl -s -K /root/conduit-api.curl -X POST -H 'Content-Type: text/plain' --data '{}' http://127.0.0.1:8080/v1/pipelines | why printf 'gRPC without token: '; { conduit pipelines ls 2>&1 || true; } | tail -1 | sed -E 's/.*code = ([A-Za-z]+) desc = (.*)/\2 \/ gRPC \1/'

no token: missing or invalid API token / api.unauthenticated foreign Host: the Host header is not allowed for this API / api.host_forbidden foreign Origin: the request Origin is not allowed / api.origin_forbidden form-style POST: mutating requests must use Content-Type: application/json / api.content_type_unsupported gRPC without token: missing or invalid API token / gRPC Unauthenticated

From your workstation, copy the certificate (Step 4) and keep the token in a file that only you can read. Then call the API the same way:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/sac/conduit-ui.pem conduit.pem openssl x509 -in conduit.pem -noout -fingerprint -sha256
# must match CONDUIT_TLS_CERT_SHA256 ( umask 077; printf 'header = "Authorization: Bearer %s"\n' "<your-token>" > conduit-api.curl )
curl -K conduit-api.curl --cacert conduit.pem https://<public-ip>/v1/pipelines

The live record streams (/v1/connectors/<id>/inspect and /v1/processors/<id>/inspect-in or inspect-out) are WebSockets that the UI opens. A script that calls them must send Sec-Fetch-Site: same-origin, because nginx serves them only to pages of this instance.

Pipelines are declared in YAML. This one generates two order records per second, tags each with a region in its metadata and writes them to the log destination. Every pipeline should name a dead-letter queue, where failed records go. This one stops the pipeline on the first failed record (window-nack-threshold: 0) instead of acknowledging it, so nothing is lost. Conduit 0.21 refuses a dead-letter queue that would acknowledge failed records while keeping only hashes of them.

mkdir -p ~/conduit-demo && cd ~/conduit-demo cat > orders.yaml <<'YAML' version: "2.2" pipelines: - id: orders-demo status: running name: orders-demo description: Generated orders, enriched with a field, logged at debug level connectors: - id: orders-source type: source plugin: builtin:generator settings: format.type: structured format.options.order_id: int format.options.customer: string rate: "2" sdk.schema.extract.key.enabled: "false" - id: orders-log type: destination plugin: builtin:log settings: level: debug sdk.schema.extract.key.enabled: "false" processors: - id: tag-region plugin: field.set settings: field: .Metadata.region value: eu-west-1 dead-letter-queue: plugin: builtin:log settings: level: warn window-size: 1 window-nack-threshold: 0 YAML
sudo conduit pipelines apply orders.yaml --yes --api.token-file /etc/conduit/api-token

Plan for pipeline "orders-demo" (hash 35e6495d): [+] create pipeline orders-demo in place [+] create connector orders-demo:orders-demo:orders-source in place [+] create connector orders-demo:orders-demo:orders-log in place [+] create processor orders-demo:orders-demo:tag-region in place 4 to create, 0 to update, 0 to delete.

Start the pipeline, then read back its dead-letter queue setting and its record counters (Conduit shows setting values as *** in API answers):

sudo conduit pipelines start orders-demo --api.token-file /etc/conduit/api-token sleep 5
H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem https://$H/v1/pipelines/orders-demo/dead-letter-queue; echo
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem https://$H/metrics \
  | awk '/^conduit_connector_bytes_count\{.*orders-demo/ {sub(/.*component_id="/, ""); sub(/",.* /, " records="); print}'

Pipeline orders-demo [running] {"plugin":"builtin:log","settings":{"level":"***"},"windowSize":"1","windowNackThreshold":"0"} orders-demo:orders-demo:orders-log records=11 orders-demo:orders-demo:orders-source records=11

Open the pipeline in the UI to watch the records (Step 5). conduit pipelines apply keeps a pipeline in step with its file. To change the pipeline, edit the file and apply it again (stop it first if Conduit says the change needs a restart). The file is the pipeline's source of truth, so the API and the UI cannot delete it. To keep the pipeline across restarts, install the file where Conduit loads pipelines at every start. Conduit drops applied pipelines whose file is not there when it restarts:

sudo install -m 0644 orders.yaml /etc/conduit/pipelines/orders.yaml
sudo systemctl restart conduit

/etc/conduit/pipelines/demo.yaml holds the demo pipeline in the same way. To drop the demo, remove the file and restart Conduit. Pipelines you create through the REST API or the UI are kept in Conduit's database instead and can be changed and deleted through the API.

Point the built-in connectors at your systems in the same YAML (or through the API). Keep their transport encrypted and pin the server's certificate for that client only:

• Kafka (builtin:kafka): set tls.enabled: "true" and give the broker's CA certificate (or, for a self-signed broker, the broker's own certificate) in the connector's caCert setting as PEM text. Check its fingerprint with the broker's administrator first, and never set insecureSkipVerify. Conduit 0.21 refuses SASL without TLS.
• PostgreSQL (builtin:postgres): use sslmode=verify-full in the connection URL and name the CA certificate with sslrootcert. Certificate files must be in /var/lib/conduit/secrets; Conduit refuses paths anywhere else.

Copy a CA certificate into the secrets directory as the conduit user, and pin a self-signed server certificate only in that connector, never in the instance's system trust store:

openssl x509 -in db-ca.pem -noout -fingerprint -sha256
# compare with the database administrator's value sudo -u conduit tee /var/lib/conduit/secrets/db-ca.pem < db-ca.pem > /dev/null

A pipeline that streams changes from PostgreSQL into a Kafka topic (host names and credentials in angle brackets):

version: "2.2" pipelines: - id: orders-cdc status: running connectors: - id: pg-orders type: source plugin: builtin:postgres settings: url: "postgres://<user>:${PG_PASSWORD}@<db-host>:5432/<database>?sslmode=verify-full&sslrootcert=/var/lib/conduit/secrets/db-ca.pem" tables: "public.orders" - id: kafka-orders type: destination plugin: builtin:kafka settings: servers: "<broker-1>:9093,<broker-2>:9093" topic: "orders" tls.enabled: "true" caCert: | -----BEGIN CERTIFICATE----- <the broker CA certificate, PEM> -----END CERTIFICATE----- dead-letter-queue: plugin: builtin:file settings: path: /var/lib/conduit/files/orders-cdc-dlq.jsonl window-size: 1 window-nack-threshold: 0

Keep passwords out of pipeline files with ${NAME} references. Conduit expands only variables you list in env.allow, so add the name there and give Conduit the value from a root-only file:

sudo nano /etc/conduit/conduit.yaml
# env: allow: [PG_PASSWORD]
sudo install -d -m 0755 /etc/systemd/system/conduit.service.d
sudo install -m 0600 /dev/null /etc/conduit/secrets.env
# then:
sudo nano /etc/conduit/secrets.env -> PG_PASSWORD=... printf '[Service]\nEnvironmentFile=/etc/conduit/secrets.env\n' | sudo tee /etc/systemd/system/conduit.service.d/secrets.conf
sudo systemctl daemon-reload && sudo systemctl restart conduit

Anyone who can create pipelines can use a listed variable's value in a setting, so list only variables meant for pipelines. By default built-in connectors may connect to any address except the cloud metadata endpoints. To restrict them, list the allowed targets under connectors.egress.allow in /etc/conduit/conduit.yaml (for example db.internal:5432 or 10.0.0.0/16:9093) and restart Conduit.

Creating a pipeline means telling Conduit to open files and connections, so the image limits what a pipeline can reach. The file connector reads and writes only in /var/lib/conduit/files; it refuses everything else, including links and ../ paths that lead out of that directory. Inside the engine's sandbox /root and the TLS key cannot even be looked up:

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt) api() {
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem -H 'Content-Type: application/json' "$@"; }
P=$(api --data '{"config":{"name":"sandbox-files"}}' https://$H/v1/pipelines | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') for f in /root/conduit-credentials.txt /etc/conduit/api-token /etc/nginx/ssl/conduit.key /var/lib/conduit/files/../conduit.db; do api --data "{\"type\":\"TYPE_SOURCE\",\"plugin\":\"builtin:file\",\"pipelineId\":\"$P\",\"config\":{\"name\":\"f\",\"settings\":{\"path\":\"$f\"}}}" https://$H/v1/connectors \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["details"][0]["reason"], "-", d["message"])' done api -X DELETE https://$H/v1/pipelines/$P >/dev/null && echo "pipeline sandbox-files deleted"

config.path_not_allowed - path "/root/conduit-credentials.txt" is not allowed: the path cannot be resolved config.path_not_allowed - path "/etc/conduit/api-token" is not allowed: the path is outside every allowed directory config.path_not_allowed - path "/etc/nginx/ssl/conduit.key" is not allowed: the path cannot be resolved config.path_not_allowed - path "/var/lib/conduit/files/../conduit.db" is not allowed: the path is outside every allowed directory pipeline sandbox-files deleted

The built-in HTTP processors (webhook.http, the Ollama, OpenAI and Cohere processors) refuse private, loopback, link-local and cloud metadata addresses at the moment they connect, whatever the URL looks like. Here a pipeline sends its record to the instance metadata service and is stopped:

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt) api() {
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem -H 'Content-Type: application/json' "$@"; } id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
P=$(api --data '{"config":{"name":"sandbox-metadata"}}' https://$H/v1/pipelines | id) api --data "{\"type\":\"TYPE_SOURCE\",\"plugin\":\"builtin:generator\",\"pipelineId\":\"$P\",\"config\":{\"name\":\"g\",\"settings\":{\"format.type\":\"structured\",\"format.options.id\":\"int\",\"recordCount\":\"1\"}}}" https://$H/v1/connectors | id > /dev/null api --data "{\"type\":\"TYPE_DESTINATION\",\"plugin\":\"builtin:log\",\"pipelineId\":\"$P\",\"config\":{\"name\":\"l\",\"settings\":{\"level\":\"debug\"}}}" https://$H/v1/connectors | id > /dev/null api --data "{\"plugin\":\"builtin:webhook.http\",\"parent\":{\"type\":\"TYPE_PIPELINE\",\"id\":\"$P\"},\"config\":{\"settings\":{\"request.url\":\"http://169.254.169.254/latest/meta-data/\"}}}" https://$H/v1/processors | id > /dev/null api -X POST --data '{}' https://$H/v1/pipelines/$P/start > /dev/null for i in $(seq 1 30); do
E=$(api https://$H/v1/pipelines/$P | python3 -c 'import json,sys; print(json.load(sys.stdin)["state"]["error"])') case "$E" in *"egress guard"*) break ;; esac; sleep 1 done printf '%s\n' "$E" | grep -o 'outbound request refused by the egress guard.*'

outbound request refused by the egress guard (dial to 169.254.169.254:80): v4_link_local_or_metadata

Behind those checks, systemd blocks the metadata service for the whole engine (Step 11), and Conduit runs as the unprivileged conduit user that can write only its own state directory:

systemctl show -p User,NoNewPrivileges,ProtectSystem,ProtectHome,PrivateTmp,ReadWritePaths,MemoryDenyWriteExecute conduit.service

User=conduit ReadWritePaths=/var/lib/conduit PrivateTmp=yes ProtectHome=yes ProtectSystem=strict NoNewPrivileges=yes MemoryDenyWriteExecute=yes

Standalone connectors. The image runs only Conduit's built-in connectors and processors. A standalone connector is a separate program you install in /usr/local/lib/conduit/connectors (owned by root, so only an administrator can add one) and Conduit starts it as the conduit user. It gets a minimal environment instead of Conduit's, and the sandbox and metadata block apply to it too. However, it is native code that can read whatever the conduit user can read, including Conduit's configuration and API token. Install only connectors you trust, and keep secrets out of Conduit's environment unless a pipeline needs them.

Clean up the pipelines of Steps 7 and 9 (the applied orders-demo is stopped here and disappears at the next restart of Conduit unless you installed its file in Step 7):

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt) api() {
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/sac/conduit-ui.pem -H 'Content-Type: application/json' "$@"; }
ID=$(api https://$H/v1/pipelines | python3 -c 'import json,sys; print(next((p["id"] for p in json.load(sys.stdin) if p["config"]["name"] == "sandbox-metadata"), ""))') if [ -n "$ID" ]; then api -X POST --data '{"force":true}' https://$H/v1/pipelines/$ID/stop > /dev/null; sleep 2 for x in $(api https://$H/v1/pipelines/$ID | python3 -c 'import json,sys; d=json.load(sys.stdin); print(" ".join(["processors/" + i for i in d.get("processorIds") or []] + ["connectors/" + i for i in d.get("connectorIds") or []]))'); do api -X DELETE "https://$H/v1/$x" > /dev/null done api -X DELETE https://$H/v1/pipelines/$ID > /dev/null && echo "deleted sandbox-metadata" fi
sudo conduit pipelines stop orders-demo --api.token-file /etc/conduit/api-token

deleted sandbox-metadata Pipeline orders-demo [stopped]

conduit-selfcheck re-proves the whole appliance with this instance's credentials. It checks the units and the per-instance files, the served certificate, the HTTPS front door (the login and the token, unknown host names, foreign origins, WebSocket rules, form posts, ; path parameters and encoded traversal), Conduit's own refusals on 127.0.0.1:8080 and on gRPC, and the exact listening set. It then creates, runs and stops a real pipeline over HTTPS with the token, and proves the sandbox from inside pipelines: the file connector refuses the credentials file, the token and the TLS key, ${VAR} references are refused, the HTTP processor cannot reach the metadata service in any of its spellings or Conduit's own API, and from inside Conduit's systemd unit the metadata service is unreachable while loopback answers. It cleans up after itself and takes about a minute:

sudo conduit-selfcheck | tail -n 1

SELFCHECK OK

Run it without | tail to see every check.

Conduit (and so every plugin process it starts), nginx and the boot service are denied the instance metadata service (169.254.169.254) by systemd IPAddressDeny rules, so neither a pipeline nor a request to the front door can read the instance's IAM credentials. The only service allowed to reach the metadata service is conduit-metadata-names, which runs once at boot, reads the instance's host names and addresses (no credentials) for the certificate, and may reach nothing else:

for u in conduit nginx conduit-firstboot conduit-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u.service)" "$(systemctl show -p IPAddressAllow --value $u.service)"; done

conduit: deny=::ffff:168.63.129.16/128 169.254.169.254/32 ::ffff:169.254.169.254/128 fd00:ec2::254/128 168.63.129.16/32 allow= nginx: deny=169.254.169.254/32 ::ffff:168.63.129.16/128 168.63.129.16/32 fd00:ec2::254/128 ::ffff:169.254.169.254/128 allow= conduit-firstboot: deny=168.63.129.16/32 ::ffff:168.63.129.16/128 fd00:ec2::254/128 169.254.169.254/32 ::ffff:169.254.169.254/128 allow= conduit-metadata-names: deny=::/0 0.0.0.0/0 allow=fd00:ec2::254/128 169.254.169.254/32

Pipelines can still reach your sources and destinations on the network and the internet; only the metadata address is blocked. Conduit's option to let built-in connectors use the instance's IAM role (connectors.ambient-credentials.enabled) is off and could not reach the role anyway, so give the S3 connector static keys of its own, scoped to its bucket.

Many Conduit installations still run 0.17 to 0.20.5. Three upstream security advisories were fixed in 0.21 (and 0.20.6), and they change how you use Conduit:

• The API now needs a token (GHSA-v79j-j88p-m4ff). Before, anyone who could reach Conduit's API could create and run pipelines, and a web page you visited could do the same through your browser. Now every call carries the API token, the API listens on loopback by default, and requests from other sites or with a rebound host name are refused. On this image you send the token yourself or sign in at nginx, which adds it.
• Pipeline settings can no longer read the engine's secrets or files (GHSA-m3mw-fpw8-w765). Before, a setting could pull any environment variable of the engine into a connector (${AWS_SECRET_ACCESS_KEY}), point the file connector at any file, or make an HTTP processor call internal addresses. Now ${NAME} works only for names you list in env.allow, file paths are confined to allowed directories, and built-in connectors and processors refuse metadata and (for processors) private addresses.
• Plugins no longer inherit the engine's environment (GHSA-gwvc-9h8p-x5r7). Standalone connectors now get a minimal environment. If they need a variable, add it under connectors.standalone.env.allow.

Two more 0.21 changes affect existing pipeline files: every pipeline needs a dead-letter queue that cannot lose a record (the default one stops the pipeline at the first failure, as in Step 7), and the log connector writes only hashes of records unless you set payload: "true". To move pipelines from an older Conduit, check those two points and any${NAME} references, and then re-create the pipelines from their YAML on this instance.

To serve Conduit under your own DNS name, point the name at the instance (an Elastic IP is recommended) and set it. This writes the name into nginx and issues a new self-signed certificate that covers it:

sudo conduit-set-hostname conduit.example.com

To install a certificate from your CA (or Let's Encrypt), copy the full chain (server certificate first) and the unencrypted private key to the instance, and then run:

sudo conduit-install-cert /path/to/fullchain.pem /path/to/privkey.pem conduit.example.com

The boot-time renewal never replaces a certificate installed this way. sudo conduit-set-hostname <name> goes back to a self-signed certificate. Both commands update the URL and fingerprint in /root/conduit-credentials.txt.

The password is stored as a bcrypt hash in /etc/nginx/conduit-htpasswd. To set a new one without putting it on a command line, let htpasswd prompt for it, then update the curl config and the credentials file:

sudo htpasswd -B -C 10 /etc/nginx/conduit-htpasswd admin
sudo nano /root/conduit-curl.conf /root/conduit-credentials.txt
sudo systemctl reload nginx

To rotate the API token, write a new 64-character hex token (for example from openssl rand -hex 32) into /etc/conduit/api-token, into the default line of /etc/nginx/conduit-upstream-auth.conf, into /root/conduit-api.curl and the credentials file. Then run sudo systemctl restart conduit and sudo systemctl reload nginx. Edit the files with an editor rather than sed, so the token never appears on a command line. Conduit reads the token once at start.

conduit --version journalctl -u conduit -b --no-pager -o cat | grep -E 'grpc API started|API token authentication|http API started|pipeline configs provisioned' | sed -E 's/^[0-9T:+-]+ //' | awk '!seen[$0]++' | awk 'n < 4 {print; n++}'

v0.21.2 linux/amd64 INF pipeline configs provisioned component=provisioning.Service created=["demo-generator-to-log"] deleted=[] pipelines_path=/etc/conduit/pipelines INF grpc API started address=127.0.0.1:8084 INF API token authentication is enabled for the HTTP and gRPC APIs INF http API started address=127.0.0.1:8080

Path Contents

/usr/local/bin/conduit the Conduit 0.21.2 binary (server and CLI)

/etc/conduit/conduit.yaml Conduit configuration (no secrets; sudo conduit config --config.path /etc/conduit/conduit.yaml prints the effective values)

/etc/conduit/api-token this instance's API token (root:conduit 0640)

/etc/conduit/pipelines/ pipeline files Conduit loads at every start (the demo)

/var/lib/conduit the pipeline database, plus files/ (the file connector's directory) and secrets/ (certificate files for connectors)

/etc/nginx/sites-available/sac-conduit the nginx front door

/root/conduit-credentials.txt, /root/conduit-curl.conf, /root/conduit-api.curl login, API token, curl configs (root only)

/usr/share/doc/conduit Conduit's licence and the release provenance

Conduit logs to the journal (journalctl -u conduit), including pipeline errors; nginx logs to /var/log/nginx/. A pipeline's last error is also in GET /v1/pipelines/<id> (state.error) and on its page in the UI.

• Back up /var/lib/conduit with Conduit stopped (sudo systemctl stop conduit) to keep API-created pipelines and connector positions, plus /etc/conduit (your pipeline files and settings). Keep your pipeline YAML in version control as well.
• Ubuntu security updates install automatically (unattended-upgrades). Conduit is a single binary in /usr/local/bin. To move to a newer Conduit release, launch the newer version from the listing and apply your pipeline files there.
• Size the instance for your pipelines. m5.large suits development and modest streams, and larger instances suit high record rates or many pipelines.

• Limit TCP 443 and 22 in the security group to your admin network. The login and the API token create pipelines that reach your systems, so share them only with people you would trust with those systems.
• Serve Conduit under your own DNS name with a certificate from a trusted CA (Step 13). Until then, check the self-signed certificate's fingerprint (Step 4) before you sign in.
• Keep the API on the loopback address with the token file in place. Conduit refuses to start if either API is bound to another address, the token file is missing or readable by others, or a wildcard browser origin is allowed.
• List in env.allow and connectors.egress.allow only what your pipelines need, give connectors credentials of their own scoped to their data, and install standalone connectors only from sources you trust.