Conduit data streaming

Search the Azure Marketplace for Conduit, choose the plan, and create the VM.

In the networking step, attach a network security group (NSG) that allows inbound TCP 22 (SSH) and TCP 80 (the API and UI, behind HTTP Basic Auth) from your client networks only. For production, put a TLS reverse proxy or certificate in front of port 80.

Run the following to create the network, accept the Marketplace terms and create the VM. Replace the angle-bracket value with your management network CIDR.

RG="conduit-prod"; LOCATION="eastus"; VM_NAME="conduit-01" IMAGE_URN="sac1647283583153:conduit-data-streaming-ubuntu-24-04:default:latest" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create -g "$RG" --name cdt-vnet --address-prefix 10.109.0.0/16 --subnet-name cdt-subnet --subnet-prefix 10.109.1.0/24 az network nsg create -g "$RG" --name cdt-nsg az network nsg rule create -g "$RG" --nsg-name cdt-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name cdt-nsg --name allow-web --priority 110 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 80 --access Allow --protocol Tcp
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:conduit-data-streaming-ubuntu-24-04:default:latest
az vm create -g "$RG" --name "$VM_NAME" --image "$IMAGE_URN" \
  --size Standard_B2s --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name cdt-vnet --subnet cdt-subnet --nsg cdt-nsg --public-ip-sku Standard

Connect to the VM as azureuser:

ssh azureuser@<vm-ip>

conduit.service, nginx.service and conduit-data-streaming-firstboot.service all start automatically on first boot.

On first boot, the appliance generates a random HTTP Basic Auth password for user admin, unique to this VM, and writes it to a root-only file. There is no shared or default credential in the image. Read the file over SSH:

sudo cat /root/conduit-data-streaming-credentials.txt

The file holds this VM's URL, username and password:

CONDUIT_URL=http://<vm-ip>/ CONDUIT_USERNAME=admin CONDUIT_PASSWORD=<unique-per-vm-password>

Keep this password safe. It is the only credential for the API and web UI, and every VM gets a different one.

Check that the services are active, that first boot has finished, and which ports are listening:

sudo systemctl is-active conduit nginx conduit-data-streaming-firstboot
sudo test -f /var/lib/sac/data/conduit-data-streaming-firstboot.done && echo FIRSTBOOT_DONE
sudo ss -tlnp | grep -E ':80 |:8080 |:8084 '

Expect all three services to be active. nginx is bound to the public port 80, and the Conduit APIs are bound to loopback 127.0.0.1:8080 and 127.0.0.1:8084 only:

active active active FIRSTBOOT_DONE LISTEN 0 4096 127.0.0.1:8080 0.0.0.0:* users:(("conduit",...)) LISTEN 0 4096 127.0.0.1:8084 0.0.0.0:* users:(("conduit",...)) LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",...))

The Conduit API and UI have no built-in login, so nginx fronts them with an HTTP Basic Auth gate. An unauthenticated /healthz is available for load balancer probes, but the API and UI are gated. Requests with no credentials or a wrong password are rejected with 401, and only the per-VM password is accepted.

Read your password from the file and test the gate:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'GET /healthz (unauth) -> HTTP %{http_code}\n' http://127.0.0.1/healthz
curl -s -o /dev/null -w 'GET /v1/pipelines (no cred) -> HTTP %{http_code}\n' http://127.0.0.1/v1/pipelines
curl -s -o /dev/null -w 'GET /v1/pipelines (wrong pw) -> HTTP %{http_code}\n' -u admin:wrong http://127.0.0.1/v1/pipelines
curl -s -o /dev/null -w 'GET /v1/pipelines (per-VM pw) -> HTTP %{http_code}\n' -u "admin:$PW" http://127.0.0.1/v1/pipelines

The health probe is open, the API is gated, a wrong password is rejected, and the per-VM password authenticates:

GET /healthz (unauth) -> HTTP 200 GET /v1/pipelines (no cred) -> HTTP 401 GET /v1/pipelines (wrong pw) -> HTTP 401 GET /v1/pipelines (per-VM pw) -> HTTP 200

Open http://<vm-ip>/openapi/ in a browser and sign in as admin with the password from Step 4.

Conduit serves an interactive OpenAPI/Swagger UI that documents every endpoint of the HTTP API. You can call the API directly from the browser to list pipelines and connectors, create and start pipelines, and inspect the running engine.

Expand GET /v1/pipelines in the Pipelines service. Swagger UI shows the operation's parameters and response schema.

Click Try it out, then Execute, to call the live API from the browser.

The response from Execute shows the running demo pipeline, demo-generator-to-log. Its state is STATUS_RUNNING and its two connectors are listed.

This is the self-contained generator-to-log pipeline that ships in the image. It shows that the engine is streaming records out of the box.

Expand GET /v1/connectors and execute it the same way. It lists the demo pipeline's connectors: a builtin:generator source that produces sample records, and a builtin:log destination that logs them locally.

Each connector shows its plugin, type and settings.

Everything the UI shows is available as JSON from the HTTP API, so you can wire Conduit's state into your own tooling. Read your password from the file and query the pipelines:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/v1/pipelines \
  | jq '[.[] | {id: .id, status: .state.status, connectors: .connectorIds}]'

The demo pipeline reports as running with its two connectors:

[ { "id": "demo-generator-to-log", "status": "STATUS_RUNNING", "connectors": [ "demo-generator-to-log:source-generator", "demo-generator-to-log:destination-log" ] } ]

The demo pipeline is defined in one declarative file, /etc/conduit/pipelines/demo.yaml. Check the Conduit version and view the config:

conduit version cat /etc/conduit/pipelines/demo.yaml

The config wires a builtin:generator source to a builtin:log destination:

v0.17.0 linux/amd64 version: 2.2 pipelines: - id: demo-generator-to-log status: running ... connectors: - id: source-generator type: source plugin: builtin:generator ... - id: destination-log type: destination plugin: builtin:log

Replace the demo pipeline with your own. Add or edit a pipeline file under /etc/conduit/pipelines/. This example streams from a Postgres source to a Kafka destination:

version: 2.2 pipelines: - id: pg-to-kafka status: running connectors: - id: source-pg type: source plugin: builtin:postgres settings: url: "postgres://user:pass@db-host:5432/mydb?sslmode=require" tables: "public.orders" - id: dest-kafka type: destination plugin: builtin:kafka settings: servers: "broker:9092" topic: "orders"

Apply your changes with sudo systemctl restart conduit. Conduit provisions the pipelines from the files on start.

The full connector and processor reference is at conduit.io/docs.

Conduit is managed by systemd, so it restarts automatically on failure and on boot. Check its state:

sudo systemctl show conduit --property=ActiveState,SubState,MainPID,NRestarts

A healthy service reports:

ActiveState=active SubState=running MainPID=... NRestarts=0

Restart it after editing your pipelines with sudo systemctl restart conduit. Read the logs, including the demo pipeline's record output, with sudo journalctl -u conduit -f.

For production, terminate TLS at nginx. Point a DNS record at the VM, then use the packaged nginx with a certificate from Let's Encrypt:

sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d <your-domain>

certbot installs the certificate, rewrites the nginx server block to listen on 443, and sets up automatic renewal. The HTTP Basic Auth gate stays in front of the API and UI over TLS.

• Restrict the NSG. Let ports 80 and 22 reach trusted networks only.
• Terminate TLS. Terminate TLS at nginx (Step 15) so the API, UI and Basic Auth credentials travel encrypted.
• Rotate the Basic Auth password. If needed, regenerate /etc/nginx/.conduit.htpasswd with sudo htpasswd -B /etc/nginx/.conduit.htpasswd admin and reload nginx.
• Keep the API private. The Conduit API can create, start and stop pipelines, so treat access as privileged even behind Basic Auth.
• Review your connectors. Make sure data flows only to the sources and destinations you intend, and keep connector credentials in the pipeline settings secured.
• Patch the OS monthly. Run sudo apt-get update && sudo apt-get upgrade. Unattended security upgrades are already enabled.

Conduit is licensed under the Apache License 2.0, with no per-CPU or per-user fee. SecureAnyCloud provides commercial support separately.

• Email: sales@secureanycloud.com
• Website: secureanycloud.com
• Support hours: 24/7, 24h response SLA

Deploy on Azure

Launch Conduit on Ubuntu 24.04 with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7. sales@secureanycloud.com

Deploy on Azure

Launch Conduit on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Conduit. Select the SecureAnyCloud listing, choose Select, then choose Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet. Create or select a security group that allows inbound port 22 from your management network and inbound port 80 from the networks your operators use. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialisation takes well under a minute after the instance state becomes Running and the status checks pass.

Run the following command to launch an instance from the SecureAnyCloud Conduit Marketplace AMI into an existing subnet and security group.

Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --metadata-options 'HttpTokens=required,HttpEndpoint=enabled' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=conduit}]'

The Basic Auth password is generated uniquely on the first boot of your instance and written to a root-only file. Read the file over SSH:

sudo cat /root/conduit-data-streaming-credentials.txt

The file contains the web URL, the username (admin) and the per-instance password. Keep it secret.

Confirm the installed version and that both services are running:

conduit version; systemctl is-active conduit nginx

You should see v0.17.0 and both services reported as active.

Confirm that Conduit is bound to loopback only and that nginx is the sole network-facing listener, on port 80:

sudo ss -tlnp | grep -E ':(80|8080|8084)\b'

conduit listens on 127.0.0.1:8080 (HTTP API) and 127.0.0.1:8084 (gRPC), loopback only, while nginx listens on 0.0.0.0:80. The Conduit API is never reachable off the instance except through the nginx Basic Auth gate.

Next, confirm that the unauthenticated health endpoint answers and that the API itself is password-protected. An unauthenticated request to the API returns 401:

curl -s -o /dev/null -w 'healthz=%{http_code}\n' http://127.0.0.1/healthz
curl -s -o /dev/null -w 'api-unauth=%{http_code}\n' http://127.0.0.1/v1/pipelines

The health endpoint returns healthz=200 and the unauthenticated API request returns api-unauth=401. Every API call must present the per-instance password.

Read the per-instance password from the credentials file into a shell variable, then list the pipelines through the authenticated API:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/v1/pipelines | jq '.[] | {id, status: .state.status}'

You will see the demo-generator-to-log pipeline with status STATUS_RUNNING.

To confirm that records are flowing, read Conduit's Prometheus metrics. The pipeline execution counter is greater than zero:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/metrics | awk '/^conduit_pipeline_execution_duration_seconds_count/{s+=$NF} END{print "records processed:", s+0}'

The count is a positive, growing number, which shows that the engine is processing the demo records out of the box.

Browse to http://<instance-public-ip>/ and sign in with the username admin and the password from /root/conduit-data-streaming-credentials.txt.

You reach the Conduit OpenAPI/Swagger UI, a browser front end for the full HTTP API. From it you can list, create, start, inspect and delete pipelines and connectors.

Expand GET /v1/pipelines, then choose Try it out and Execute to see the live demo pipeline. The 200 response shows demo-generator-to-log in STATUS_RUNNING with its source and destination connectors. It is a real pipeline processing records, with nothing leaving the instance.

Expand GET /v1/connectors/plugins and execute it to browse the built-in connector catalog. It lists Kafka, PostgreSQL, Amazon S3, files and more: the sources and destinations you can wire into your own pipelines.

Pipelines are defined as YAML files under /etc/conduit/pipelines. The shipped demo.yaml is a self-contained example. Replace it with your own source and destination.

For instance, to stream from Apache Kafka into PostgreSQL, write a pipeline that uses the builtin:kafka source and a builtin:postgres destination. Edit the pipeline file with your preferred editor, for example sudo nano /etc/conduit/pipelines/demo.yaml, then apply it by restarting Conduit:

sudo systemctl restart conduit

Conduit reloads every pipeline under /etc/conduit/pipelines on start.

You can also create and start pipelines dynamically through the HTTP API (POST /v1/pipelines, POST /v1/connectors, POST /v1/pipelines/{id}/start), directly from the Swagger UI and without editing files.

For production, put Conduit behind a real domain and a certificate from a trusted authority, so that the Basic Auth password and API traffic are encrypted in transit.

Point a DNS record at your instance and open port 443 in your security group. Then obtain a certificate, for example with certbot for your-domain.example.com. Add a TLS server block to the nginx site at /etc/nginx/sites-available/sac-conduit that proxies to the same 127.0.0.1:8080 upstream. Reload nginx after any change:

sudo nginx -t && sudo systemctl reload nginx