Apache StreamPipes

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region. Standard_B8ms (8 vCPU / 32 GiB RAM) is the recommended starting size, because the multi-container StreamPipes stack needs the memory.

In the NSG, allow inbound 22/tcp from your management network and 80/tcp. StreamPipes serves plain HTTP on port 80. For production, terminate TLS in front of it with your own domain.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, select Create a resource, and search the Marketplace for Apache StreamPipes by SecureAnyCloud. Select Create.

On the Basics tab, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab. Then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:apache-streampipes-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name streampipes \
  --image sac1647283583153:apache-streampipes-ubuntu-24-04:default:latest \
  --size Standard_B8ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name streampipes --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the stack is running

The StreamPipes containers run under Docker, fronted by nginx. On first boot, the stack seeds the per-VM administrator and starts. The multi-container stack can take a few minutes to become fully healthy after the VM is created.

systemctl is-active docker.service nginx.service

Both report active. You can list the running containers with:

sudo docker compose -f /opt/streampipes/compose/docker-compose.yml ps

The whole stack is also managed as a single systemd unit. Use sudo systemctl status streampipes and sudo systemctl restart streampipes to control every container together.

Step 5 - Retrieve your per-VM admin credentials

The administrator password is generated uniquely on the first boot of your VM and written to a root-only file:

sudo cat /root/streampipes-credentials.txt

This file contains STREAMPIPES_ADMIN_EMAIL (admin@sac.local) and STREAMPIPES_ADMIN_PASSWORD. These are the credentials for the web UI and the REST API. Store them in a safe place.

Step 6 - Confirm the health endpoint

nginx exposes an unauthenticated static health endpoint for load balancers and probes, and proxies everything else to the StreamPipes web UI:

curl -s http://localhost/health

It returns ok. The web UI is reachable through the same nginx on port 80:

curl -s -o /dev/null -w 'UI -> HTTP %{http_code}\n' http://localhost/

It returns HTTP 200.

Step 7 - Confirm the admin login API

The StreamPipes backend exposes a login endpoint that returns a JWT token for valid credentials and rejects invalid ones. Substitute the password from Step 5 for <STREAMPIPES_ADMIN_PASSWORD>, then confirm that a correct password is accepted and a wrong one is rejected:

curl -s -o /dev/null -w 'login -> HTTP %{http_code}\n' \
  -X POST -H 'Content-Type: application/json' \
  -d '{"username":"admin@sac.local","password":"<STREAMPIPES_ADMIN_PASSWORD>"}' \
  http://localhost/streampipes-backend/api/v2/auth/login

A correct password returns HTTP 200, and a wrong password returns HTTP 401. This confirms that the per-VM administrator was seeded and that authentication is enforced.

Step 8 - Sign in to the web UI

Browse to http://<vm-public-ip>/ and sign in with the email admin@sac.local and the password from Step 5. The StreamPipes web UI is where you connect data, build pipelines and explore your streams.

• Pipelines is where you create, start, stop and monitor your data pipelines and reusable functions.
• StreamPipes Connect is where you add data sources. Choose a built-in adapter (MQTT, OPC-UA, Kafka, HTTP, files and many more from the IIoT extensions library) to ingest a live stream.
• Dashboards let you visualise the output of your running pipelines with live widgets: charts, gauges, tables and maps.

Step 9 - Confirm data lives on the dedicated disk

Docker's data-root is relocated onto the dedicated Azure data disk. Every container image and volume, including the CouchDB user and metadata store, the InfluxDB time-series data and the backend assets, lives there and survives OS changes:

findmnt /var/lib/streampipes

A separate Azure data disk backs the mount. It is captured into the image and re-provisioned on every VM. To confirm that Docker is using it, run:

sudo docker info -f '{{.DockerRootDir}}'

It prints /var/lib/streampipes/docker.

Building your first pipeline

StreamPipes turns raw streams into actionable insight without code. In the web UI:

• Open Connect and add an adapter for your data source, for example an MQTT topic or an OPC-UA server.
• Open Pipelines and drag your new stream onto the canvas. Then chain data processors (filters, aggregations, enrichments) and a sink (store to the built-in time-series database, forward to a broker, or trigger a notification).
• Start the pipeline and open a Dashboard to watch the live output.

See the Apache StreamPipes documentation for the full catalogue of adapters, processors and sinks.

Maintenance

• Admin credentials: the per-VM administrator is in /root/streampipes-credentials.txt. Change the password from the user menu in the web UI after your first sign-in.
• Data: all container volumes live on the /var/lib/streampipes data disk (Docker data-root). Snapshot or resize that disk for backups.
• Stack: manage the whole stack with sudo systemctl <status|restart|stop> streampipes, or manage individual containers with sudo docker compose -f /opt/streampipes/compose/docker-compose.yml <ps|logs|restart>.
• TLS: StreamPipes serves plain HTTP on port 80. Front it with TLS (for example with certbot) and your own domain before you run it in production.
• Security patches: unattended-upgrades remains enabled so the OS continues to receive security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.

Deploy on Azure

Launch Apache StreamPipes on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target region for SSH access
• A security group allowing inbound TCP 22 from your management network and inbound TCP 80 from the clients that will use the interface

The recommended instance type is m5.large (2 vCPU, 8 GiB RAM). The six-container stack idles at roughly 2 GiB of memory, so m5.large leaves comfortable headroom for pipelines and adapters. Increase the instance size if you plan to run many concurrent pipelines or high-throughput adapters.

StreamPipes serves plain HTTP on port 80. For production, terminate TLS in front of it with your own domain, as described under Enabling HTTPS below.

Step 1 - Launch from the AWS Marketplace

Sign in to the AWS Marketplace and search for Apache StreamPipes by SecureAnyCloud. Choose Continue to Subscribe, then Continue to Configuration. Select the software version and your region, then choose Continue to Launch.

Choose Launch through EC2 for full control over networking and storage. Pick the m5.large instance type, select your key pair, choose a VPC and subnet, and attach a security group that allows inbound TCP 22 and TCP 80. The dedicated 60 GiB data volume is part of the image and appears automatically under Storage. Review the settings and choose Launch instance.

Step 2 - Launch from the AWS CLI

Replace the AMI ID, key name, security group and subnet with your own values.

aws ec2 run-instances \
  --image-id <ami-id-from-the-marketplace-listing> \
  --instance-type m5.large \
  --key-name <your-key-pair> \
  --security-group-ids <your-security-group-id> \
  --subnet-id <your-subnet-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=streampipes}]'

First boot takes a few minutes. The one-shot initialisation service generates this instance's secrets, provisions the administrator against an empty user store, and waits for the backend to report healthy before it finishes.

Connecting to your instance

Connect over SSH on port 22 using the key pair you selected at launch. The login user depends on the operating system variant you launched:

• Ubuntu 24.04 LTS: the SSH login user is ubuntu

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Step 3 - Retrieve the administrator credentials

The administrator email, password and interface URL are written to a root-only file on first boot. Read it over SSH:

sudo cat /root/streampipes-credentials.txt

The file records STREAMPIPES_URL, STREAMPIPES_ADMIN_EMAIL and STREAMPIPES_ADMIN_PASSWORD. The administrator email is admin@sac.local. Treat the password as a secret, and change it from the interface after you sign in.

To confirm that the file exists and is protected without displaying its contents, run:

sudo test -f /root/streampipes-credentials.txt && echo "credentials file present (mode $(sudo stat -c %a /root/streampipes-credentials.txt), owner $(sudo stat -c %U /root/streampipes-credentials.txt))"

credentials file present (mode 600, owner root)

To verify that the generated password authenticates and that a wrong one is refused, without printing the secret, run:

sudo bash -c 'PW=$(grep "^STREAMPIPES_ADMIN_PASSWORD=" /root/streampipes-credentials.txt | cut -d= -f2-); [ -n "$PW" ] && echo "password present, length ${#PW}"; curl -s -o /dev/null -w "correct password -> HTTP %{http_code}\n" -X POST -H "Content-Type: application/json" -d "{\"username\":\"admin@sac.local\",\"password\":\"$PW\"}" http://127.0.0.1/streampipes-backend/api/v2/auth/login; curl -s -o /dev/null -w "wrong password -> HTTP %{http_code}\n" -X POST -H "Content-Type: application/json" -d "{\"username\":\"admin@sac.local\",\"password\":\"definitely-wrong\"}" http://127.0.0.1/streampipes-backend/api/v2/auth/login'

password present, length 24 correct password -> HTTP 200 wrong password -> HTTP 401

Step 4 - Sign in to the web interface

Browse to http://<public-ip>/ and sign in with the administrator email and the password from the credentials file. Because the administrator is provisioned automatically on first boot, you are taken straight to the sign-in page rather than an installation wizard.

After you sign in, you land on the home view. It summarises your pipelines, adapters, dashboards and charts, and links to each StreamPipes module.

The first time you sign in, a welcome dialog offers an interactive tour. Choose Start Tour to take it, or Do not show again to dismiss it permanently.

Step 5 - Connect a data source

Choose Connect in the left sidebar, then choose New adapter. The bundled industrial IoT extensions library supplies adapters for a wide range of protocols and systems, including Apache Kafka, Apache Pulsar, Apache RocketMQ, MQTT, OPC UA, BACnet/IP, Allen Bradley ETH and generic HTTP and file sources.

Pick the adapter that matches your source, choose Create, and work through the configuration wizard. StreamPipes samples the incoming data and infers a schema. You can adjust field names, data types and units before you save. Once saved, the adapter produces a data stream that any pipeline can consume.

Step 6 - Build a pipeline

Choose Pipelines & Functions in the sidebar, then choose New Pipeline. The editor shows your data streams and the catalogue of data processors and sinks on the left, and a canvas on the right.

Drag a data stream onto the canvas. Drag one or more processors after it, such as a filter, aggregation or boolean rule, and finish with a sink such as the live dashboard, the data lake or a notification. Connect the elements and configure each one in the dialog that opens. Then choose Save and start the pipeline. Live results appear under Dashboard, and historical data appears under Data Explorer.

Verifying the deployment

Confirm that the three services that make up the deployment are running:

systemctl is-active docker.service nginx.service streampipes.service

active active active

Confirm that the unauthenticated health endpoint responds. This is the check to point a load balancer at:

curl -s -o /dev/null -w 'health endpoint: HTTP %{http_code}\n' http://127.0.0.1/health

health endpoint: HTTP 200

Confirm the StreamPipes release that this image ships:

grep '^SP_VERSION=' /opt/streampipes/compose/.env

SP_VERSION=0.98.0

Storage layout

The whole StreamPipes footprint lives on a dedicated gp3 data volume mounted at /var/lib/streampipes, separate from the operating system disk and independently resizable. Both the Docker data root and the containerd image store are relocated there before any container image is pulled, so the container image layers, the CouchDB user and metadata store, the InfluxDB time-series data and the backend assets all sit on that volume.

findmnt -no SOURCE,TARGET,FSTYPE /var/lib/streampipes; df -h --output=target,size,used,avail /var/lib/streampipes | tail -1

/dev/nvme1n1 /var/lib/streampipes ext4 /var/lib/streampipes 59G 2.0G 54G

The mount is recorded in /etc/fstab by filesystem UUID rather than by device name, so it survives reboots and instance type changes. To grow the volume, expand it in the EC2 console. Then run sudo resize2fs /dev/nvme1n1 on the instance.

Operating the stack

The six containers are managed as one systemd unit. The compose project lives at /opt/streampipes/compose.

• Check status: systemctl status streampipes
• Start the stack: sudo systemctl start streampipes
• Stop the stack: sudo systemctl stop streampipes
• Restart the stack: sudo systemctl restart streampipes
• Inspect a container's recent output: sudo docker logs --tail 50 streampipes-backend-1

Stopping streampipes.service brings the whole stack down cleanly, and starting it brings every container back up. All state is on the data volume, so stopping and starting the stack, or rebooting the instance, does not lose pipelines, adapters or recorded data.

Enabling HTTPS

The image serves plain HTTP on port 80. For production, put your own certificate in front of it. Point a DNS record at the instance, open TCP 443 in the security group, then install certbot and obtain a certificate for your domain:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

Certbot edits the nginx site at /etc/nginx/sites-available/sac-streampipes in place, adds the TLS listener and sets up automatic renewal. The proxy headers already present in that site are the ones StreamPipes needs, so no further changes are required.

Alternatively, terminate TLS at an Application Load Balancer with a certificate from AWS Certificate Manager, and restrict the instance security group so port 80 only accepts traffic from the load balancer.

Backup and maintenance

Everything worth preserving is on the data volume at /var/lib/streampipes, so the simplest backup is a scheduled EBS snapshot of that volume, which you can automate with Amazon Data Lifecycle Manager. Take snapshots with the stack stopped, or accept crash-consistent snapshots for a running stack.

Keep the operating system patched with sudo apt-get update && sudo apt-get upgrade. To move to a newer StreamPipes release, edit SP_VERSION in /opt/streampipes/compose/.env, then restart the stack with sudo systemctl restart streampipes. Review the upstream release notes first, because some releases require migration steps.

Troubleshooting

The interface does not load. Check that the security group allows inbound TCP 80 from your address, then confirm the health endpoint responds locally with the check under Verifying the deployment. If the health endpoint returns 200 but the interface does not load, the nginx proxy is up but the interface container is not ready; give it a minute and check systemctl status streampipes.

The credentials file only contains a comment. First boot has not finished. The initialisation service waits for the backend to report healthy, which takes a few minutes on first launch. Check its progress with systemctl status apache-streampipes-firstboot, then read the file again.

Sign-in is rejected. Confirm you are using admin@sac.local and the password from /root/streampipes-credentials.txt on that specific instance. Each instance generates its own password, so a password from another instance will not work.

An adapter or pipeline will not start. Check the backend and extensions containers are running with systemctl status streampipes, then inspect the relevant container's recent output with sudo docker logs --tail 50 streampipes-extensions-all-iiot-1.

Support

This image is published and supported by SecureAnyCloud with 24/7 technical support by email and chat, covering deployment, adapter and pipeline configuration, sizing, persistence and backup, TLS termination and upgrades.

Apache, Apache StreamPipes, StreamPipes and the Apache StreamPipes logo are trademarks of The Apache Software Foundation. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch Apache StreamPipes on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com