Deployment guide

Apache Storm

Overview

Apache Storm is the open-source distributed real-time stream computation system. It processes unbounded streams of data with low latency, guarantees that every record is processed, and scales horizontally across a cluster of worker nodes.

The SecureAnyCloud image installs Apache Storm 2.7.1 on OpenJDK 17 at /opt/storm. It runs a single-node cluster (Nimbus master, one Supervisor and the Storm UI) coordinated by a local Apache ZooKeeper. An nginx reverse proxy on TCP 80 fronts the Storm UI and protects it with HTTP Basic authentication. Storm and ZooKeeper state persist on a dedicated Azure data disk, and a unique UI password is generated on the first boot of every VM. 24/7 SecureAnyCloud support backs the image.

What you get:

Apache Storm 2.7.1 (Nimbus, Supervisor, UI) on OpenJDK 17 at /opt/storm.
A local single-node Apache ZooKeeper 3.9.3, which provides the coordination service Storm requires.
An nginx reverse proxy on :80 in front of the Storm UI (bound to loopback :8080), with HTTP Basic authentication because the Storm UI ships with no authentication of its own.
A dedicated Azure data disk at /var/lib/storm that holds storm.local.dir and the ZooKeeper data directories. The disk is separate from the OS disk and is re-provisioned with every VM.
A per-VM Storm UI password (STORM_ADMIN_PASSWORD) generated at first boot and stored in a root-only file.
zookeeper.service, storm-nimbus.service, storm-supervisor.service, storm-ui.service and nginx.service as systemd units, enabled and active.
The bundled storm-starter WordCount topology, submitted on first boot so the UI shows a running topology.
24/7 SecureAnyCloud support.

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet with a subnet in the target region. Standard_B4ms (4 vCPU / 16 GiB RAM) is a good starting point for the Storm and ZooKeeper JVMs and a single worker. Scale up for higher throughput or more worker slots.

In the NSG, allow inbound 22/tcp from your management network and 80/tcp for the Storm UI. Front the UI with TLS for public exposure (see Enabling HTTPS).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Apache Storm by SecureAnyCloud, and select Create. On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then select Review + create, followed by Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:apache-storm-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name storm \
  --image sac1647283583153:apache-storm-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name storm --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active zookeeper.service storm-nimbus.service storm-supervisor.service storm-ui.service nginx.service

All five services report active. Storm and ZooKeeper run on the JVM, and on first boot they can take a minute or two to start fully.

Step 5 - Retrieve the per-VM Storm UI password

On the first boot of every VM, apache-storm-firstboot.service generates a unique password for the Storm UI and writes it to a root-only file. Read it with:

sudo cat /root/storm-credentials.txt

The file contains the URL, the user name (admin) and the generated STORM_ADMIN_PASSWORD. Keep it safe, because it is the credential nginx checks before proxying any request to the Storm UI.

Step 6 - Open the Storm UI

Browse to http://<vm-public-ip>/. nginx prompts for HTTP Basic authentication. Sign in as admin with the password from Step 5. The Storm UI Cluster Summary shows the Storm version, Nimbus uptime, the number of supervisors, used and free worker slots, and total executors and tasks.

The Nimbus Summary and Supervisor Summary sections list the Nimbus host (leader) and the worker supervisor with its slot usage, so you can confirm the cluster is healthy at a glance.

Step 7 - Inspect the running topology

The image submits the bundled storm-starter WordCount topology on first boot, so the Topology Summary lists a running wordcount topology. Click it to open the topology view. The view shows the spouts and bolts, the stream graph, and per-component statistics (emitted, transferred, acked and failed tuples).

Drill into a spout or bolt to see its executor and task counts and its tuple-level statistics over the selected time window.

Step 8 - Submit your own topology

Storm runs topologies packaged as a JAR. Copy your topology JAR to the VM and submit it with the storm CLI as the storm service user. For example, to resubmit the bundled WordCount example under a new name:

sudo -u storm JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \
  STORM_CONF_DIR=/opt/storm/current/conf \
  /opt/storm/current/bin/storm jar \
  /opt/storm/current/examples/storm-starter/storm-starter-topologies-2.7.1.jar \
  org.apache.storm.starter.WordCountTopology demo

List the running topologies:

sudo -u storm JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \
  STORM_CONF_DIR=/opt/storm/current/conf \
  /opt/storm/current/bin/storm list

To remove a topology, use storm kill <name> with the same command prefix.

Step 9 - Where data lives

storm.local.dir and the ZooKeeper data and transaction-log directories live on the dedicated Azure data disk mounted at /var/lib/storm:

findmnt /var/lib/storm

ls /var/lib/storm

Because the data disk is captured into the image and re-provisioned with every VM, the layout is identical on each deployment. You can resize this disk independently of the OS disk from the Azure Portal.

The release on disk, the Java runtime and the live cluster summary returned by the Storm UI REST API confirm the deployment at a glance:

Step 10 - Enabling HTTPS

nginx serves the Storm UI over plain HTTP on port 80. To secure it, terminate TLS at nginx with your own certificate and domain. Install certbot, obtain a certificate, and let certbot configure nginx:

sudo snap install --classic certbot
sudo ln -sf /snap/bin/certbot /usr/bin/certbot
sudo certbot --nginx -d storm.example.com

certbot adds a listen 443 ssl server block and a redirect from port 80, and renews the certificate automatically. The HTTP Basic authentication in front of the Storm UI is preserved.

Step 11 - Maintenance

The OS receives unattended security updates. To update packages manually:

sudo apt-get update && sudo apt-get -y upgrade

To restart the Storm cluster (for example after changing storm.yaml):

sudo systemctl restart zookeeper.service storm-nimbus.service storm-supervisor.service storm-ui.service

Support

This image is maintained by SecureAnyCloud with 24/7 support. Contact sales@secureanycloud.com for help with deployment, configuration or scaling Apache Storm on Azure.

Deploy on Azure

Launch Apache Storm on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

This image runs Apache Storm, a distributed, fault-tolerant, real-time computation system. Storm processes unbounded streams of data with low latency by running topologies, which are directed graphs of spouts (sources that emit tuples) and bolts (operators that transform, aggregate or route them). It is used for real-time analytics, continuous ETL, online machine learning, distributed RPC and event processing.

What you get: a production Storm deployment has several moving parts, and this image runs them all on a single instance so the cluster is useful the moment it boots.

Nimbus: the master, or control plane. It accepts topology submissions, distributes code, assigns work and monitors for failures. Its Thrift API listens on 127.0.0.1:6627.

Supervisor: the worker daemon. It runs the topology's tasks in worker slots (this image is configured with four, on ports 6700-6703).

Storm UI: the web dashboard, which reports cluster, topology, spout and bolt statistics. It listens on 127.0.0.1:8080 and is published on port 80 by nginx.

Apache ZooKeeper: the coordination service Storm requires for cluster state. A single-node instance is bundled and binds 127.0.0.1:2181.

The Storm UI and the Nimbus Thrift API ship upstream with no authentication, and Nimbus can submit or kill topologies, which is arbitrary code execution on the cluster. Treat Nimbus as a privileged control plane and keep it private. This image does that for you: the Storm UI, Nimbus and ZooKeeper all bind to the loopback interface, a loopback-only firewall keeps their ports unreachable from off the instance regardless of your security group, and nginx publishes only the Storm UI on port 80 behind HTTP Basic authentication. The only network-reachable surfaces are SSH on port 22 and the Storm UI on port 80.

Storm's local working state (storm.local.dir), the ZooKeeper data and transaction logs, and the Storm and worker logs all live at /var/lib/storm, which is a dedicated, independently resizable EBS data volume kept off the operating system disk.

The administrator password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/storm-credentials.txt with mode 0600, so only the root user can read it. First boot also submits the bundled storm-starter WordCount topology, so the Storm UI shows a genuine running topology processing tuples immediately.

Connecting to your instance

Connect over SSH as the default login user for the operating system variant you launched. This listing currently ships the following variant:

OS variant SSH login user Notes

Ubuntu 24.04 LTS ubuntu OpenJDK 17; services zookeeper, storm-nimbus, storm-supervisor, storm-ui, nginx

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Prerequisites

Before you deploy this image you need:

An Amazon Web Services account where you can launch EC2 instances

IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products

An EC2 key pair in the target Region for SSH access to the instance

A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the Storm UI

The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Apache Storm. Select the SecureAnyCloud listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. The instance runs several Java virtual machines (Nimbus, the Supervisor, the Storm UI and ZooKeeper), so smaller burstable types are not recommended.

Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet, and either create or select a security group. The group must open port 22 from your management network and port 80 for the Storm UI. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialisation takes a short time after the instance state becomes Running and the status checks pass. During this time the image generates its administrator password and submits the demonstration topology. The Storm and ZooKeeper JVMs take a little longer to become fully ready than a typical web app.

Launch the Instance from the AWS CLI

You can launch the same image from the command line. Replace the AMI ID with the one shown on the Marketplace listing for your Region, and set your key pair, subnet and security group.

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type m5.large \
  --key-name your-key \
  --subnet-id subnet-xxxxxxxx \
  --security-group-ids sg-xxxxxxxx \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-storm}]'

Retrieve the Administrator Password

SSH into the instance as ubuntu, then read the per-instance credentials file. It is mode 0600 and owned by root, so use sudo:

sudo cat /root/storm-credentials.txt

The file contains the Storm UI URL, the administrator user (admin) and the generated password:

STORM_URL=http://<public-ip>/ STORM_ADMIN_USER=admin STORM_ADMIN_PASSWORD=<generated-per-instance>

Sign In to the Storm UI

Open your browser to the instance's public address on port 80:

http://<public-ip>/

nginx prompts for HTTP Basic authentication. Sign in as admin with the password from the credentials file. The Storm UI dashboard opens on the Cluster Summary, showing the Storm version, the number of supervisors and worker slots, and the Nimbus leader.

The full cluster view lists the Nimbus leader on port 6627, the Owner and Topology summaries, the Supervisor and its worker slots, and the complete Nimbus configuration.

Explore the Running Topology

Every instance boots with the bundled storm-starter WordCount topology already submitted. Click the wordcount topology in the Topology Summary to open its detail page. You can see the topology is ACTIVE and watch its statistics climb: the topology-wide emitted and transferred tuple counts, and the per-spout and per-bolt breakdown. WordCount is a three-stage pipeline - a spout that emits random sentences, a split bolt that splits them into words, and a count bolt that counts word occurrences - so you can see tuples flowing from stage to stage.

From this page you can also Activate, Deactivate, Rebalance or Kill the topology, and open the visualization and per-component views.

Verify the Cluster from the Command Line

You can confirm the whole stack is healthy over SSH. Check that every service is active:

systemctl is-active zookeeper storm-nimbus storm-supervisor storm-ui nginx

nginx serves an unauthenticated health endpoint used for monitoring:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/health

The Storm UI REST API is reachable through nginx with the administrator credential. A request without credentials is rejected with 401, and a request with the per-instance password returns the cluster summary as JSON. Read the generated password from the credentials file and pass it to the API:

PW=$(sudo grep '^STORM_ADMIN_PASSWORD=' /root/storm-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/api/v1/cluster/summary

The response includes the stormVersion and the supervisor and slot counts.

Submit and Manage Your Own Topologies

Storm is operated with the storm command-line tool at /opt/storm/current/bin/storm, run as the storm service user. The examples below list running topologies, submit a packaged topology jar to Nimbus, and kill a topology by name. Package your topology as a jar with its main class, copy it to the instance, then:

# Run these on your instance after connecting:
ssh ubuntu@<public-ip>
# List running topologies
sudo -u storm /opt/storm/current/bin/storm list
# Submit a topology jar to Nimbus (main class then topology name and args)
sudo -u storm /opt/storm/current/bin/storm jar /path/to/your-topology.jar com.example.MyTopology my-topology
# Kill a topology by name (Storm drains it gracefully)
sudo -u storm /opt/storm/current/bin/storm kill my-topology

Nimbus assigns the topology to worker slots on the Supervisor, and it appears in the Storm UI within a few seconds. To scale beyond a single node, add more Supervisor instances pointed at this instance's ZooKeeper and Nimbus. Contact SecureAnyCloud support for a multi-node reference architecture.

Security Model

Understanding what is exposed matters for a stream-processing control plane:

Nimbus is privileged. Anyone who can reach the Nimbus Thrift API (127.0.0.1:6627) can submit or kill topologies, which runs arbitrary code on the cluster. This image binds Nimbus to loopback and a firewall keeps port 6627 unreachable from off the instance. Do not expose it.

The Storm UI has no native authentication. It binds to 127.0.0.1:8080; nginx fronts it on port 80 with HTTP Basic authentication, and the per-instance password is generated at first boot. There is no shared or default credential in the AMI.

ZooKeeper binds 127.0.0.1:2181 and is firewalled to loopback.

Your security group should open only port 22 (from your management network) and port 80 (to your users). Never open 6627, 8080 or 2181.

For production, terminate TLS in front of the Storm UI. The simplest path is to point a DNS name at the instance and configure a certificate (for example with Let's Encrypt via certbot) on the nginx server block that currently proxies port 80 to 127.0.0.1:8080. SecureAnyCloud support can provide a tested nginx TLS configuration.

Data Volume

Storm's local state, the ZooKeeper data and transaction logs, and the Storm and worker logs live on a dedicated 30 GiB gp3 EBS volume mounted at /var/lib/storm, separate from the operating system disk and independently resizable. It is mounted by filesystem UUID with the nofail option, so the layout is reproduced on every instance launched from the image.

findmnt /var/lib/storm

To grow it, modify the EBS volume in the console, then extend the filesystem with resize2fs. Contact support if you need the exact procedure for your instance.

Backup and Maintenance

Topology code is your source of truth: keep your topology jars in version control and redeploy them with storm jar.

Operating state (which topologies are running, ZooKeeper state) lives under /var/lib/storm; snapshot the EBS volume for point-in-time recovery.

Service management uses systemctl; the units arezookeeper, storm-nimbus, storm-supervisor, storm-ui and nginx.

OS updates apply with sudo apt-get update && sudo apt-get upgrade in the usual way.

Support

Every SecureAnyCloud deployment is backed by 24/7 support via email and live chat at sales@secureanycloud.com, with a one-hour average response for critical issues. Support covers deployment and instance sizing, scaling to a multi-node Storm and ZooKeeper cluster, topology submission and parallelism tuning, Storm UI authentication and TLS termination, ZooKeeper configuration, and Apache Storm and ZooKeeper version upgrades.

Apache Storm and Apache ZooKeeper are trademarks of the Apache Software Foundation. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch Apache Storm on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS