Apache Flink 2

Search for Apache Flink 2 Ubuntu 24.04 in the Azure Marketplace, select the SecureAnyCloud entry and choose Create. On the Basics tab, choose SSH public key authentication with azureuser as the user name. On the Networking tab, allow SSH (22) from your address and HTTP (80) from the client networks that need the Web Dashboard.

First boot generates the VM's Web Dashboard password within seconds of the VM starting. The first-boot service itself finishes about three minutes after boot.

RG="flink-prod"; LOCATION="eastus"; VM_NAME="flink-01" IMAGE_URN="sac1647283583153:apache-flink-2-ubuntu-24-04:default:latest" CLIENT_CIDR="<your-mgmt-cidr>" az group create --name "$RG" --location "$LOCATION" az network nsg create -g "$RG" --name flink-nsg az network nsg rule create -g "$RG" --nsg-name flink-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "$CLIENT_CIDR" --destination-port-ranges 22 --access Allow --protocol Tcp az network nsg rule create -g "$RG" --nsg-name flink-nsg --name allow-flink-dashboard --priority 110 \
  --source-address-prefixes "$CLIENT_CIDR" --destination-port-ranges 80 --access Allow --protocol Tcp
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn "$IMAGE_URN"
az vm create -g "$RG" --name "$VM_NAME" --image "$IMAGE_URN" \
  --size Standard_B2s --admin-username azureuser --ssh-key-values ~/.ssh/id_rsa.pub \
  --nsg flink-nsg --public-ip-sku Standard \
  --query publicIpAddress -o tsv

The last command in the block prints the VM's public IP address.

ssh azureuser@<vm-ip>

First boot finishes about three minutes after the VM starts. If you connect as soon as the VM shows as running, confirm that the per-VM setup has finished before you continue:

bash systemctl is-active flink-firstboot.service

It prints activating while first boot runs and active once it has finished.

cd /tmp
sudo systemctl is-active flink-jobmanager.service flink-taskmanager.service nginx.service
sudo -u flink /opt/flink/bin/flink --version

Run the Flink CLI as the flink service user, from a directory that user can read, such as /tmp. As azureuser, the CLI cannot write its client log under /opt/flink/log and prints a Java logging stack trace instead of the version.

curl -sf http://127.0.0.1:8081/v1/overview | head -c 400

The cluster overview returns flink-version, the number of registered TaskManagers, the total task slots, and counters for running, finished, cancelled and failed jobs.

sudo cat /stage/scripts/flink-credentials.log

Note the values of FLINK_WEB_URL, FLINK_WEB_USER and FLINK_WEB_PASSWORD. The password is a fresh 32-character hex string generated at first boot, so no two VMs share credentials.

Browse to http://<vm-ip>/ and sign in as SecureAnyCloud with the password from Step 6. The Cluster Overview tile shows the current task slot occupancy, the number of running jobs, and quick links to Jobs, Task Managers, Job Manager and Submit New Job.

The Task Managers page lists every registered TaskManager with its data port, JVM heap, network buffers and last heartbeat. The SecureAnyCloud image starts with one TaskManager that exposes two slots. To scale horizontally, add more flink-taskmanager instances on additional VMs and point them at this JobManager.

Choose Submit New Job to upload a JAR (the flink run equivalent) directly through the Web Dashboard. Drag a JAR file in, enter the entry class and any program arguments, set the parallelism, and choose Submit.

Job submission is enabled behind the authentication layer (web.submit.enable: true), so any operator with the basic-auth credential can deploy jobs.

The Flink CLI is at /opt/flink/bin/flink. The bundled WordCount example is a quick smoke test:

sudo -u flink /opt/flink/bin/flink run /opt/flink/examples/streaming/WordCount.jar

The job runs to completion against an embedded sample text stream and prints word counts. Reopen the Web Dashboard's Completed Jobs list to see the run.

Edit /opt/flink/conf/config.yaml, then restart the services:

sudo systemctl restart flink-jobmanager.service flink-taskmanager.service

Defaults on Standard_B2s:

• jobmanager.memory.process.size: 1024m

• taskmanager.memory.process.size: 1536m

• taskmanager.numberOfTaskSlots: 2

• parallelism.default: 1

For production, raise the TaskManager process size and slot count to match your VM size, for example 6 slots and 6144m on a D4s_v5.

Replace <new-password> with your chosen password. htpasswd -bB writes a bcrypt hash non-interactively:

sudo htpasswd -bB /etc/nginx/flink.htpasswd sac <new-password>
sudo systemctl reload nginx

If you keep /stage/scripts/flink-credentials.log as an operator reference, update FLINK_WEB_PASSWORD in it.

Replace <new-password> with the operator's chosen password:

sudo htpasswd -bB /etc/nginx/flink.htpasswd alice <new-password>
sudo systemctl reload nginx

Each operator signs in with their own basic-auth credential against the same nginx virtual host.

The SQL Gateway is bundled but not started by default. Start it on demand for ANSI SQL streaming:

sudo -u flink /opt/flink/bin/sql-gateway.sh start

The gateway listens on localhost:8083. If you need network access to it, front it with an additional nginx location /sql/ block protected by the same authentication.

Flink savepoints (durable snapshots of running job state) are stored under /var/lib/flink/savepoints. The example below takes the first running job ID from the REST API and triggers a savepoint. If no job is running, it prints a message instead:

JOB_ID=$(curl -sf -m 5 http://127.0.0.1:8081/v1/jobs/overview | grep -oE '"jid":"[^"]+"' | head -1 | cut -d\" -f4) if [ -n "$JOB_ID" ]; then sudo -u flink /opt/flink/bin/flink savepoint "$JOB_ID"; else echo "no running jobs to snapshot"; fi

Copy the savepoints directory to Azure Blob Storage (az storage blob upload-batch) on a regular schedule for off-VM retention.

sudo journalctl -u flink-jobmanager.service --no-pager -n 100
sudo journalctl -u flink-taskmanager.service --no-pager -n 100
sudo journalctl -u nginx.service --no-pager -n 50
sudo ls -1 /var/log/flink/ 2>/dev/null || sudo ls -1 /opt/flink/log/ 2>/dev/null

JobManager and TaskManager log files are also written under /opt/flink/log/. The nginx access log is at /var/log/nginx/access.log and the error log is at /var/log/nginx/error.log.