Apache Flink

Sign in to the AWS Management Console, open the EC2 service and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Apache Flink. Select the SecureAnyCloud listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger, because stream processing is memory and CPU sensitive. Choose your EC2 key pair under Key pair (login).

Under Network settings, select your VPC and subnet. Create or select a security group that allows inbound port 22 from your management network and inbound port 8081 from the trusted networks that need the Web Dashboard. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialization takes well under a minute after the instance state becomes Running and the status checks pass.

This command launches an instance from the SecureAnyCloud Apache Flink Marketplace AMI into an existing subnet and security group. Make these replacements:

• Replace <ami-id> with the AMI ID shown on the Marketplace listing.
• Replace <key-name> with your EC2 key pair name.
• Replace <subnet-id> with your subnet ID.
• Replace <security-group-id> with a security group that opens ports 22 and 8081 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":30,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-flink-01}]'

On success, the command prints a JSON document. Note the instance ID. Once the instance is running, retrieve its public address withaws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].PublicIpAddress" --output text.

Connect over SSH with the key pair you selected and the public IP address from step 2. On the Apache Flink 2 on Ubuntu 24.04 variant, the SSH login user is ubuntu.

The first boot service runs before the SSH daemon becomes ready, so the cluster information file is always in place when you log in for the first time. Read it:

sudo cat /stage/scripts/flink-info.log

The file is plain text. It records the dashboard URL, the REST API endpoint, the Flink home directory, the data volume mount point and the names of the two systemd services. Apache Flink has no built-in passwords, so the file holds no secret. It is an operator reference.

The JobManager and the TaskManager each run as a systemd service. Confirm that both are active:

sudo systemctl is-active flink-jobmanager.service flink-taskmanager.service

Both lines report active. Check the installed Flink release with the command-line client:

cd /tmp && sudo -u flink /opt/flink/bin/flink --version

The Java runtime that Flink runs on is OpenJDK 17:

java -version

The JobManager REST API is the programmatic interface to the cluster. The overview endpoint reports cluster health in one call:

curl -fsS http://127.0.0.1:8081/overview

The JSON response returns flink-version, the count of registered TaskManagers, the total and available task slots, and counters for running, finished, canceled and failed jobs. The /taskmanagers endpoint lists the registered TaskManagers in detail:

curl -fsS http://127.0.0.1:8081/taskmanagers | head -c 320

Open a web browser and go to http://<public-ip>:8081/. The Web Dashboard opens on the cluster Overview. It shows the available task slots, the running job count, and the lists of running and completed jobs, with the Flink version and commit in the header.

Apache Flink has no authentication, so anyone who can reach port 8081 has full control of the cluster, including the ability to submit and cancel jobs. Keep the port 8081 security group rule restricted to trusted networks, and read Step 12 before you expose the dashboard more widely.

The Task Managers page in the left navigation lists every registered TaskManager with its path and ID, data port, last heartbeat, total and free task slots, CPU cores and memory. On this image you start with one TaskManager that exposes two task slots.

Select a TaskManager to open its detail view with per-TaskManager metrics, logs and a thread dump. To scale the cluster out, add more TaskManager processes on additional instances and point them at this JobManager (see Step 11).

The Flink command-line client at /opt/flink/bin/flink submits, lists and cancels jobs. The Flink distribution ships a set of example jobs under /opt/flink/examples.

The streaming TopSpeedWindowing example is a continuous job that is useful for confirming the cluster end to end. Submit it in detached mode so the client returns immediately:

cd /tmp && sudo -u flink /opt/flink/bin/flink run -d /opt/flink/examples/streaming/TopSpeedWindowing.jar

The client prints the assigned JobID. List the running jobs to confirm it was accepted:

cd /tmp && sudo -u flink /opt/flink/bin/flink list

Back in the Web Dashboard, select Jobs, then Running Jobs, and open the job you submitted. The job view shows the job graph, which is the streaming dataflow as a directed graph of operators. It also shows the job state, the start time and duration, and a per-subtask metrics table with the records and bytes flowing through each operator.

The Submit New Job page in the left navigation does the same thing as the command-line client, through the browser. Upload a job JAR, set the entry class, the program arguments and the parallelism, and submit. Job submission and cancellation through the dashboard are both enabled on this image.

To cancel a running job from the command line, pass its JobID to flink cancel. The example below cancels every job that is currently running, so use it only when you intend to stop all of them. To stop one specific job, replace the loop with a single flink cancel <job-id>.

cd /tmp for jid in $(sudo -u flink /opt/flink/bin/flink list -r 2>/dev/null | grep -oE '[0-9a-f]{32}'); do
sudo -u flink /opt/flink/bin/flink cancel "$jid" done

You can also cancel a job from its job view in the Web Dashboard. Use the Cancel Job control in the top right.

The cluster configuration is /opt/flink/conf/config.yaml. After editing it, restart the services so the change takes effect:

sudo systemctl restart flink-jobmanager.service flink-taskmanager.service

The defaults on this image are sized for a modest instance:

• jobmanager.memory.process.size: 1600m
• taskmanager.memory.process.size: 2048m
• taskmanager.numberOfTaskSlots: 2
• parallelism.default: 1

For a larger instance type, raise taskmanager.memory.process.size and taskmanager.numberOfTaskSlots to match the available memory and vCPUs.

To scale beyond one instance, launch further instances, install Flink and start additional TaskManager processes pointed at this JobManager's RPC address. The cluster then has more task slots and can run jobs at higher parallelism.

For production workloads, run the JobManager in a highly available configuration with multiple JobManagers and a durable metadata store, as described in the official Flink documentation.

Apache Flink does not authenticate Web Dashboard or REST API requests, so the network is your security boundary. Take at least one of these measures before the instance is reachable from any untrusted network:

• Keep the port 8081 inbound rule in the instance's security group restricted to specific trusted CIDR ranges, never 0.0.0.0/0.
• Place the dashboard behind an Application Load Balancer that terminates TLS and is itself protected, for example by AWS WAF or by an authentication layer.
• Reach the dashboard over an SSH tunnel and leave port 8081 closed to the internet entirely. Open the tunnel like this:

ssh -L 8081:127.0.0.1:8081 <login-user>@<public-ip>

With the tunnel open, browse to http://127.0.0.1:8081/ on your workstation.

Flink's fault tolerance is built on two kinds of snapshot. Checkpoints are periodic snapshots of running job state. Savepoints are durable snapshots that you trigger explicitly for upgrades and migrations.

On this image, both directories live on the dedicated data volume, under /opt/flink-data/checkpoints and /opt/flink-data/savepoints.

The example below picks the first running job from the REST API and triggers a savepoint for it. It prints a message when nothing is running:

JOB_ID=$(curl -fsS -m 5 http://127.0.0.1:8081/jobs/overview | grep -oE '"jid":"[^"]+"' | head -1 | cut -d'"' -f4) if [ -n "$JOB_ID" ]; then cd /tmp && sudo -u flink /opt/flink/bin/flink savepoint "$JOB_ID"; else echo "no running jobs to snapshot"; fi

The state that matters on a Flink instance is the savepoints, plus any job JARs and configuration you have added. Archive the data volume directory and copy the archive to durable off-instance storage:

sudo tar -czf /var/backups/flink-data-$(date +%F).tgz -C /opt flink-data

Ship the archive to an Amazon S3 bucket or another object store, and synchronize the savepoints directory on a schedule for off-instance retention.

Ubuntu's unattended-upgrades is enabled by default, so operating system security patches apply automatically.

To move to a newer Flink release, follow these steps:

1. Download the new binary distribution from https://flink.apache.org/.
2. Take a savepoint of each running job.
3. Stop the services.
4. Swap the /opt/flink symlink to the new versioned directory.
5. Carry your configuration across.
6. Restart the services.
7. Resume each job from its savepoint.

The JobManager and the TaskManager log to the systemd journal and to log files on the data volume:

sudo journalctl -u flink-jobmanager.service --no-pager -n 100
sudo journalctl -u flink-taskmanager.service --no-pager -n 100
sudo ls -1 /opt/flink-data/log/

If a job fails, its exceptions appear on the Exceptions tab of the job view in the Web Dashboard. If the TaskManager does not appear on the Task Managers page, confirm that flink-taskmanager.service is active and check its journal for connection errors to the JobManager.