Apache Airflow

Sign in to the AWS Management Console and open the EC2 service. Select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Apache Airflow. Select the SecureAnyCloud listing and choose Select, then choose Continue on the subscription summary.

Pick an instance type of m5.large or larger. Airflow runs a scheduler, a DAG processor, an API server and a PostgreSQL database, so it benefits from at least two vCPUs and 8 GiB of memory.

Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet. Create or select a security group that allows inbound port 22 from your management network and inbound port 8080 from the network you reach the web interface from. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialisation takes about one to two minutes after the instance state becomes Running and the status checks pass.

This command launches an instance from the SecureAnyCloud Apache Airflow Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing. Replace <key-name> with your EC2 key pair name and <subnet-id> with your subnet ID. Replace <security-group-id> with a security group that opens ports 22 and 8080, as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":30,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-airflow-01}]'

The command prints a JSON document on success. Note the instance ID. When the instance is running, retrieve its public address with aws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].PublicIpAddress" --output text.

Connect over SSH with the key pair you selected and the public IP address from step 2. The SSH login user depends on the operating system of the AMI variant you launched:

AMI variant SSH login user

Apache Airflow on Ubuntu 24.04 ubuntu

The first-boot service runs before the SSH daemon becomes ready, so the credentials file is always in place when you log in for the first time. Read the file:

sudo cat /root/airflow-credentials.txt

The plain-text file contains the Airflow URL, the administrator username admin, the administrator password, and the PostgreSQL database name, user and password. Copy these values to a secure place such as a password manager or encrypted vault. Do not commit them to source control.

From the same SSH session, confirm that the deployment is healthy. The Airflow health endpoint needs no authentication:

curl -fsS http://127.0.0.1:8080/api/v2/monitor/health

A JSON document that reports the metadatabase, scheduler and DAG processor components confirms that the full stack is up.

Open a web browser and go to http://<public-ip>:8080/. Airflow shows the sign-in form to visitors who do not yet have a session.

Enter the administrator username admin and the administrator password from /root/airflow-credentials.txt. Select Sign In. After a successful sign-in, Airflow records your session and shows the navigation rail down the left of every page.

After you sign in, the Dags page lists every workflow registered on the instance. The image registers one example DAG, example_hello_world, tagged SecureAnyCloud and example.

Each row shows the DAG's schedule, its most recent run, the next scheduled run and a small bar of recent run history. The toggle on the right pauses or unpauses the DAG, and the play icon triggers a manual run. The filter controls at the top of the page narrow the list by run state, paused status or tag.

To run the example DAG from the web interface, find example_hello_world on the Dags page and make sure its toggle is set to unpaused. Select the play icon, then select Trigger. Click the DAG name to open it and watch the run.

The DAG detail view shows the grid of task instances down the left, with run history and duration charts on the Overview tab. The single task greet moves from queued to running to success within a few seconds.

You can run the same pipeline from the command line, without the web interface. The airflow dags test subcommand executes the DAG in process. Run it as the airflow service user:

sudo -u airflow env AIRFLOW_HOME=/opt/airflow \
  /opt/airflow/venv/bin/airflow dags test example_hello_world

The command exits with status 0 when the DAG completes. SecureAnyCloud's validate script uses the same command to confirm that the image boots into a working DAG before it is captured.

DAG files live in /opt/airflow/dags. The DAG processor watches that directory and registers new files automatically. Create a DAG file owned by the airflow user:

sudo -u airflow tee /opt/airflow/dags/my_first_dag.py >/dev/null <<'PYEOF' from __future__ import annotations import logging from datetime import datetime from airflow.decorators import dag, task @dag( dag_id="my_first_dag", start_date=datetime(2026, 1, 1), schedule=None, catchup=False, tags=["example"], ) def my_first_dag(): @task def hello(name: str) -> str: msg = f"hello {name} from my first DAG" logging.info(msg) return msg hello("world") my_first_dag() PYEOF

Refresh the Dags page after about thirty seconds, and my_first_dag appears. If it does not, check for import errors:

sudo -u airflow env AIRFLOW_HOME=/opt/airflow \
  /opt/airflow/venv/bin/airflow dags list-import-errors

The deployed image contains these components:

Component Purpose

Apache Airflow 3.x Workflow orchestration: DAG authoring, scheduling, and monitoring

Python 3 virtual environment Isolated interpreter at /opt/airflow/venv with Airflow and its providers installed

PostgreSQL Metadata backend, local only, bound to 127.0.0.1

LocalExecutor Runs task instances as subprocesses of the scheduler

systemd units airflow-apiserver, airflow-scheduler, airflow-dag-processor, postgresql

The Airflow processes run under the dedicated airflow system user. The API server hosts both the web interface and the REST API on port 8080. The scheduler decides which task instances to run. The DAG processor parses the files in /opt/airflow/dags.

Path Purpose

/opt/airflow/ AIRFLOW_HOME: configuration, logs, DAGs and the virtual environment, on its own EBS volume

/opt/airflow/airflow.cfg Main Airflow configuration, including the executor, the Fernet key and the database connection string

/opt/airflow/venv/ Python virtual environment with Apache Airflow installed

/opt/airflow/dags/ DAG source files: the DAG processor watches this directory

/opt/airflow/logs/ Task and scheduler logs

/var/lib/postgresql/ PostgreSQL data directory holding the Airflow metadata database, on its own EBS volume

/root/airflow-credentials.txt Per-instance administrator and PostgreSQL credentials, generated on first boot, root only

The image places the Airflow home and the PostgreSQL data directory on separate EBS volumes, so you can resize each tier independently of the operating system disk.

systemd starts the three Airflow units and PostgreSQL at boot. Check their status:

sudo systemctl status airflow-apiserver airflow-scheduler airflow-dag-processor postgresql

Restart a single unit, for example after you edit airflow.cfg:

sudo systemctl restart airflow-scheduler

Read the recent logs of a unit:

sudo journalctl -u airflow-apiserver -n 50 --no-pager

The Airflow REST API is served on the same port 8080 as the web interface. Request a token with the administrator credentials, then pass the token as a bearer token. This command block authenticates and lists the DAGs:

PASS=$(sudo grep '^airflow.admin.pass=' /root/airflow-credentials.txt | cut -d= -f2-)
TOKEN=$(curl -s -X POST http://127.0.0.1:8080/auth/token \
  -H 'Content-Type: application/json' \
  -d "{\"username\":\"admin\",\"password\":\"${PASS}\"}" \
  | sed -E 's/.*"access_token" *: *"([^"]+)".*/\1/')
curl -s http://127.0.0.1:8080/api/v2/dags -H "Authorization: Bearer ${TOKEN}"

The Apache Airflow documentation publishes the full REST API reference at https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html.

For a production deployment, rotate the administrator password that was generated on first boot. From the command line, use the airflow users subcommand:

sudo -u airflow env AIRFLOW_HOME=/opt/airflow \
  /opt/airflow/venv/bin/airflow users reset-password \
  --username admin --password '<new-password>'

You can also create additional named users and assign them roles such as Viewer, User or Op from Security in the web interface. This way, team members do not share the administrator account.

The Airflow API server listens on plain HTTP on port 8080. For any deployment beyond a trusted management network, place an HTTPS reverse proxy in front of it, so that session cookies and API tokens cannot be intercepted. nginx is a common choice:

sudo apt-get update && sudo apt-get install -y nginx certbot python3-certbot-nginx
sudo certbot --nginx -d airflow.your-domain.example \
  --non-interactive --agree-tos -m you@your-domain.example --redirect

Configure the nginx server block to proxy to http://127.0.0.1:8080. Set base_url in the [api] section of /opt/airflow/airflow.cfg to your HTTPS URL, so that Airflow generates correct absolute links. Then restrict the instance security group so that port 8080 is reachable only from the proxy.

Airflow has two data sources that you must back up together. The first is the PostgreSQL metadata database, which holds DAG run history, connection records, variables and user accounts. The second is the DAG files in /opt/airflow/dags. Back up both:

sudo -u postgres pg_dump airflow > /var/backups/airflow-db-$(date +%F).sql
sudo tar -czf /var/backups/airflow-dags-$(date +%F).tgz -C /opt/airflow dags

Ship both artifacts to an Amazon S3 bucket or another object store. The Airflow home and the PostgreSQL data directory are on their own EBS volumes, so you can also take EBS snapshots of those volumes for point-in-time recovery.

For kernel and package updates, Ubuntu's unattended-upgrades is enabled by default, so security patches apply automatically. To update Airflow itself, install the new version into the virtual environment with pip, pinned to the matching Airflow constraints file. Then run airflow db migrate and restart the services.

For larger deployments, decouple Airflow from the single-instance pattern:

• Move the metadata database to Amazon RDS for PostgreSQL and update the connection string in /opt/airflow/airflow.cfg.
• Switch from the LocalExecutor to the CeleryExecutor, with Amazon ElastiCache for Redis as the broker, and run workers on a fleet of instances.
• Alternatively, run Airflow on Amazon EKS with the KubernetesExecutor, so each task runs in its own pod.
• Put the API server behind an Application Load Balancer for high availability of the web interface and REST API.

The official Apache Airflow documentation covers each of these under its deployment and executor sections, at https://airflow.apache.org/docs/ under the deployment and executor sections.