Deployment guide

Dagster

Overview

Dagster is the data orchestrator for the whole development lifecycle, built around data assets. You define your tables, models and reports as assets. Dagster builds a lineage graph, runs and schedules them, and gives you a rich web UI to observe every run, asset materialization, schedule and sensor.

This SecureAnyCloud image installs Dagster 1.13.10 in a dedicated Python virtualenv. It runs the Dagster webserver and daemon as systemd services bound to loopback behind an nginx reverse proxy on port 80 with HTTP Basic auth. Run and event storage persist on a dedicated Azure data disk, and each VM generates a unique login password on first boot.

What you get:

Dagster 1.13.10 (webserver and daemon) in a Python virtualenv at /opt/dagster/venv
A sample asset project at /opt/dagster/project
The Dagster UI on :80 via nginx, with HTTP Basic auth (Dagster has no built-in auth)
A per-VM admin password in a root-only file
A dedicated Azure data disk at /var/lib/dagster holding the SQLite run/event/schedule storage, separate from the OS disk and re-provisioned with every VM
dagster-webserver.service, dagster-daemon.service and nginx.service as systemd units
24/7 SecureAnyCloud support

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet and subnet in the target region.

Standard_B2ms (2 vCPU / 8 GiB RAM) is a good starting point. Scale up for heavier pipelines.

Configure the NSG to allow inbound 22/tcp from your management network and 80/tcp for the UI. Front the UI with TLS for public exposure (see Enabling HTTPS).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Dagster by SecureAnyCloud, and select Create.

On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab, then choose Review + create and Create.

Step 2 - Deploy from the Azure CLI

Run the following commands, replacing the values in angle brackets:

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:dagster-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name dagster \
  --image sac1647283583153:dagster-ubuntu-24-04:default:latest \
  --size Standard_B2ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name dagster --port 80 --priority 1010

Step 3 - Connect to your VM

Connect over SSH as the administrator user you chose at deployment:

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Check that the three services are active:

systemctl is-active dagster-webserver.service dagster-daemon.service nginx.service

All three services report active. The webserver serves the UI, and the daemon runs schedules and sensors.

Step 5 - Retrieve your password

Your VM generates a unique admin password on first boot and writes it to a root-only file. Read the file with:

sudo cat /root/dagster-credentials.txt

This file contains DAGSTER_ADMIN_USER (admin), DAGSTER_ADMIN_PASSWORD and the URL. Store the password in a safe place.

Step 6 - Check the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes. Request it with:

curl -s http://localhost/health

The endpoint returns ok.

Step 7 - Open the Dagster UI

Browse to http://<vm-public-ip>/ and sign in as admin with the password from Step 5. The Lineage view shows the global asset graph from the bundled sample project. Use Catalog, Runs and Automation to drill into assets, runs, schedules and sensors.

The asset catalog lists every asset and its latest materialization:

The Runs page shows every run with its status, duration and logs:

Step 8 - Materialize the sample assets

In the Lineage view, click Materialize all to run the sample pipeline. Dagster executes raw_numbers > doubled_numbers > total and records the run and asset materializations. You can inspect them on the Runs page.

The bundled demo_pipeline schedule also runs the pipeline every 15 minutes once you turn the schedule on under Automation.

Step 9 - Confirm the server from the command line

The Dagster webserver reports its version at /server_info, behind the same Basic auth. Request it with:

curl -s -u admin:<DAGSTER_ADMIN_PASSWORD> http://localhost/server_info; echo

The response is JSON including "dagster_version": "1.13.10".

Step 10 - Confirm state lives on the dedicated disk

The Dagster instance (DAGSTER_HOME) and its SQLite storage are on the dedicated Azure data disk, so they survive OS changes and can be resized independently. Confirm the mount:

findmnt /var/lib/dagster

The mount is backed by a separate Azure data disk that is captured into the image and re-provisioned on every VM.

Bring your own pipelines

Replace the sample project at /opt/dagster/project/definitions.py with your own assets and jobs, or point workspace.yaml at your own Python module. Then restart the services:

sudo systemctl restart dagster-webserver dagster-daemon

Install any extra Python packages your code needs into the bundled virtualenv with sudo /opt/dagster/venv/bin/pip install <packages>.

Enabling HTTPS

The nginx reverse proxy serves plain HTTP on port 80. For public exposure, put a certificate in front of it. Add a DNS name for the VM and use the companion SecureAnyCloud nginx-ssl-certbot image as a TLS reverse proxy. Alternatively, install certbot and extend the existing nginx site with a listen 443 ssl; server block.

Keep the Dagster webserver bound to loopback, so the only public surface is the authenticated, TLS-terminated proxy.

Maintenance

Backups: snapshot the /var/lib/dagster data disk to back up run and event history.
Database: the image uses SQLite, which suits a single server. For high throughput, configure PostgreSQL storage in /var/lib/dagster/dagster.yaml.
Upgrades: run sudo /opt/dagster/venv/bin/pip install -U dagster dagster-webserver, then sudo systemctl restart dagster-webserver dagster-daemon.
Security patches: unattended-upgrades stays enabled, so the OS continues to receive security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.

Deploy on Azure

Launch Dagster on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

This image runs Dagster, the fast, open-source data orchestration platform for the development, production and observation of data assets. You define assets and jobs in plain Python. Dagster gives you a rich web interface to materialize them, schedule them, watch them run and inspect their lineage, with full visibility into every run.

This SecureAnyCloud AMI installs Dagster 1.13.10 into a dedicated Python virtual environment under /opt/dagster and runs it as an unprivileged dagster system account. An nginx reverse proxy publishes the UI on port 80 behind HTTP Basic authentication, and each instance generates its own admin password on first boot.

What you get:

Two systemd services: dagster-webserver.service serves the UI and the GraphQL API on the loopback interface (127.0.0.1:3000), and dagster-daemon.service drives schedules, sensors and the run queue.
A Dagster instance directory, DAGSTER_HOME, at /var/lib/dagster, on a dedicated, independently resizable EBS data volume. It holds the instance configuration and the run, event and schedule storage.
A small demo project at /opt/dagster/demo, so the UI is not empty out of the box and you have a working example to learn from.
An nginx reverse proxy that forwards the GraphQL WebSocket upgrade the UI needs to render and stream live run updates. Dagster's webserver has no built-in authentication, so it binds to the loopback interface only and is never exposed directly.
An admin password generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. The password is written to /root/dagster-credentials.txt with mode 0600, so only the root user can read it.

Prerequisites

Before you deploy this image, make sure you have:

An Amazon Web Services account where you can launch EC2 instances.
IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products.
An EC2 key pair in the target Region for SSH access to the instance.
A VPC and subnet in the target Region, with a security group that allows inbound port 22 from your management network and port 80 for the Dagster UI.
The AWS CLI (version 2) installed locally, if you plan to deploy from the command line.

Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Dagster. Select the SecureAnyCloud listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet. Create or select a security group that opens port 22 from your management network and port 80 for the Dagster UI. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialization takes a few seconds after the instance state becomes Running and the status checks pass. This is when the per-instance admin password is generated.

Launch the Instance from the AWS CLI

The following command launches an instance from the SecureAnyCloud Dagster Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=dagster}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant. For Ubuntu 24.04, the login user is ubuntu.

ssh -i <key-name>.pem ubuntu@<public-ip>

Retrieve the Admin Password

The admin password is unique to your instance and was generated on first boot. Read the credentials file as root with sudo cat /root/dagster-credentials.txt.

The file lists the Dagster UI URL, the admin user name (admin) and the generated password. It also includes hints for pointing the workspace at your own code location and installing extra Python packages. Keep this password in a safe place.

Sign In to the Dagster UI

The Dagster UI is served on port 80 by nginx, which proxies to the webserver on 127.0.0.1:3000 with the GraphQL WebSocket upgrade the UI needs. In a browser, go to:

http://<instance-public-ip>/

The nginx proxy prompts you for credentials. Sign in as admin with the password from the credentials file, and the Dagster UI loads. Open the Lineage view to see the global asset graph of the demo project, with the three software-defined assets raw_records, filtered_records and summary wired together.

Materialize the assets with Materialize all in the top right of the asset graph, or run the bundled demo_job from the Jobs view. Dagster launches a run, and the daemon executes each asset in dependency order. Open the run to watch it stream live, with a Gantt timeline of the steps and a structured event log delivered over the GraphQL websocket.

The Runs view lists every run, with its target assets, who launched it, its status and its duration. From here you can re-execute a run, filter by status, or drill into any run for its full event history.

Confirm Dagster Is Running

Over SSH, confirm the webserver, the daemon and the nginx proxy are active:

sudo systemctl is-active dagster-webserver dagster-daemon nginx

Then confirm the webserver answers on the loopback interface. The /server_info endpoint returns the Dagster version JSON and is left unauthenticated so external health checks work:

curl -s http://127.0.0.1:3000/server_info

All three services report active, and /server_info returns a small JSON document with the Dagster version. The webserver listens on 127.0.0.1:3000 (loopback only), and nginx publishes the UI on port 80 behind the login.

Confirm the installed Dagster version with:

/opt/dagster/venv/bin/dagster --version

Replace the Demo Project with Your Own

A demo project ships at /opt/dagster/demo: a definitions.py with three assets and a job, and a workspace.yaml the webserver loads on start. To go live with your own code, point the workspace at your own code location and restart the services. Edit the workspace file to load your module or Python file instead of the demo definitions.py, then restart both services:

sudo systemctl restart dagster-webserver.service dagster-daemon.service

After the restart, reload the UI in your browser. The Deployment view shows your code location loaded. Keep your project files readable by the dagster user, so the services can load them.

The webserver runs dagster-webserver -h 127.0.0.1 -p 3000 -w /opt/dagster/demo/workspace.yaml, so anything the workspace points at is served on port 80 through the same authenticating proxy.

Install Extra Python Packages

Dagster and its dependencies are installed into a dedicated virtual environment at /opt/dagster/venv. To use additional Python libraries or Dagster integration packages in your project, install them into that same virtual environment, so the services pick them up:

/opt/dagster/venv/bin/pip --version

Then install whatever your project needs, for example sudo /opt/dagster/venv/bin/pip install dagster-aws dagster-dbt pandas. Restart the services with sudo systemctl restart dagster-webserver.service dagster-daemon.service.

Because the virtual environment lives under /opt/dagster, your installed packages persist with the image.

The Data Volume

The Dagster instance directory, DAGSTER_HOME, lives on a dedicated EBS volume mounted at /var/lib/dagster. This volume holds the dagster.yaml instance configuration and the SQLite-backed run, event log and schedule storage, so a single-node deployment needs no external database. Keeping it on its own volume lets you resize or snapshot the run history independently of the operating system disk. Confirm the mount with:

df -h /var/lib/dagster

To grow the instance directory, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device. For a larger or multi-node deployment, swap the default SQLite storage for PostgreSQL by editing dagster.yaml and restarting the services.

Enable HTTPS

The Dagster UI is served over plain HTTP on port 80 by nginx. For production use, place it behind TLS. Obtain a certificate for your domain, for example a managed certificate on an Application Load Balancer in front of the instance, or Certbot installed on the instance.

Then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:3000, exactly as the bundled site does for port 80. Keep the GraphQL WebSocket upgrade headers and the HTTP Basic authentication in place, so the UI continues to render and stream. Restrict the security group so ports 80 and 443 are reachable only from the networks that need the UI.

Backup and Maintenance

Back up your deployment by snapshotting the /var/lib/dagster EBS volume, which captures the instance configuration and the entire run, event and schedule history. Keep your Dagster project under version control, and redeploy it to a fresh instance to recover the code.

Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade, and reboot when a new kernel is installed. The webserver, the daemon and nginx start automatically on boot.

Support

This image is published and supported by SecureAnyCloud. Support covers deployment, defining assets and jobs, code locations and workspaces, scheduling and sensors, swapping the SQLite instance storage for PostgreSQL, the authenticating proxy, TLS and performance tuning. Contact SecureAnyCloud through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch Dagster on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS