Mage AI
Overview
Mage AI is an open-source data pipeline tool for transforming and integrating data. It is a modern, developer-friendly alternative to Apache Airflow. Instead of writing DAGs by hand, you build pipelines from composable blocks (data loaders, transformers and exporters) in a visual, notebook-style editor with live output previews. You then schedule them with built-in triggers.
The SecureAnyCloud image installs Mage AI 0.9.79 into a Python virtualenv at /opt/mage/venv and runs it in standalone mode as a dedicated mage system user. The web UI is bound to loopback behind an nginx reverse proxy on TCP 80 with HTTP Basic auth. The Mage project and all pipeline data persist on a dedicated Azure data disk, and every VM generates its own unique web password on first boot.
What you get:
- Mage AI 0.9.79 in a Python virtualenv at
/opt/mage/venv, run by themagesystem user - The visual pipeline editor and scheduler UI, fronted by nginx on
:80 - nginx HTTP Basic auth (Mage's web UI has no built-in authentication by default) with a per-VM password in a root-only file
- WebSocket pass-through in nginx, so the live editor and pipeline run output stream correctly
- A dedicated Azure data disk at
/var/lib/magethat holds the Mage project and all pipeline metadata, separate from the OS disk and re-provisioned with every VM mage.serviceandnginx.serviceas systemd units, enabled and active- 24/7 SecureAnyCloud support
Prerequisites
- An active Azure subscription
- An SSH key pair
- A VNet and subnet in the target region
Standard_B2ms (2 vCPU, 8 GiB RAM) is a good starting point. Scale up for heavier pipelines and concurrent runs. Allow 22/tcp from your management network and 80/tcp for the web UI in the NSG inbound rules. Front the UI with TLS for public exposure (see Enabling HTTPS).
Step 1 - Deploy from the Azure Marketplace
Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Mage AI by SecureAnyCloud and select Create.
On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).
Review the dedicated data disk on the Disks tab, then choose Review + create and Create.
Step 2 - Deploy from the Azure CLI
az vm create \
--resource-group <your-rg> \
--name mage-ai \
--image <marketplace-image-urn> \
--size Standard_B2ms \
--admin-username azureuser \
--ssh-key-values ~/.ssh/id_ed25519.pub \
--vnet-name <your-vnet> --subnet <your-subnet> \
--public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name mage-ai --port 80 --priority 1010Step 3 - Connect to your VM
ssh azureuser@<vm-public-ip>Step 4 - Confirm the services are running
systemctl is-active mage.service nginx.serviceBoth services report active. Mage starts its web server and scheduler in standalone mode and is ready to build pipelines.
Step 5 - Retrieve your web password
Your VM generates a unique admin password on first boot and writes it to a root-only file:
sudo cat /root/mage-ai-credentials.txtThe file contains mage.admin.user (admin), mage.admin.pass and the URL for the web UI. Store the password somewhere safe.
Step 6 - Check the health endpoint
nginx serves an unauthenticated health endpoint for load balancers and probes:
curl -s http://localhost/healthIt returns ok.
Step 7 - Open the visual pipeline editor
Browse to http://<vm-public-ip>/ and sign in with the browser's HTTP Basic auth prompt as admin with the password from Step 5. Mage opens on the Pipelines dashboard. It lists every pipeline in the project and lets you create a new one with + New.
Opening a pipeline shows the notebook-style editor. Each block (loader, transformer or exporter) is a cell with its own code and live output. The block tree on the right shows how the blocks are wired together:
A data loader or transformer block holds Python (or SQL or R) code with an @data_loader or @transformer decorator. Run the block to preview its output inline before you wire it into the pipeline:
The Triggers and Runs views schedule the pipeline and list every run with its status, so you can monitor and re-run executions:
Step 8 - Confirm the API behind the auth wall
The Mage web server answers behind the same Basic auth. From the VM, a request without credentials is rejected, and a request with the per-VM password is accepted:
curl -s -o /dev/null -w 'no-auth: %{http_code}\n' http://localhost/
curl -s -o /dev/null -w 'with-auth: %{http_code}\n' -u admin:<MAGE_ADMIN_PASSWORD> http://localhost/The first prints 401 and the second prints 200. This proves that nginx enforces authentication in front of Mage's auth-less web server.
Step 9 - Create your first pipeline
- In the UI, choose + New > Standard (batch) to scaffold a pipeline.
- Choose + Data loader > Python > API/Base template to add a loader block.
- Edit the block's
@data_loaderfunction and choose the block's run button to preview the output. - Add a Transformer block downstream and wire it in.
Mage saves the pipeline definition to the project on the data disk automatically.
Step 10 - Inspect the project on the data disk
The Mage project (pipelines, blocks and metadata.yaml) lives under the data directory. List it as the mage user:
sudo ls /var/lib/mage/default_repoYou see metadata.yaml, pipelines/, data_loaders/, transformers/ and data_exporters/. This is the project Mage reads and writes as you build.
Step 11 - Confirm data lives on the dedicated disk
The Mage project and all pipeline data are stored on a dedicated Azure data disk. They survive OS changes, and you can resize the disk independently:
findmnt /var/lib/mageThe mount is backed by a separate Azure data disk that is captured into the image and re-provisioned on every VM.
Enabling HTTPS
The nginx reverse proxy terminates plain HTTP on port 80. For public exposure, put a certificate in front of it.
The simplest approach is to add a DNS name for the VM and use the companion SecureAnyCloud nginx-ssl-certbot image as a TLS reverse proxy. Alternatively, install certbot and extend the existing nginx site with a listen 443 ssl; server block and your certificate paths.
Keep Mage itself bound to loopback, so the only public surface is the authenticated, TLS-terminated proxy. The existing nginx site already passes the WebSocket upgrade headers Mage needs, so the live editor keeps working over TLS.
Maintenance
- Configuration: the listener host, port and project path are set in
/etc/default/mage. Edit it and runsudo systemctl restart mageto apply changes. - Backups: snapshot the
/var/lib/magedata disk, or copy thedefault_repoproject tree to Azure Blob Storage. - Upgrades: activate the virtualenv (
source /opt/mage/venv/bin/activate), runpip install --upgrade mage-ai, then runsudo systemctl restart mage. - Security patches: unattended-upgrades stays enabled, so the OS keeps receiving security updates automatically.
Support
SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.
Mage is a trademark of Mage Technologies, Inc. This image is produced by SecureAnyCloud and is not affiliated with or endorsed by the Mage project. Mage AI is distributed under the Apache License 2.0.
Deploy on Azure
Launch Mage AI on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.
View on Marketplace
Need Help?
Our support team is available 24/7.
sales@secureanycloud.com
Overview
This image runs Mage AI, the open-source data pipeline tool for transforming and integrating data. Mage gives data engineers and analysts a browser-based, notebook-style editor. They use it to build, run and monitor batch and streaming pipelines from modular blocks written in Python, SQL and R. It includes a built-in scheduler, a run history and a data preview at every step.
What you get:
- Mage AI installed from PyPI into a dedicated Python virtual environment at
/opt/mage/venv, running as a dedicated unprivilegedmagesystem account under a systemd service that starts on boot and restarts on failure. - The Mage project directory and the SQLite metadata and user database under
/var/lib/mage, on a dedicated, independently resizable EBS data volume. - Mage's own user authentication, enabled. The application binds to the loopback interface only and is never exposed directly.
- An nginx reverse proxy that publishes the pipeline editor and the REST API on port 80.
- An owner password generated on the first boot of every instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to
/root/mage-ai-aws-credentials.txtwith mode 0600, so only the root user can read it.
Prerequisites
Before you deploy this image, you need:
- An Amazon Web Services account where you can launch EC2 instances
- IAM permissions to launch instances, create security groups and subscribe to AWS Marketplace products
- An EC2 key pair in the target Region for SSH access to the instance
- A VPC and subnet in the target Region, with a security group that allows inbound port 22 from your management network and port 80 for the pipeline editor and the REST API
- The AWS CLI (version 2) installed locally, if you plan to deploy from the command line
Launch the Instance from the AWS Marketplace
Sign in to the AWS Management Console, open the EC2 service and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Mage AI. Select the SecureAnyCloud listing, choose Select, then choose Continue on the subscription summary.
Pick an instance type of t3.medium or larger. Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet. Create or select a security group that opens port 22 from your management network and port 80 for the pipeline editor and the REST API. Leave the root volume at the default size or larger.
Select Launch instance. First-boot initialisation generates the owner password shortly after the instance state becomes Running and the status checks pass. The Python application takes a minute or two to finish its first start, so allow a little time before the editor answers.
Launch the Instance from the AWS CLI
This command launches an instance from the SecureAnyCloud Mage AI Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID and <security-group-id> with a security group that opens ports 22 and 80.
aws ec2 run-instances \
--image-id <ami-id> \
--instance-type t3.medium \
--key-name <key-name> \
--subnet-id <subnet-id> \
--security-group-ids <security-group-id> \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=mage-ai}]'When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.
Connect to Your Instance
Connect over SSH with your key pair and the login user for your operating system variant.
OS variant SSH login user
Ubuntu 24.04 ubuntu
ssh -i <key-name>.pem ubuntu@<public-ip>Retrieve the Owner Password
Your instance generated a unique Mage owner password on first boot. Read it as root:
sudo cat /root/mage-ai-aws-credentials.txtThe file lists the editor URL, the owner email (admin@sac.local), the owner username (admin), the generated password and a REST API example. Keep this password somewhere safe.
Sign In to the Pipeline Editor
nginx serves the Mage pipeline editor on port 80. In a browser, go to:
http://<instance-public-ip>/On the sign-in page, enter the email admin@sac.local and the password from the credentials file. You land on the pipelines dashboard, which lists the pipelines in the project with their type and status.
Open a pipeline to enter the notebook-style editor. The block tree is on the left, and each block's code and configuration is in the centre. Blocks are small pieces of Python, SQL or R wired together into a directed graph. You can run a block, preview its output, and add data loader, transformer and data exporter blocks from the palette.
The pipeline runs view lists every executed run with its trigger, status and timing, drawn from the built-in scheduler and run history. Open a run to inspect its block-level logs.
Confirm Mage AI Is Running
Over SSH, confirm that the application and the nginx proxy are active and that the ports are listening:
sudo systemctl is-active mage nginxsudo ss -tlnp | grep -E ':(80|6789) 'You should see both services reported as active, the application listening on 127.0.0.1:6789 (loopback only), and nginx listening on port 80. You can also check the application's health endpoint over loopback:
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6789/api/statusThis returns 200 when Mage is ready.
Use the REST API
The Mage REST API is served on the same port 80 behind the same login. First create a session to obtain a token. From a remote client, POST your owner email and password to /api/sessions. The public API key shown is the standard Mage client key and is not a secret. The request looks like this:
curl -s -X POST http://<instance-public-ip>/api/sessions \
-H 'Content-Type: application/json' \
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--data '{"session":{"email":"admin@sac.local","password":"<password>"}}'The response contains a session.token, which is a JSON Web Token. Mage's bearer authentication expects the OAuth access token carried inside that JWT, not the JWT string itself. Decode the token claim from the JWT payload and use it as the bearer.
This one-liner signs in, extracts the access token and lists the pipelines, all over loopback on the instance:
ACCESS_TOKEN=$(curl -s -X POST http://127.0.0.1/api/sessions \
-H 'Content-Type: application/json' \
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--data "{\"session\":{\"email\":\"admin@sac.local\",\"password\":\"$(sudo grep '^mage.owner.pass=' /root/mage-ai-aws-credentials.txt | cut -d= -f2-)\"}}" \
| python3 -c 'import sys,json,base64; jwt=json.load(sys.stdin)["session"]["token"]; p=jwt.split(".")[1]; p+="="*(-len(p)%4); print(json.loads(base64.urlsafe_b64decode(p))["token"])')
curl -s http://127.0.0.1/api/pipelines \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'From a remote client, substitute http://<instance-public-ip> for http://127.0.0.1 and pass the owner password in place of the sudo grep lookup.
Build and Schedule Pipelines
In the editor, create a new pipeline and add blocks from the palette. A typical batch pipeline has a data loader block that reads from a database, warehouse, file or API. It then has one or more transformer blocks written in Python or SQL, and a data exporter block that writes the result to your destination. Run each block to preview its output as you build.
When the pipeline is ready, create a trigger to schedule it. Mage supports schedule triggers on a cron interval, event triggers and API triggers. The scheduler runs active triggers automatically and records every run in the run history with its logs and status.
Database and warehouse connections are configured in the project's io_config.yaml, which lives in the project directory on the data volume. Secrets such as passwords are encrypted with a per-instance key held in the .mage_data directory. The key is generated on first boot and never shipped in the image.
The Data Volume
The Mage project directory and the SQLite metadata and user database live on a dedicated EBS volume mounted at /var/lib/mage. This keeps your pipelines and owner account off the operating system disk and lets you resize or snapshot them independently. Confirm the mount with:
df -h /var/lib/mageTo grow the project store, expand the EBS volume in the AWS console. Then grow the filesystem on the instance with sudo resize2fs on the underlying device. The Mage project is at /var/lib/mage/sac and the metadata database is the SQLite file /var/lib/mage/mage-metadata.db.
Enable HTTPS
nginx serves the pipeline editor and REST API over plain HTTP on port 80. For production use, place them behind TLS.
Obtain a certificate for your domain, for example a managed certificate on an Application Load Balancer in front of the instance, or Certbot installed on the instance. Then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:6789, exactly as the bundled site does for port 80. Keep the WebSocket upgrade headers in place so the live editor and terminal work. Restrict the security group so ports 80 and 443 are reachable only from the networks that build and run pipelines.
Backup and Maintenance
Back up your work by snapshotting the /var/lib/mage EBS volume. The snapshot captures the project directory, the pipelines and the metadata database in one consistent image.
Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade, and reboot when a new kernel is installed. Mage and nginx start automatically on boot.
To manage users, sign in as the owner and open the workspace users settings. There you can add users and assign roles.
Support
SecureAnyCloud publishes and supports this image. Support covers deployment, pipeline design, block development, database and warehouse integration, scheduling and triggers, user management, TLS and runtime tuning. Contact SecureAnyCloud through the support channel on the AWS Marketplace listing.
All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
Deploy on AWS
Launch Mage AI on AWS with 24/7 support from SecureAnyCloud.
View on Marketplace
Need Help?
Our support team is available 24/7.
sales@secureanycloud.com