Deployment guide

dbt Core

Overview

dbt Core is the open-source data transformation framework that lets analytics engineers build, test and document data pipelines using SQL and Jinja. You write select statements as models. dbt handles dependency ordering, materialization, testing and documentation, and turns your raw data into trusted, version-controlled datasets.

This SecureAnyCloud image installs dbt Core 1.9.4 into a Python virtualenv and exposes it system-wide as the dbt command. It includes the self-contained dbt-duckdb adapter and a worked example project that builds, tests and documents a small data mart against an embedded DuckDB database, so no external data warehouse is required. This is a command-line product with no web UI.

What you get:

dbt Core 1.9.4 installed in a virtualenv at /opt/dbt/venv, on PATH as /usr/local/bin/dbt
The dbt-duckdb 1.9.3 adapter so the appliance runs end to end with no external warehouse
A worked example project at /opt/dbt/example (seeds, staging views, a mart and schema tests)
A per-user writable copy at ~/dbt-example for every login user
A login banner pointing at the example and the user guide
Ubuntu 24.04 LTS base with latest security patches at build time
24/7 SecureAnyCloud support

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet and subnet in the target region.

Standard_B2s (2 vCPU / 4 GiB RAM) is plenty for a dbt workstation or CI runner.

Configure the NSG to allow inbound 22/tcp from your management network. The bundled DuckDB example needs no outbound network. To manage a real warehouse, dbt connects out to your Snowflake, BigQuery, Redshift or Postgres endpoint over its standard port.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for dbt Core by SecureAnyCloud, and select Create.

On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22). Then choose Review + create and Create.

Step 2 - Deploy from the Azure CLI

Run the following commands, replacing the values in angle brackets:

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:dbt-core-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name dbt \
  --image sac1647283583153:dbt-core-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

Step 3 - Connect to your VM

Connect over SSH as the administrator user you chose at deployment:

ssh azureuser@<vm-public-ip>

Step 4 - Confirm dbt is installed

Print the dbt version:

dbt --version

The output reports installed: 1.9.4 for dbt Core and lists the duckdb: 1.9.3 plugin.

Step 5 - Get your writable copy of the example

Every login user gets their own writable copy of the worked example. The DuckDB adapter writes a local database file, so the project must be user-writable. Copy the read-only reference from /opt/dbt/example:

[ -d ~/dbt-example ] || cp -r /opt/dbt/example ~/dbt-example cd ~/dbt-example && dbt --version

The example ships with a profiles.yml whose dev target points at a local jaffle.duckdb. It also includes two CSV seeds, two staging models, a customer_orders mart and not_null and unique schema tests.

Step 6 - Load the seeds

dbt seed loads the bundled CSVs (raw_customers, raw_orders) into DuckDB:

cd ~/dbt-example && dbt seed

dbt reports Completed successfully with PASS=2.

Step 7 - Build the models

dbt run builds the staging views and the customer_orders mart in dependency order:

cd ~/dbt-example && dbt run

dbt creates stg_customers, stg_orders and the customer_orders table, reporting PASS=3.

Step 8 - Test the data

dbt test runs the schema tests declared in models/marts/schema.yml:

cd ~/dbt-example && dbt test

The not_null and unique tests pass with PASS=3 WARN=0 ERROR=0.

Step 9 - Build everything in one command

dbt build runs seeds, models and tests together in dependency order. This is the command you would wire into CI:

cd ~/dbt-example && dbt build

Step 10 - Generate documentation and lineage

dbt can generate a documentation site and a lineage graph describing every model, seed and test:

cd ~/dbt-example && dbt docs generate

dbt writes target/catalog.json and target/manifest.json. Serve the docs site locally with dbt docs serve, which binds to 127.0.0.1:8080. Tunnel over SSH to view it, or publish the target/ artifacts to your own static host.

Step 11 - Inspect your DuckDB results

The pipeline materialized a real customer_orders table that you can query. List the project's models and confirm the data was built:

cd ~/dbt-example && dbt ls --resource-type model && ls -la jaffle.duckdb

You can query the DuckDB file directly from Python (import duckdb; duckdb.connect("jaffle.duckdb")) or with the DuckDB CLI.

Step 12 - Connect to your own data warehouse

The DuckDB target makes the appliance self-contained, but dbt's real power is over your production warehouse. Replace the outputs block in profiles.yml with your warehouse connection, and install the matching adapter into the virtualenv. For example, for Snowflake:

jaffle_duckdb: target: prod outputs: prod: type: snowflake account: <your-account> user: <your-user> password: <your-password> role: TRANSFORMER database: ANALYTICS warehouse: TRANSFORMING schema: dbt threads: 8

Install the adapter and point dbt at the new target:

sudo /opt/dbt/venv/bin/pip install dbt-snowflake cd ~/dbt-example && dbt run --target prod

The same pattern applies to dbt-bigquery, dbt-redshift and dbt-postgres.

Maintenance

Reference template: the read-only example lives at /opt/dbt/example; your writable copy is ~/dbt-example.
Virtualenv: dbt and its adapters live in /opt/dbt/venv. Add adapters with sudo /opt/dbt/venv/bin/pip install dbt-<adapter>.
Upgrades: upgrade in place with sudo /opt/dbt/venv/bin/pip install --upgrade dbt-core dbt-duckdb.
Profiles: dbt reads profiles.yml from the project directory by default. Set DBT_PROFILES_DIR to relocate it, for example to ~/.dbt.
Security patches: unattended-upgrades stays enabled, so the OS continues to receive security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. Contact sales@secureanycloud.com.

Deploy on Azure

Launch dbt Core on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

This image runs dbt Core, the open-source data transformation framework that lets analytics engineers build, test and document data models in plain SQL. It applies software engineering best practices to the transform step of an ELT pipeline.

The image is a ready-to-use, self-contained analytics engineering workstation. dbt Core is installed with the DuckDB adapter and a complete working sample project, so as soon as you connect over SSH you can run a full transform, test and document workflow with no external data warehouse and no setup.

What you get:

A headless command-line tool, with no web interface and no credentials to manage.
dbt Core in a dedicated Python virtual environment at /opt/dbt/venv, and a wrapper at /usr/local/bin/dbt that puts dbt on every login user's PATH, so no manual virtual-environment activation is needed.
A DuckDB adapter preconfigured to materialize models into a local DuckDB database file, so the entire toolchain works with no Snowflake, BigQuery or Redshift account.
A local DuckDB warehouse at /var/lib/dbt/warehouse.duckdb, on a dedicated, independently resizable EBS data volume mounted at /var/lib/dbt. The built warehouse sits on durable storage rather than the operating system disk, and you can resize or snapshot it independently.
A security group that opens port 22 only.

Prerequisites

Before you deploy this image, make sure you have:

An Amazon Web Services account where you can launch EC2 instances.
IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products.
An EC2 key pair in the target Region for SSH access to the instance.
A VPC and subnet in the target Region, with a security group that allows inbound port 22 from your management network.
The AWS CLI (version 2) installed locally, if you plan to deploy from the command line.

Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for dbt Core. Select the SecureAnyCloud listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.small or larger. Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet. Create or select a security group that opens port 22 from your management network. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialization takes a few seconds after the instance state becomes Running and the status checks pass. The dbt toolchain and the sample project are ready by the time you can connect.

Launch the Instance from the AWS CLI

The following command launches an instance from the SecureAnyCloud dbt Core Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID and <security-group-id> with a security group that opens port 22 from your management network.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.small \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=dbt-core}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant. For Ubuntu 24.04, the login user is ubuntu.

ssh -i <key-name>.pem ubuntu@<public-ip>

A welcome banner prints the most useful commands. Run sudo cat /root/dbt-core-info.txt to see the full getting-started notes: what is installed, the sample project path, the local DuckDB warehouse path, and how to connect your own data platform.

Verify dbt and the DuckDB Adapter

dbt Core is installed in a virtual environment at /opt/dbt/venv, and the dbt wrapper on the system path runs it. Confirm the installed version. This image pins dbt Core 1.11.11 with the dbt-duckdb 1.10.1 adapter:

dbt --version

The output shows installed: 1.11.11 for Core and duckdb: 1.10.1 in the Plugins section.

The bundled, self-contained sample project ships read-only at /opt/dbt/jaffle_shop. List its files to see the structure: raw CSV seeds, staging models, a tested mart, and the profiles.yml that points the DuckDB adapter at the data volume:

find /opt/dbt/jaffle_shop -type f | sort

Build the Sample Project

The sample project materializes into a local DuckDB warehouse on the data volume, so it needs no external data platform and no credentials. DBT_PROFILES_DIR is exported to the project for every login shell, so dbt finds the profile from anywhere.

Work in your own writable copy, so the shipped project stays pristine:

cp -r /opt/dbt/jaffle_shop ~/jaffle_shop cd ~/jaffle_shop && pwd

Resolve any packages that the project declares. The bundled project needs none, so this is a quick no-op that confirms the command works:

cd ~/jaffle_shop && dbt deps

Load the raw CSV seeds into the local DuckDB warehouse:

cd ~/jaffle_shop && dbt seed

Build the staging views and the mart tables in dependency order:

cd ~/jaffle_shop && dbt run

Run the schema and data tests on the models:

cd ~/jaffle_shop && dbt test

You can run all three steps in dependency order with a single command. dbt build seeds, runs and tests the whole project, and it is the command you will use most often:

cd ~/jaffle_shop && dbt build

Every seed loads, every model is created and every test passes. The output ends with a Done. PASS=21 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=21 summary and Completed successfully.

Generate Documentation and Query the Warehouse

dbt compiles your project into a browsable documentation site with model descriptions, column-level detail and a lineage graph. Generate the documentation catalog:

cd ~/jaffle_shop && dbt docs generate

To browse the documentation, run dbt docs serve and forward the port, or copy the generated target/ directory to a static host.

The models dbt built live in the local DuckDB warehouse on the data volume. Query the warehouse with the bundled DuckDB Python to list the tables and views that dbt materialized and to read from a mart:

/opt/dbt/venv/bin/python -c "import duckdb; con = duckdb.connect('/var/lib/dbt/warehouse.duckdb', read_only=True); print(con.execute('select table_type, table_name from information_schema.tables order by 1, 2').fetchall()); print(con.execute('select customer_id, first_name, number_of_orders, customer_lifetime_value from customers order by customer_id limit 5').fetchall())"

The output shows the raw_* seed tables, the stg_* staging views and the customers and orders mart tables, then the first rows of the customers mart with their order counts and lifetime value, confirming the toolchain works end to end.

The Data Volume

The local DuckDB warehouse lives on a dedicated EBS volume mounted at /var/lib/dbt, at /var/lib/dbt/warehouse.duckdb. This volume keeps the built warehouse off the operating system disk and lets you resize or snapshot it independently. Confirm the mount with:

df -h /var/lib/dbt

The sample project's profile at /opt/dbt/jaffle_shop/profiles.yml points the DuckDB path at this volume, so every dbt run and dbt build materializes into durable storage.

To grow the volume, expand the EBS volume in the AWS console. Then grow the filesystem on the instance with sudo resize2fs on the underlying device.

Connect Your Own Data Platform

To use dbt against your own warehouse, install the matching adapter into the environment and add a profile. Install an adapter, such as Snowflake, BigQuery or Redshift, into the dbt virtual environment:

sudo /opt/dbt/venv/bin/pip install dbt-snowflake

Then create a ~/.dbt/profiles.yml for your project that points at your warehouse with its connection details. Run dbt debug to confirm the connection resolves before you run your models.

The bundled DuckDB profile remains available for local development and prototyping, so you can build and test models locally before you promote them to your cloud warehouse.

Support

This image is published and supported by SecureAnyCloud. Support covers project structure, model materializations, tests and snapshots, packages and macros, incremental models, documentation, connecting to your data platform, and CI integration. Contact SecureAnyCloud through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch dbt Core on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS