OLake

1. In the Azure portal, choose Create a resource and search the Marketplace for "OLake on Ubuntu 24.04 LTS by SecureAnyCloud".
2. Select the offer and choose Create.
3. Select your subscription, resource group and region.
4. Choose the size Standard_B2s or larger.
5. Set the authentication type to SSH public key and the username to azureuser.
6. On the Networking tab, allow inbound SSH (22) and HTTP (80) from your own address range only.
7. Choose Review and create, then choose Create.

az group create --name olake-rg --location eastus
az vm image terms accept --urn sac1647283583153:olake-ubuntu-24-04:default:latest
az vm create \
  --resource-group olake-rg \
  --name olake-vm \
  --image sac1647283583153:olake-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
az vm open-port --resource-group olake-rg --name olake-vm --port 80 --priority 1010

Restrict port 80 to your own address range rather than leaving it open to the internet. See Step 18 to put the console behind HTTPS.

ssh azureuser@<vm-ip>

To find the address of the VM you created, run:

az vm show --resource-group olake-rg --name olake-vm --show-details --query publicIps --output tsv

The image runs five systemd units. olake-firstboot generates this VM's unique secrets before anything starts. olake brings the container stack up. olake-postboot creates the administrator account and seeds the demo data.

systemctl is-active docker.service olake-firstboot.service olake.service olake-postboot.service nginx.service

All five report active. The container stack itself is six containers:

sudo docker ps --format 'table {{.Names}}\t{{.Status}}'

You should see olake-ui, olake-worker, olake-temporal, olake-postgres, olake-sample-db and olake-storage, all Up, with the health-checked ones reporting healthy.

Nothing in this image ships with a known password. On the first boot of every VM, the image generates a unique console administrator password, metadata database password, credential encryption key, demo database password and object storage key pair. It writes them to a root-only file.

sudo cat /root/olake-credentials.txt

The file is mode 0600 and owned by root. It lists the administrator login, the web console URL and the connection details for the bundled demo source and destination.

The console API refuses anonymous access and forged session cookies. It accepts only this VM's administrator credential.

curl -s -o /dev/null -w 'unauthenticated -> HTTP %{http_code}\n' http://127.0.0.1/api/v1/project/123/sources
curl -s -o /dev/null -w 'forged cookie -> HTTP %{http_code}\n' -H 'Cookie: olake-session=not-a-real-session' http://127.0.0.1/api/v1/project/123/sources

Both return 401. Now authenticate with this VM's administrator credential and call the same URL:

curl -s -c /tmp/olake-cj -o /dev/null -X POST -H 'Content-Type: application/json' \
  -d '{"username":"<ADMIN_USERNAME>","password":"<ADMIN_PASSWORD>"}' http://127.0.0.1/login
curl -s -b /tmp/olake-cj -o /dev/null -w 'authenticated -> HTTP %{http_code}\n' http://127.0.0.1/api/v1/project/123/sources rm -f /tmp/olake-cj

This returns 200. The pairing matters. The same URL that was refused anonymously is accepted with the per-instance credential, which shows that the refusals were real authorization decisions and not a service that was simply not running.

The image ships a verification script that proves a real replication end to end. It writes a table of known rows into the demo source and runs a sync through OLake. It then reads back the Parquet that OLake wrote and compares it to the source row by row.

sudo /usr/local/sbin/olake-roundtrip.sh

The script prints a line beginning OK round-trip: that reports the replicated row count and confirms that the content digest matches the source exactly. The check asserts on the data itself, not on the job status, so a sync that reports success while moving zero rows fails it.

Open <WEB_URL> in a browser. Sign in with the ADMIN_USERNAME and ADMIN_PASSWORD from the credentials file.

The image arrives with a demo job already registered, called Demo: customers to lakehouse. It replicates the demo.customers table from the bundled PostgreSQL source into the bundled object store as Parquet. It has never been run, so the first sync is yours.

Open Jobs, find the demo job and choose Sync from its actions menu. The run takes a few seconds for the demo dataset. When it finishes, the Last Run status column shows Completed.

Select the job to open Job Logs and History. It lists every run with its start time, runtime and status. Choose View logs to see the connector logs for a run.

Sources lists the bundled demo database and shows which jobs are associated with it. Destinations lists the bundled object store. Both are ordinary OLake entries. Edit them, or add your own alongside them.

The bundled source is a normal PostgreSQL 16 database. It is published on the container gateway address, not on a public interface. You can reach it from the VM and from the connector containers, but not from the internet.

sudo docker exec -e PGPASSWORD='<SAMPLE_DB_PASSWORD>' olake-sample-db \
  psql -U olake_demo -d olake_demo -c 'SELECT count(*) AS customers FROM demo.customers'

sudo docker exec -e PGPASSWORD='<SAMPLE_DB_PASSWORD>' olake-sample-db \
  psql -U olake_demo -d olake_demo -c 'SELECT count(*) AS orders FROM demo.orders'

The dataset is 500 customers and 2000 orders, generated on this VM at first boot. It contains no third-party data.

The destination is an S3-compatible object store, so you can list and read the objects that OLake writes with ordinary S3 tooling. The image includes DuckDB, which reads Parquet directly.

sudo python3 - <<'PY' import boto3, os from botocore.config import Config env = dict(l.strip().split("=", 1) for l in open("/etc/olake/olake.env") if "=" in l and not l.startswith("#")) s3 = boto3.client("s3", endpoint_url=env["S3_ENDPOINT"], aws_access_key_id=env["S3_ACCESS_KEY"], aws_secret_access_key=env["S3_SECRET_KEY"], region_name="us-east-1", config=Config(s3={"addressing_style": "path"})) for o in (s3.list_objects_v2(Bucket=env["S3_BUCKET"]).get("Contents") or [])[:10]: print(o["Key"], o["Size"], "bytes") PY

Each sync writes Parquet under a prefix named for the destination database and table. To query a downloaded object directly, run:

sudo duckdb -c "SELECT 1 AS duckdb_ready"

To replicate your own data, add a source and a destination in the console, then create a job across them.

• Sources accepts PostgreSQL, MySQL, MongoDB, Oracle, SQL Server, Db2, Kafka and S3. For change data capture from PostgreSQL, the source database needs wal_level=logical and a replication slot. The bundled demo database is already configured this way, so you can try the mode before you point OLake at your own systems.
• Destinations accepts Amazon S3 (written as Parquet) and Apache Iceberg. For Iceberg, supply your catalog (AWS Glue, a REST catalog, Hive or JDBC) together with the S3 location of the warehouse. The bundled object store is a normal S3 endpoint, so you can point a destination at your own bucket by replacing the endpoint, keys and bucket name.

OLake pulls the connector image for a source type the first time you use it. The PostgreSQL connector is already present in this image. Other source types download on first use, so allow outbound access to your container registry.

Component Version Role

OLake console and API v0.4.9 Web UI and REST API on 127.0.0.1:8000 behind nginx

OLake worker v0.3.10 Runs each sync in its own connector container

OLake PostgreSQL connector v0.8.2 Reads the source database

Temporal 1.28.4 Workflow engine, PostgreSQL-backed visibility

PostgreSQL (metadata) 16 Temporal and console state

PostgreSQL (demo source) 16 Bundled synthetic dataset, wal_level=logical

SeaweedFS 4.39 Bundled S3-compatible destination

nginx Ubuntu 24.04 Front door on port 80

DuckDB 1.5.4 Reads Parquet for the bundled verification script

Every container image is pinned by digest, so the versions above are exactly the versions that run.

Path Description

/ Root filesystem

/etc/olake/compose.yaml The container stack definition

/etc/olake/olake.env Per-VM secrets, mode 0600 root only

/etc/olake/seaweed-s3.json Object store identities for this VM

/var/lib/olake/config OLake job configuration and connector working files

/var/lib/olake/storage Object store data for the bundled destination

/root/olake-credentials.txt Per-VM credentials, mode 0600 root only

/mnt Azure temporary resource disk

sudo systemctl status olake.service --no-pager

To restart the whole stack, or to stop it while the VM is idle, run:

sudo systemctl restart olake.service

sudo systemctl stop olake.service

To read the container logs for a single component, run:

sudo docker logs --tail 50 olake-ui

The console is served over plain HTTP on port 80, which is appropriate for a private network or an evaluation. For anything beyond that, put it behind TLS before you expose it.

Point a DNS record at the VM, then install a certificate with Certbot and the nginx plugin. The plugin rewrites /etc/nginx/sites-available/olake to serve HTTPS and redirect port 80. Restrict ports 80 and 443 in the network security group to the address ranges that need them.

• Restrict inbound ports 22 and 80 in the network security group to your own address ranges. Nothing else in this image listens on a public interface. The console binds to 127.0.0.1 behind nginx, and the demo database and object store bind to the container gateway address, which is not routable from outside the VM.
• Change the administrator password from the console after your first sign-in, and keep /root/olake-credentials.txt readable only by root.
• Put the console behind HTTPS before you expose it beyond a trusted network, as described in Step 18.
• Treat the bundled demo source and destination as a demonstration. Remove the demo job, source and destination once you have connected your own systems.
• The worker runs syncs by launching containers through the Docker socket, which is root-equivalent on the host. Restrict SSH access to the VM accordingly.
• Keep the operating system patched. Unattended security upgrades are enabled by default.

The console does not answer on port 80. Check that the stack is up with sudo systemctl status olake.service --no-pager and sudo docker ps. On the very first boot, the image creates the Temporal schema before the console starts. This takes up to a minute longer than later boots.

A job stays in a running state. Look at the run in Job Logs and History and open View logs for the connector output. The worker launches one connector container per sync, so sudo docker ps -a shows the connector container and its exit state.

A sync fails immediately for a source you added. OLake downloads the connector image for that source type on first use. Confirm that the VM has outbound access to the container registry, then retry.

The demo job is missing. The image creates it on first boot only. Check sudo journalctl -u olake-postboot.service --no-pager for the seeding log.

OLake is licensed under the Apache License 2.0. Temporal, SeaweedFS, PostgreSQL and DuckDB ship under their own open-source licenses. This image is not affiliated with, endorsed by or sponsored by OLake, Datazip, Temporal, SeaweedFS or the PostgreSQL Global Development Group. This guide uses those names only to identify the software included in the image.

For help with this image, contact SecureAnyCloud support:

• Email: sales@secureanycloud.com
• Response time: 24/7, with a guaranteed 24-hour response SLA
• Website: https://secureanycloud.com

Deploy on Azure

Find this image on the Azure Marketplace as OLake on Ubuntu 24.04 LTS by SecureAnyCloud.

Deploy on Azure

Launch OLake on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Sign in to the AWS Management Console, open the EC2 service and choose Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for OLake. Select the SecureAnyCloud listing and choose Select, then choose Continue on the subscription summary.

Choose an instance type of m5.large or larger. The instance runs a six-container stack (the OLake API and worker, a Temporal workflow engine, a metadata database, the demo source database and the object store), so smaller burstable types are not recommended.

Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet, then create or select a security group that opens port 22 from your management network and port 80 for the console. Leave the root volume at the default size or larger. The container images and volumes live on a separate data volume that the image provisions automatically.

Choose Launch instance. First-boot initialization takes a few minutes after the instance state becomes Running and the status checks pass. During this time, the image generates its per-instance credentials, starts the container stack, creates the administrator account and seeds the demonstration dataset.

The following command launches an instance from the SecureAnyCloud OLake Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=olake}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name. You can find it in the EC2 console or with aws ec2 describe-instances.

Connect over SSH with your key pair and the login user for your operating system variant.

OS variant SSH login user

Ubuntu 24.04 ubuntu

ssh -i <key-name>.pem ubuntu@<public-ip>

The administrator login is unique to your instance and was generated on first boot. The credentials file records the console URL, the administrator user name, the generated password and the connection details for the bundled demo source and destination.

Confirm that the file is present and read the non-secret fields:

sudo grep -E '^(WEB_URL|ADMIN_USERNAME|S3_ENDPOINT|S3_BUCKET|SAMPLE_DB_HOST|SAMPLE_DB_PORT|SAMPLE_DB_NAME|SAMPLE_DB_USER)=' /root/olake-credentials.txt

On a freshly launched instance, this reports the console URL, the administrator user name (olakeadmin) and the demo source and object store details:

WEB_URL=http://203.0.113.10/ ADMIN_USERNAME=olakeadmin SAMPLE_DB_HOST=172.17.0.1 SAMPLE_DB_PORT=55432 SAMPLE_DB_NAME=olake_demo SAMPLE_DB_USER=olake_demo S3_ENDPOINT=http://172.17.0.1:8333 S3_BUCKET=olake-lake

To read the generated administrator password, run sudo cat /root/olake-credentials.txt and copy the value on the ADMIN_PASSWORD line. Keep it somewhere safe. The file is mode 0600 and owned by root, so you can read it only through sudo. The same file records the demo database password and the object store access and secret keys, all unique to this instance.

Over SSH, confirm that the host services are active:

sudo systemctl is-active docker olake nginx

All three report active. The olake service brings up the container stack. To see the individual containers and their health, use Docker:

sudo docker ps --format 'table {{.Names}}\t{{.Status}}'

The six containers report Up and, where they define a health check, (healthy):

NAMES STATUS olake-worker Up 7 minutes olake-ui Up 7 minutes (healthy) olake-temporal Up 7 minutes (healthy) olake-storage Up 8 minutes (healthy) olake-sample-db Up 8 minutes (healthy) olake-postgres Up 8 minutes (healthy)

Upstream OLake ships with a published default administrator account. This image does not. It removes that account and creates a per-instance administrator on first boot, and the console refuses any unauthenticated API request. Confirm this on your own instance. The console front door answers on port 80:

curl -s -o /dev/null -w 'front door: HTTP %{http_code}\n' http://127.0.0.1/

An unauthenticated request to the sources API is refused:

curl -s -o /dev/null -w 'unauthenticated API: HTTP %{http_code}\n' http://127.0.0.1/api/v1/project/123/sources

The front door returns 200 and the unauthenticated API request returns 401, so nothing behind the console is served without a session.

Browse to http://<public-ip>/ in your browser. OLake redirects to its sign-in page. Enter olakeadmin as the user name and the password you retrieved in Step 4.

After you sign in, the console opens the Jobs view. A demonstration job named Demo: customers to lakehouse is already registered. It replicates the seeded customers table from the bundled PostgreSQL source (Sample Postgres) to the bundled object store destination (Local Lakehouse).

The Sources view under Ingestion lists the registered sources. The bundled Sample Postgres source is a real PostgreSQL 16 database. It is seeded on first boot with a synthetic dataset of 500 customers and 2000 orders, generated on the instance itself.

The Destinations view lists the registered destinations. The bundled Local Lakehouse destination writes Parquet to the instance's own S3-compatible object store, so you can demonstrate a full replication with no external cloud account.

Open the Demo: customers to lakehouse job from the Jobs view and choose Sync to run it. OLake spawns an isolated connector container, reads the customers table from the demo source and writes it as Parquet to the object store. When the run completes, the job's Last Run status shows Completed. The run history records the row counts and timing.

The demo source runs with wal_level=logical, so you can also switch the streams from full refresh to change data capture mode and rerun the job. This shows incremental replication, which reads the database write-ahead log instead of rescanning the table.

To replicate your own data, register a new source with + Create Source and a new destination with + Create Destination. Then create a job that connects them.

• Source: Choose the database type (PostgreSQL, MySQL, MongoDB, Oracle and more) and enter its host, port, database, user and password. OLake discovers the available tables and columns, so you can select which streams to replicate and whether to use full refresh or change data capture.
• Destination: Choose Apache Iceberg or Parquet and provide the object store bucket, region, path and credentials. The bundled object store is a convenience for the demo. Point new destinations at your own Amazon S3 bucket or Iceberg catalog for production data.

You can leave the bundled demo source and object store in place, or delete them once your own pipelines are running.

The Settings view under System configures global behavior such as logging, data retention and job defaults. It also configures outgoing webhook alerts, which notify Slack, Teams or a custom endpoint when a job succeeds or fails.

The container image layers, the containerd snapshots and the OLake container volumes sit on their own EBS data volume. It is mounted at /var/lib/docker by filesystem UUID, so the layout survives reboots and instance replacement. This keeps several gigabytes of image and volume data off the operating system disk.

findmnt -no SOURCE,TARGET,FSTYPE,SIZE /var/lib/docker

The mount reports the dedicated volume:

/dev/nvme1n1 /var/lib/docker ext4 48.9G

You can resize the volume independently in the EC2 console, without touching the operating system disk. If you replicate large tables into the bundled object store, grow this volume so that the Parquet output and container storage have room.

The bundled demo components make the console useful immediately. Both are addressed on the internal Docker gateway (172.17.0.1). The connector containers can reach them, but nothing outside the instance can:

• Sample Postgres: A PostgreSQL 16 database on the gateway at port 55432, database olake_demo, seeded with the synthetic demo.customers and demo.orders tables. Its password is recorded in the credentials file.
• Local Lakehouse: An S3-compatible object store on the gateway at port 8333, bucket olake-lake. Its access and secret keys are recorded in the credentials file, and anonymous access is disabled.

These are ordinary components. Delete them from the console when you are ready to work only with your own systems.