Deployment guide

Evidently

Overview

Evidently is an open source platform for evaluating and monitoring machine learning models and data pipelines. You compute Reports over your datasets (data drift, data quality, data summary, classification and regression performance, and LLM/text evaluations), and the Evidently UI monitoring service renders them as interactive projects in a web dashboard. Teams can watch production data and model behavior over time and catch regressions before they reach users.

This SecureAnyCloud image installs the pinned official Evidently 0.7.21 pip package into a dedicated virtual environment and runs the Evidently UI server (evidently ui, a Litestar/uvicorn service) under systemd. Evidently itself is bound to loopback only, and nginx handles all customer-facing traffic on port 80 behind a per-VM HTTP Basic Auth gate.

The open source Evidently UI has no built-in user accounts, so the nginx Basic Auth credential (user admin, with a unique password generated on first boot) is the gate on the UI and API.

What you get:

Evidently 0.7.21 installed via pip into a dedicated virtual environment, running as the evidently systemd service
The Evidently UI monitoring dashboard and API on :80, fronted by nginx, with Evidently bound to loopback only
Per-VM HTTP Basic Auth (user admin) protecting the UI and API, with a unique password generated on first boot
A dedicated 20 GiB Azure data disk at /var/lib/evidently holding the Evidently workspace (projects and report snapshots), durable across restarts and reboots
A seeded demo project (sac-demo) containing a real data-drift and data-summary report over a synthetic sample, so the dashboard shows a rendered report on first login
evidently.service and nginx.service as enabled, active systemd units
An unauthenticated /healthz endpoint on :80 for Azure Load Balancer health probes
24/7 SecureAnyCloud support

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet with a subnet in the target region.

Standard_B2s (2 vCPU / 4 GiB RAM) is a comfortable starting size for a small team's monitoring dashboard. Choose a larger size for bigger workspaces or heavier report computation.

Configure the network security group (NSG) to allow inbound 22/tcp from your management network, 80/tcp for the UI and API, and 443/tcp if you add TLS. Evidently serves plain HTTP, so for production, terminate TLS in front of it with your own domain.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Evidently by SecureAnyCloud, and select Create.

On the Basics tab, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab, then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:evidently-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name evidently \
  --image sac1647283583153:evidently-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name evidently --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active evidently.service nginx.service

Both report active. Evidently serves its monitoring UI and API on the loopback address 127.0.0.1:8000 only, and nginx fronts port 80 with the per-VM HTTP Basic Auth gate.

The Evidently workspace lives on the dedicated data disk at /var/lib/evidently. The projects and report snapshots the dashboard renders are stored there, so they survive service restarts and VM reboots.

Step 5 - Retrieve your web UI password

nginx protects the Evidently UI and API with HTTP Basic Auth. The username is admin. A unique password is generated on the first boot of your VM and written to a root-only file. Read it with sudo:

sudo cat /root/evidently-credentials.txt

The file contains EVIDENTLY_USERNAME, EVIDENTLY_PASSWORD, the EVIDENTLY_URL to open in a browser, and EVIDENTLY_WORKSPACE (the workspace path on the data disk).

nginx stores the password only as a bcrypt hash in /etc/nginx/.evidently.htpasswd, so no plaintext password ships in the image. Keep the password somewhere safe.

Step 6 - Confirm the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes:

curl -s http://localhost/healthz

It returns ok. This endpoint never requires authentication, so you can use it for an Azure Load Balancer health probe.

Step 7 - Confirm authentication and browse the seeded demo project

A password is set on first boot, so an unauthenticated request to the UI or API returns HTTP 401. Nobody reaches your monitoring data without the password.

The following command reads the per-VM password from the credentials file, checks the authentication round-trip, and lists the demo project seeded on first boot through the authenticated API:

PW=$(sudo grep '^EVIDENTLY_PASSWORD=' /root/evidently-credentials.txt | cut -d= -f2-)
echo "unauth : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/api/projects)"
echo "wrongpw : $(curl -s -o /dev/null -w '%{http_code}' -u admin:wrong-pw http://127.0.0.1/api/projects)"
echo "authed : $(curl -s -o /dev/null -w '%{http_code}' -u admin:$PW http://127.0.0.1/api/projects)" curl -s -u admin:$PW http://127.0.0.1/api/projects | python3 -m json.tool

It prints unauth : 401, wrongpw : 401 and authed : 200, and the API response lists the sac-demo project. Only the per-VM password reaches the Evidently UI and API, because Evidently itself is bound to loopback and nginx on port 80 is the only way in.

Step 8 - Sign in and browse your projects

Browse to http://<vm-public-ip>/. When your browser prompts for credentials, enter admin and the password from Step 5.

Evidently opens on the Projects list, which shows the sac-demo project seeded on first boot. Each project groups the reports and test suites you add to it over time.

Step 9 - Open the demo project

Click the sac-demo project to open it. The project view has a Dashboard tab for monitoring panels you build from report metrics over time, and a Reports tab that lists the report snapshots stored in the project.

On a fresh appliance, the dashboard is ready for you to add panels. The seeded report is available under Reports.

Step 10 - View the seeded data-drift report

Open the Reports tab and click the seeded report. Evidently renders the full report in the browser.

The Data Drift section compares the current dataset against the reference dataset column by column, flags which columns drifted, and draws the distribution of each feature so you can see the shift. This is the core of the product: a data observability report computed over your data and rendered interactively.

Step 11 - Review data quality and summary metrics

Scroll to the Data Summary section of the same report. It shows per-column statistics, missing values, and distributions for the reference and current datasets side by side.

Together, the drift and summary sections give you a complete first-pass observability view of a dataset without writing any dashboard code.

Step 12 - Monitor your own data

Point your own datasets at this workspace. Use the Evidently Python SDK to compute an Evidently Report over your data and add its snapshot to a project in the workspace. On the VM, or anywhere the workspace path is available, it looks like this:

import pandas as pd from evidently import Dataset, DataDefinition, Report from evidently.presets import DataDriftPreset, DataSummaryPreset from evidently.ui.workspace import Workspace
# your reference and current datasets reference = pd.read_csv("reference.csv") current = pd.read_csv("current.csv") data_def = DataDefinition()
# or specify numerical_columns / categorical_columns ref_ds = Dataset.from_pandas(reference, data_definition=data_def) cur_ds = Dataset.from_pandas(current, data_definition=data_def) report = Report(metrics=[DataDriftPreset(), DataSummaryPreset()]) snapshot = report.run(current_data=cur_ds, reference_data=ref_ds) ws = Workspace.create("/var/lib/evidently/workspace") project = ws.create_project("my-monitoring")
# or reuse an existing project ws.add_run(project.id, snapshot)

Run pip install evidently==0.7.21 in your environment to match the server's SDK version.

After you add a run, refresh the dashboard. The server reads the workspace on startup, so restart evidently.service if a newly added project does not appear. See the upstream Evidently documentation for building monitoring dashboard panels, test suites, and LLM/text evaluations.

Maintenance

Password: the UI and API password is set on first boot and stored as a bcrypt entry in /etc/nginx/.evidently.htpasswd. To change it, run sudo htpasswd -B /etc/nginx/.evidently.htpasswd admin, then sudo systemctl reload nginx.
Durable storage: the Evidently workspace (/var/lib/evidently/workspace) survives evidently service restarts and VM reboots. If your monitoring history matters, back up or snapshot the data disk as you would any production data store.
Restrict access: Evidently serves plain HTTP on port 80. For production, restrict access to trusted IP ranges in your NSG and front it with TLS (for example, certbot with your own domain) terminating on :443.
Loopback binding: Evidently is bound to 127.0.0.1:8000 in /etc/systemd/system/evidently.service (--host 127.0.0.1 --port 8000), so nginx is the only path in. Keep it that way, and do not change the bind address to a public interface.
Disk space: the workspace grows with the number of reports and snapshots you add. Monitor /var/lib/evidently usage (df -h /var/lib/evidently) and resize the data disk if you store a large monitoring history.
Logs: journalctl -u evidently shows recent Evidently server output, and journalctl -u nginx shows the web server's own logs.
Security patches: unattended-upgrades stays enabled, so the OS keeps receiving security updates automatically.

Support

SecureAnyCloud provides 24/7 expert support for this image. For questions or help with your deployment, contact sales@secureanycloud.com.

Deploy on Azure

Launch Evidently ML & Data Observability on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

Evidently is an open source platform for evaluating, testing and monitoring machine learning, large language model and data pipelines. It computes data drift, data quality and model performance reports over your datasets and renders them in an interactive monitoring dashboard. Each report snapshot is stored as a versioned run inside a project, so you can track how your data and models change over time.

This SecureAnyCloud image installs Evidently into a dedicated Python virtual environment at /opt/evidently/venv, pinned to version 0.7.21, and runs it as a dedicated unprivileged evidently system account. An nginx reverse proxy publishes the dashboard on port 80 behind HTTP Basic Auth, with a unique password generated on the first boot of every instance.

What you get:

The Evidently UI server (evidently.service), which runs on boot, restarts on failure, and is bound to 127.0.0.1:8000 so it is never exposed directly
An nginx reverse proxy on port 80 as the only public entry point, with every request gated by HTTP Basic Auth, because the open source Evidently UI has no built-in authentication of its own
A fresh dashboard password generated on the first boot of each instance, stored as a bcrypt hash in the nginx Basic Auth file, with the plaintext written to /root/evidently-credentials.txt (mode 0600, readable by root only)
No shared or default credential in the image: two instances launched from the same Amazon Machine Image never share a password
An unauthenticated static /healthz endpoint for load balancer probes only. Every other path requires the password.
All workspace state (the projects and report snapshots the dashboard renders) at /var/lib/evidently, a dedicated, independently resizable EBS data volume that survives instance replacement
A demo project named sac-demo, seeded on first boot with a genuine data drift and data quality report over a synthetic sample, so you see an explorable report instead of an empty workspace

Prerequisites

Before you deploy this image, you need:

An Amazon Web Services account where you can launch EC2 instances
IAM permissions to launch instances, create security groups and subscribe to AWS Marketplace products
An EC2 key pair in the target Region for SSH access to the instance
A VPC and subnet in the target Region, with a security group that allows inbound port 22 from your management network and port 80 for the monitoring dashboard
The AWS CLI (version 2) installed locally, if you plan to deploy from the command line

Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Evidently. Select the SecureAnyCloud listing, choose Select, then choose Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet, then create or select a security group that opens port 22 from your management network and port 80 for the dashboard. Leave the root volume at the default size or larger.

Select Launch instance. After the instance state becomes Running and the status checks pass, first-boot initialization takes up to a minute or two. During that time the demo project is seeded, the dashboard password is generated and the proxy starts.

Launch the Instance from the AWS CLI

The following command launches an instance from the SecureAnyCloud Evidently Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=evidently}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name. You can find it in the EC2 console or with aws ec2 describe-instances.

Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant:

OS variant SSH login user

Ubuntu 24.04 ubuntu

ssh -i <key-name>.pem ubuntu@<public-ip>

Retrieve the Dashboard Password

The dashboard password is unique to your instance and is generated on first boot. Read it as root:

sudo cat /root/evidently-credentials.txt

The file lists the dashboard URL, the user name (admin) and the generated password:

EVIDENTLY_URL=http://<instance-public-ip>/ EVIDENTLY_USERNAME=admin EVIDENTLY_PASSWORD=<EVIDENTLY_PASSWORD> EVIDENTLY_WORKSPACE=/var/lib/evidently/workspace

Keep the password somewhere safe.

Sign In to the Monitoring Dashboard

The Evidently dashboard is served on port 80 by nginx. In a browser, go to:

http://<instance-public-ip>/

When your browser prompts for credentials, enter the user admin and the password from the credentials file. After you authenticate, you land on the Evidently project list, which shows the seeded sac-demo project.

A project is a container for a set of report snapshots and the monitoring panels built over them. Open sac-demo to explore the seeded report, or create a new project with the plus button to organize your own reports.

Explore the Seeded Demo Report

Open the sac-demo project and select the Reports tab. The seeded snapshot appears in the report list, timestamped from first boot, with View, Download and delete actions.

Select View to open the report. The demo runs a data drift preset over a synthetic reference dataset and current dataset that were deliberately shifted. The report therefore shows drift detected across every column, with per-column statistical tests, drift scores and reference-versus-current distribution charts.

Scroll down for the data summary. It shows per-column statistics such as count, missing values, min, max, mean and standard deviation, alongside distribution charts that compare the reference and current data for numerical and categorical features.

Confirm Evidently Is Running

Over SSH, confirm the Evidently UI server and the nginx proxy are active:

sudo systemctl is-active evidently.service nginx.service

Both are reported as active:

active active

Confirm the ports. The Evidently UI server listens on loopback 127.0.0.1:8000 and is never exposed, while nginx listens on port 80:

sudo ss -tlnp | grep -E ':(80|8000) '

LISTEN 0 2048 127.0.0.1:8000 0.0.0.0:* users:(("evidently",...)) LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",...))

The security group exposes only port 80 (and port 22 for SSH), so the dashboard is reachable only through the authenticating nginx proxy. Keep port 8000 closed in your security group.

The unauthenticated health endpoint returns a plain ok for load balancer probes:

curl -s http://127.0.0.1/healthz

Every other path requires the password. An authenticated API call over loopback lists the seeded project:

PW=$(sudo grep '^EVIDENTLY_PASSWORD=' /root/evidently-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/api/projects

[{"id":"019f9530-02d8-7645-a203-988c05584973","name":"sac-demo","description":"sac demo: data drift and data quality over a synthetic sample",...}]

Monitor Your Own Data

To monitor your own data, use the bundled Evidently Python SDK to compute Evidently reports over your datasets, and add their snapshots to a project in the workspace at /var/lib/evidently/workspace. The Evidently package is already installed in the virtual environment at /opt/evidently/venv, so you can run Python that imports it directly, for example as the evidently user.

The following example computes a data drift report over two pandas dataframes and stores it in a new project so it appears in the dashboard:

import pandas as pd from evidently import Dataset, DataDefinition, Report from evidently.presets import DataDriftPreset, DataSummaryPreset from evidently.ui.workspace import Workspace reference = pd.read_csv("reference.csv") current = pd.read_csv("current.csv") schema = DataDefinition( numerical_columns=["feature_a", "feature_b"], categorical_columns=["segment"], ) ref_ds = Dataset.from_pandas(reference, data_definition=schema) cur_ds = Dataset.from_pandas(current, data_definition=schema) report = Report(metrics=[DataDriftPreset(), DataSummaryPreset()]) snapshot = report.run(current_data=cur_ds, reference_data=ref_ds) ws = Workspace.create("/var/lib/evidently/workspace") project = ws.create_project("my-model", description="production monitoring") ws.add_run(project.id, snapshot)

Refresh the dashboard. Your new project appears in the project list, with the report available under its Reports tab.

To build a monitoring history over time, wire this into a scheduled job or your pipeline so it adds a snapshot on each run. Full API and metric documentation is on the Evidently documentation site, linked from the dashboard header.

The Data Volume

All workspace state lives on a dedicated EBS volume mounted at /var/lib/evidently. This keeps projects and report snapshots off the operating system disk and lets you resize or snapshot them independently. Confirm the mount:

df -h /var/lib/evidently

Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 20G 412K 19G 1% /var/lib/evidently

To grow the storage, expand the EBS volume in the AWS console. Then grow the filesystem on the instance by running sudo resize2fs on the underlying device.

Enable HTTPS

nginx serves the dashboard over plain HTTP on port 80. For production, place it behind TLS.

Obtain a certificate for your domain, for example a managed certificate on an Application Load Balancer in front of the instance, or Certbot installed on the instance. Then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:8000, exactly as the bundled site does for port 80. Keep the Basic Auth directives in place.

Restrict the security group so ports 80 and 443 are reachable only from the networks that use the dashboard.

Backup and Maintenance

Back up the workspace by snapshotting the /var/lib/evidently EBS volume, which captures every project and report snapshot in one consistent set.

Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade, and reboot when a new kernel is installed. Evidently and nginx start automatically on boot.

Support

SecureAnyCloud publishes and supports this image. Support covers deployment, workspace configuration, building data drift, data quality and model performance reports, wiring Evidently into batch or streaming pipelines with the Evidently Python SDK, nginx Basic Auth and TLS configuration, data volume resizing, and runtime tuning. Contact SecureAnyCloud through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch Evidently on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS