Deployment guide

Rucio

Overview

Rucio is the open-source scientific data-management system created at CERN. It keeps a single catalogue of your data files, records which storage systems hold each copy, and lets you find, upload, download and move data through one consistent REST API and command-line client.

Data is addressed by a logical name inside a namespace (a scope) rather than by a physical path. A file can therefore be located and retrieved no matter which storage element holds it.

This SecureAnyCloud image runs the official Rucio 41.1.1 server (rucio/rucio-server, an Apache httpd + mod_wsgi application), pinned to the Rucio 41 long-term-support line. It is backed by a bundled PostgreSQL 16 catalogue. First boot pre-creates Rucio's core objects, including a storage element called LOCAL and a scope called test, so a real data round-trip works as soon as the VM boots. This guide covers deploying and using the image on Ubuntu 24.04 on Azure.

What you get:

  • The official Rucio 41.1.1 server (rucio/rucio-server), run under systemd as rucio.service via Docker Compose, serving the REST API on port 80
  • A bundled PostgreSQL 16 catalogue store, reachable only on the internal Docker network and never exposed to the host or the internet
  • A pre-created root account with a per-VM userpass identity (username admin)
  • A ready-to-use local POSIX/file storage element (RSE) named LOCAL, a scope named test, and unlimited root quota, so register, upload, list and locate all work on first boot
  • Per-VM secrets written to /root/rucio-credentials.txt (mode 0600, root only)
  • A REST API with a health check at GET /ping, and the rucio command-line client pre-configured inside the server container

Secure by default, with no shipped credentials. There is no default login in the image. The PostgreSQL password and the Rucio administrator password are both generated uniquely on the first boot of every VM and written to a file that only root can read. The well-known upstream demonstration credential (ddmlab / secret) is never created, and the appliance proves that it is rejected.

Prerequisites

  • An active Azure subscription, an SSH public key, and a VNet with a subnet in the target region
  • A subscription to the Rucio listing on Azure Marketplace
  • Network Security Group rules that allow TCP 22 (admin) and, if you want to reach the REST API remotely, TCP 80. For a single-node evaluation, nothing needs to be open to the internet.

Recommended virtual machine size: Standard_B2s (2 vCPU, 4 GB RAM) is ample for the single-node server and catalogue. Choose a larger size for bigger catalogues and higher request rates.

1

Deploy from the Azure Portal

  • Open the Rucio offer on the Azure Marketplace and choose Get It Now, then Create.
  • Select your subscription, resource group and region, and choose the Standard_B2s size.
  • Provide your SSH public key for the azureuser account.
  • In the Network Security Group, allow inbound TCP 22 from your admin network. Add TCP 80 if you want to reach the REST API from other hosts.
  • Create the VM.
2

Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:rucio-ubuntu-24-04:default:latest
az vm create \
  --resource-group my-rg \
  --name rucio-01 \
  --image sac1647283583153:rucio-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

On first boot, the appliance generates its per-VM secrets, initialises the catalogue, creates the LOCAL storage element and the test scope, and starts the server. This takes a minute or two, and the steps below wait for it.

3

Retrieve the per-VM credentials

Connect over SSH as azureuser and read the credentials file that first boot generated. Only root can read it, so use sudo.

sudo cat /root/rucio-credentials.txt

The file lists the reachable URL, the account (root), the administrator username admin, the storage element and scope, and the unique per-VM administrator password.

The ddmlab / secret demonstration credential that upstream ships is not present, and the server rejects it.

4

Confirm the server is healthy

The Rucio REST API answers an unauthenticated health check at /ping, which returns the running server version. Confirm that the service and the two stack containers (the server and the PostgreSQL catalogue) are up:

systemctl is-active rucio.service
sudo docker ps --format '{{.Names}}\t{{.Status}}' curl -s http://127.0.0.1/ping

The /ping response is the Rucio version ({"version":"41.1.1"}). Port 80 serves the REST API. PostgreSQL has no published host port and is reachable only on the internal Docker network.

5

Explore the catalogue

The rucio and rucio-admin clients are pre-configured inside the rucio-server container. Run them with docker exec.

The commands below confirm who you are authenticated as, list the storage elements and your scopes, and inspect the local RSE's POSIX/file protocol:

sudo docker exec -u root rucio-server rucio whoami
sudo docker exec -u root rucio-server rucio-admin rse list
sudo docker exec -u root rucio-server rucio-admin scope list --account root
sudo docker exec -u root rucio-server rucio-admin rse info LOCAL

whoami reports the root account. The LOCAL RSE uses the native rucio.rse.protocols.posix.Default implementation with the file scheme and prefix /data/local. Uploads therefore write straight to local disk on this single node.

6

Register, upload and list a data file

This is the core Rucio workflow. Create a data file and upload it to the LOCAL storage element under the test scope. The upload registers the file as a data identifier and records a replica in the catalogue.

Then list the data identifiers in the scope and locate the physical replica:

sudo docker exec -u root rucio-server bash -lc 'echo "sac Rucio demo data file" > /tmp/hello-rucio.txt'
sudo docker exec -u root rucio-server rucio --account root upload --rse LOCAL --scope test /tmp/hello-rucio.txt
sudo docker exec -u root rucio-server rucio --account root did list --filter 'type=all' 'test:*'
sudo docker exec -u root rucio-server rucio --account root replica list file test:hello-rucio.txt

The upload reports a successful replica and replication rule on LOCAL. The data identifier test:hello-rucio.txt then appears in the listing. Files need --filter type=all, because datasets and containers are listed by default.

The replica listing shows the file's size, its Adler-32 checksum and its physical location on the RSE.

7

Use the Rucio client from your workstation

The client inside the container is the quickest way to drive the server. You can also install the rucio client on another host and point it at this VM.

Install the client, for example with pip install rucio-clients, and create a rucio.cfg with a [client] section:

# rucio.cfg on your workstation (adjust the host to your VM's address): [client] rucio_host = http://YOUR-VM-IP:80 auth_host = http://YOUR-VM-IP:80 auth_type = userpass username = admin password = <RUCIO_ADMIN_PASSWORD> account = root

Use the RUCIO_ADMIN_PASSWORD from /root/rucio-credentials.txt. For anything beyond a private network, put the API behind TLS first (see the next step).

8

Secure it for production

The appliance serves the REST API over plain HTTP on port 80, which is fine on a private network. Before you expose it more widely, terminate TLS in front of it.

Put a reverse proxy such as Caddy or nginx with certbot in front of port 80 for your domain. Alternatively, enable the server's own SSL (RUCIO_ENABLE_SSL) with your own hostcert.pem. Then restrict inbound access in the Network Security Group to the hosts that need it.

To grow beyond the single node, add more storage elements with rucio-admin rse add, give accounts quota on them, and use replication rules to place copies where you want them.

Notes and support

  • Credentials. The per-VM credentials live in /root/rucio-credentials.txt (mode 0600). They are unique to each VM and never shipped in the image, so keep them safe.
  • Service management. rucio.service manages the Rucio stack. Restart it with sudo systemctl restart rucio.service. The Compose project lives in /var/lib/rucio.
  • Daemons. No Rucio daemons (conveyor, reaper, judge) run on this single-node appliance. You need them only for cross-RSE transfers, deletion and rule evaluation across multiple storage elements.
  • Patching and support. The image is patched at build time and has unattended security upgrades enabled. Every deployment is backed by 24/7 SecureAnyCloud support.

Deploy on Azure

Launch Rucio 41.1.1 on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure