OpenObserve with Vector

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet with a subnet in the target region.

Standard_B2s (2 vCPU / 4 GiB RAM) is a comfortable starting size. On a freshly deployed VM the whole stack uses well under 1 GiB. Scale the VM and the data disk up as your ingest volume grows.

Configure the network security group (NSG) to allow inbound 22/tcp from your management network and 443/tcp from wherever you browse the UI and ship telemetry from. 80/tcp is optional and only redirects callers to HTTPS.

The TLS certificate generated on first boot is self-signed, so browsers show a warning until you replace it with your own certificate (see Maintenance). The curl examples below therefore use -k.

Step 1 - Deploy from the Azure Marketplace

In the Azure portal, choose Create a resource, search for OpenObserve with Vector on Ubuntu 24.04 LTS by SecureAnyCloud, and select Create.

Pick your subscription, resource group and region, and choose Standard_B2s or larger. Select SSH public key authentication and enter azureuser as the username.

On the Networking tab, allow inbound 22 and 443. Then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

Replace the resource group and region with your own values.

az group create --name observability-rg --location eastus
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:openobserve-vector:default:latest
az vm create \
  --resource-group observability-rg \
  --name openobserve-vector-01 \
  --image sac1647283583153:openobserve-vector:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
az vm open-port --resource-group observability-rg --name openobserve-vector-01 --port 443 --priority 1001

Step 3 - Connect to your VM

Connect over SSH as azureuser:

ssh azureuser@<your-vm-ip>

Step 4 - Confirm the services are running

Three services make up the appliance: OpenObserve (the backend), Vector (the collector) and nginx (the TLS front door). Check that all three are active:

systemctl is-active openobserve vector nginx

Expected output:

active active active

Next, confirm the versions and the network posture. Only SSH, HTTP and HTTPS are reachable from outside the VM. OpenObserve, Vector's management API and Vector's ingest listeners are all bound to loopback.

sudo /usr/local/bin/openobserve --version vector --version | awk '{print $1, $2}' ss -tlnH | awk '{print $4}' | grep -vE '^(127\.|\[::1\])' | sed -E 's/.*:([0-9]+)$/\1/' | sort -un | tr '\n' ' '

Expected output:

openobserve v0.92.2 vector 0.58.0 22 80 443

Step 5 - Retrieve your credentials

Every secret on this VM was generated during its first boot. Nothing shipped inside the image, and no two deployments share a value. Read the credentials file with sudo:

sudo cat /root/openobserve-vector-credentials.txt

The file contains your OpenObserve sign-in, the credential your other machines use to ship telemetry in, and the ingest token Vector itself uses:

OPENOBSERVE_URL=https://<your-vm>/ OPENOBSERVE_USERNAME=admin@openobserve.local OPENOBSERVE_PASSWORD=<generated on this VM> INGEST_URL=https://<your-vm>/ingest/ OTLP_URL=https://<your-vm>/otlp/ INGEST_USERNAME=ingest INGEST_PASSWORD=<generated on this VM> OPENOBSERVE_INGEST_USER=default OPENOBSERVE_INGEST_TOKEN=<generated on this VM>

Confirm that the TLS front door answers and that plain HTTP redirects to it:

curl -sk -o /dev/null -w 'healthz: HTTP %{http_code}\n' https://127.0.0.1/healthz
curl -sk -o /dev/null -w 'port 80: HTTP %{http_code}\n' http://127.0.0.1/

Expected output:

healthz: HTTP 200 port 80: HTTP 301

/healthz is deliberately unauthenticated so that an Azure Load Balancer health probe can use it. Everything else requires a login.

Step 6 - Sign in to the web UI

Open https://<your-vm-ip>/ in a browser. Accept the certificate warning. The certificate was generated on your VM and is not signed by a public authority (see Maintenance to replace it).

Sign in with the OPENOBSERVE_USERNAME and OPENOBSERVE_PASSWORD from Step 5.

Step 7 - View your host logs

Your host logs are already there. Vector has been reading this host's journal since first boot and writing it into the host_logs stream.

In the UI, choose Logs in the left sidebar and select the host_logs stream. To make the records easier to read, hover a field in the left-hand field list and use its interesting field control to promote service and message into table columns.

You can confirm the same thing from the shell. This command asks OpenObserve for the streams Vector is feeding:

PW='<OPENOBSERVE_PASSWORD>'
RESP=$(curl -sk -u "admin@openobserve.local:${PW}" https://127.0.0.1/api/default/streams)
echo "${RESP}" | jq -r '(.list // [])[].name' 2>/dev/null \
  || echo "not authenticated - check OPENOBSERVE_PASSWORD from Step 5"

Expected output:

host_logs host_metrics remote_logs

The otlp_logs stream appears as soon as the first OTLP event arrives (Step 10).

Step 8 - Read the shipped dashboard

Choose Dashboards in the sidebar and open Host Overview (Vector). First boot created this dashboard. It charts the telemetry Vector collects from this host: log volume, memory used, load average and filesystem usage.

Vector's host_metrics source collects host metrics and stores them in the host_metrics stream. Each record is one metric sample, with the metric name and its value. The dashboard panels query that stream directly, so you can build your own panels the same way.

Step 9 - Ship logs from your other machines

The appliance exposes an authenticated JSON endpoint at https://<your-vm-ip>/ingest/. Vector processes anything you POST there and writes it to the remote_logs stream. Authenticate with the INGEST_USERNAME and INGEST_PASSWORD from Step 5.

INGEST_USER='<INGEST_USERNAME>' INGEST_PASS='<INGEST_PASSWORD>' curl -sk -u "${INGEST_USER}:${INGEST_PASS}" \
  -H 'Content-Type: application/json' \
  -d '[{"message":"order-service timeout id=8821","level":"warn","app":"order-service"}]' \
  -o /dev/null -w 'authenticated ingest: HTTP %{http_code}\n' \
  https://127.0.0.1/ingest/

Expected output:

authenticated ingest: HTTP 200

Without credentials, the endpoint refuses the request:

curl -sk -X POST -H 'Content-Type: application/json' \
  -d '[{"message":"no credentials"}]' \
  -o /dev/null -w 'anonymous ingest: HTTP %{http_code}\n' \
  https://127.0.0.1/ingest/

Expected output:

anonymous ingest: HTTP 401

Read the event back from the remote_logs stream:

PW='<OPENOBSERVE_PASSWORD>'
S=$(( ($(date -u +%s) - 600) * 1000000 ));
E=$(( $(date -u +%s) * 1000000 )) curl -sk -u "admin@openobserve.local:${PW}" -H 'Content-Type: application/json' \
  -d "{\"query\":{\"sql\":\"SELECT app, level, message FROM \\\"remote_logs\\\" ORDER BY _timestamp DESC\",\"start_time\":${S},\"end_time\":${E},\"from\":0,\"size\":3}}" \
  https://127.0.0.1/api/default/_search > /tmp/remote-logs.json jq -c '(.hits // [])[]' /tmp/remote-logs.json 2>/dev/null \
  || echo "not authenticated - check OPENOBSERVE_PASSWORD from Step 5"

From another machine, point any HTTP-capable log shipper at the same endpoint. For example, configure a Vector instance elsewhere in your estate:

sinks: sac_appliance: type: http inputs: [your_source] uri: "https://<your-vm-ip>/ingest/" method: post auth: strategy: basic user: "ingest" password: "<your INGEST_PASSWORD>" encoding: codec: json

Step 10 - Ship OpenTelemetry (OTLP) telemetry

The appliance also accepts OTLP over HTTP at https://<your-vm-ip>/otlp/, protected by the same ingest credential. Point any OpenTelemetry SDK or Collector at it:

OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-vm-ip>/otlp OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64 of ingest:INGEST_PASSWORD>

Logs that arrive this way land in the otlp_logs stream. Like /ingest/, the endpoint rejects unauthenticated callers:

curl -sk -X POST -H 'Content-Type: application/json' -d '{}' \
  -o /dev/null -w 'anonymous OTLP: HTTP %{http_code}\n' \
  https://127.0.0.1/otlp/v1/logs

Expected output:

anonymous OTLP: HTTP 401

The OTLP gRPC listener is deliberately bound to loopback only and is not exposed outside the VM, because the front door's HTTP authentication cannot protect gRPC. Use the OTLP/HTTP endpoint above from remote machines.

Step 11 - Verify the whole pipeline end to end

The image ships a probe that confirms the pipeline is connected end to end. It writes a uniquely tagged line into this host's journal, then asks OpenObserve's search API for that exact line.

sudo /usr/local/sbin/openobserve-vector-e2e.sh

Expected output:

E2E_PIPELINE_OK tag=sac-e2e-103075c9cee57be6 path=journald->vector->openobserve/host_logs

A second script also checks the authentication posture. It confirms that anonymous and wrong-password calls are refused, that your per-VM password works, and that Vector's ingest token is ingest-only and cannot read your data back:

sudo /usr/local/sbin/openobserve-vector-roundtrip.sh

Step 12 - Where your data lives

The telemetry store sits on a dedicated 20 GiB Azure data disk, separate from the OS disk, so you can grow it independently. Confirm the mount:

findmnt -no SOURCE,TARGET,FSTYPE /var/lib/openobserve df -h /var/lib/openobserve | tail -1

Expected output. The device name varies with how Azure attaches the disk, and both /dev/sda and /dev/sdc are normal. The mount point is what matters:

/dev/sdc /var/lib/openobserve ext4 /dev/sdc 20G 80M 19G 1% /var/lib/openobserve

To grow the store, resize the disk in the Azure portal, then extend the filesystem with sudo resize2fs "$(findmnt -no SOURCE /var/lib/openobserve)".

Step 13 - Add your own sources and destinations

Vector's pipeline lives at /etc/vector/vector.yaml. First boot generates it from the template at /usr/share/sac/vector.yaml.tmpl and substitutes this VM's credentials. List the component names in the pipeline:

sudo grep -E '^\s{2}[a-z_]+:$' /etc/vector/vector.yaml | tr -d ' :' | tr '\n' ' '

Expected output:

host_journal host_metrics remote_http remote_otlp drop_self_noise journal_tagged metrics_as_logs metrics_tagged remote_tagged otlp_tagged oo_host_logs oo_host_metrics oo_remote_logs oo_otlp_logs

To add a source, such as a file, a syslog listener or a cloud provider's log stream, edit that file. Add your source and wire it into one of the existing OpenObserve sinks. Then validate the configuration and reload Vector:

sudo vector validate /etc/vector/vector.yaml
sudo systemctl reload vector

Keep two things in mind when you edit the pipeline:

• Vector's journald source emits field names in upper case (_SYSTEMD_UNIT, PRIORITY). OpenObserve lower-cases them when it stores them, so the name you see in the search UI is not the name to use in a VRL expression.
• The shipped pipeline drops OpenObserve's own routine INFO log lines. Without that filter, OpenObserve logs every request Vector makes to it, Vector ships that line back, and the two feed each other. In testing, that loop accounted for around 90% of all collected log volume. Warnings and errors from OpenObserve are still collected.

Maintenance

Replace the TLS certificate. The certificate generated at first boot is self-signed. To use your own, replace the two files and reload nginx:

sudo cp your-certificate.crt /etc/ssl/sac/openobserve-vector.crt
sudo cp your-private.key /etc/ssl/sac/openobserve-vector.key
sudo chown root:www-data /etc/ssl/sac/openobserve-vector.key
sudo chmod 0640 /etc/ssl/sac/openobserve-vector.key
sudo nginx -t && sudo systemctl reload nginx

Rotate the ingest credential. The credential your remote machines use is an nginx basic-auth entry. To change it, run:

sudo htpasswd -B /etc/nginx/.htpasswd-ingest ingest

Rotate Vector's OpenObserve token. Vector authenticates with an ingest-only token. You can disable or rotate it in the UI under Data sources > Manage Tokens. If you rotate it, update /etc/openobserve/ingest-user.env and the sink credentials in /etc/vector/vector.yaml, then run sudo systemctl restart vector.

Change the OpenObserve password. Open Settings > Users in the web UI. You can also create additional users there.

Set retention. OpenObserve's retention settings are under Settings. Configure per-stream retention on the Streams page.

Apply operating system updates. Ubuntu's unattended-upgrades delivers them and stays enabled on the image.

Restart the stack. Run:

sudo systemctl restart openobserve vector nginx

Licensing and trademarks

OpenObserve is distributed under the GNU Affero General Public License v3.0. This image ships the unmodified open-source build of OpenObserve 0.92.2. The corresponding source is available from github.com/openobserve/openobserve. OpenObserve also publishes a separately licensed enterprise build, which this image does not contain.

Vector is distributed under the Mozilla Public License 2.0. This image ships the unmodified official release of Vector 0.58.0. The corresponding source is available from github.com/vectordotdev/vector.

"OpenObserve" is a trademark of OpenObserve Inc. and "Vector" is a project of Datadog, Inc. Both names are used only to identify the software this image contains. SecureAnyCloud is not affiliated with, endorsed by or sponsored by OpenObserve Inc. or Datadog, Inc.

Support

SecureAnyCloud provides 24/7 support for this image. Contact sales@secureanycloud.com and include your Azure subscription ID and the VM name.

Deploy on Azure

Launch OpenObserve with Vector on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com