SecureAnyCloud
Deployment guide

OpenSearch log Stack with Fluent Bit

Overview

This image is a working log pipeline on one virtual machine. Fluent Bit collects records from the systemd journal and from any log file you drop into a watched directory, tags each one with the host it came from, and ships it over an encrypted local connection into OpenSearch, which stores them as daily indices. OpenSearch Dashboards is the web interface where you search, filter and chart everything that has been collected.

The three components are already wired to each other and proven to deliver end-to-end, so there is no pipeline to design before your first query returns. All three are Apache 2.0 licensed.

What you get:

  • OpenSearch 3.9.0 with the Security plugin, listening on loopback only.
  • OpenSearch Dashboards 3.9.0, the single externally reachable service, served over this machine's own TLS certificate.
  • Fluent Bit 5.1.3, collecting the systemd journal plus /var/log/sac/sac-logs/*.log.
  • A sac-logs-* index pattern created on first boot, so Discover is populated the moment you log in.
  • A per-VM administrator password, three separate service accounts and a private certificate authority, all generated on the first boot of every machine.
  • An end-to-end pipeline probe you can run at any time.
  • Ubuntu's security-maintained OpenJDK 21 rather than a vendored Java runtime.
  • 24/7 SecureAnyCloud support.

Key facts:

Item Value

Platform Microsoft Azure

Default SSH user azureuser

Dashboards URL https://<vm-ip>:5601/

OpenSearch REST API https://127.0.0.1:9200 (loopback only)

Credentials file /root/opensearch-fluent-bit-credentials.txt

Log drop-in directory /var/log/sac/sac-logs/

Recommended VM size Standard_B2ms (2 vCPU, 8 GB)

No published credential ships in this image. OpenSearch normally bootstraps with the widely documented admin/admin login and a set of demo TLS certificates whose private keys are published upstream. That bootstrap is disabled during the build, so none of it is ever written to disk. Your machine generates its own credentials and its own certificate authority the first time it boots.

Prerequisites

  • An Azure subscription with permission to create virtual machines.
  • A resource group and a virtual network in the region you are deploying into.
  • A network security group that allows inbound TCP 22 from your management address and inbound TCP 5601 from the networks that should reach the dashboard.
  • An SSH key pair.
  • The Azure CLI version 2.50 or later, if you use the command-line path in Step 2.

Recommended VM size: Standard_B2ms (2 vCPU, 8 GB). OpenSearch is a JVM service, Dashboards is a Node service and Fluent Bit runs alongside both, so 8 GB is the minimum for comfortable operation. The image sizes the OpenSearch heap from the memory it finds at boot, so a larger machine is used without any configuration change.

1

Deploy the virtual machine from the Azure Portal

  • In the Azure Portal, choose Create a resource and search the Marketplace for OpenSearch Log Stack with Fluent Bit on Ubuntu 24.04 LTS.
  • On the Basics tab, pick your subscription, resource group and region. Set the virtual machine name and choose size Standard_B2ms or larger.
  • Set the authentication type to SSH public key, set the username to azureuser and paste your public key.
  • On the Networking tab, select your virtual network and subnet. Attach a network security group that allows inbound TCP 22 from your management address and inbound TCP 5601 from the networks that need the dashboard. Leave every other port closed.
  • Choose Review + create.

Only port 5601 needs to be reachable. The OpenSearch REST API on 9200, its transport port 9300 and the Fluent Bit monitoring port 2020 are all bound to the loopback interface inside the machine. They cannot be reached from the network.

2

Deploy the virtual machine from the Azure CLI

RG="logging-prod" LOCATION="eastus" VM_NAME="sac-logstack" az group create --name "$RG" --location "$LOCATION" az network vnet create --resource-group "$RG" --name logstack-vnet \
  --address-prefix 10.10.0.0/16 --subnet-name logstack-subnet \
  --subnet-prefix 10.10.1.0/24 --location "$LOCATION" az network nsg create --resource-group "$RG" --name logstack-nsg --location "$LOCATION" az network nsg rule create --resource-group "$RG" --nsg-name logstack-nsg \
  --name allow-ssh --priority 1000 --destination-port-ranges 22 \
  --access Allow --protocol Tcp --source-address-prefixes "<your-management-ip>/32" az network nsg rule create --resource-group "$RG" --nsg-name logstack-nsg \
  --name allow-dashboards --priority 1010 --destination-port-ranges 5601 \
  --access Allow --protocol Tcp --source-address-prefixes "<your-management-ip>/32" az vm image terms accept --urn sac1647283583153:opensearch-fluent-bit:default:latest az vm create --resource-group "$RG" --name "$VM_NAME" \
  --image "sac1647283583153:opensearch-fluent-bit:default:latest" \
  --size Standard_B2ms --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values ~/.ssh/id_rsa.pub \
  --vnet-name logstack-vnet --subnet logstack-subnet --nsg logstack-nsg \
  --public-ip-sku Standard --location "$LOCATION"
3

Connect over SSH

Connect as the azureuser account with your SSH key.

ssh azureuser@<vm-ip>

Find the VM address with this command:

az vm show --resource-group "$RG" --name "$VM_NAME" --show-details --query publicIps -o tsv
4

Wait for first boot to finish

First boot generates this machine's certificate authority, its passwords and its Dashboards index pattern, then starts the three services. OpenSearch loads roughly twenty plugins on a cold start, so allow several minutes on a 2 vCPU machine.

The sentinel file appears when first boot is complete:

if [ -f /var/lib/sac/data/opensearch-fluent-bit-firstboot.done ]; then echo "first boot complete" else echo "first boot still running" fi

Check that all three services are running:

systemctl is-active opensearch opensearch-dashboards fluent-bit

Expected output:

active active active
5

Retrieve your credentials

Every password on this machine was generated on its first boot and exists nowhere else. Read them with sudo:

sudo cat /root/opensearch-fluent-bit-credentials.txt

The file is mode 0600 and owned by root. It contains the dashboard URL, the admin password you log in with, and the two service-account passwords used internally by Dashboards and Fluent Bit.

To confirm that the published demo login does not work, run:

sudo curl -s -o /dev/null -w 'HTTP %{http_code}\n' \
  --cacert /etc/sac/opensearch-root-ca.pem \
  -u 'admin:admin' https://127.0.0.1:9200/

Expected output:

HTTP 401
6

Open OpenSearch Dashboards

Browse to https://<vm-ip>:5601/. The certificate was generated on this machine, so your browser warns that it is not signed by a public authority. The Security Recommendations section explains how to replace it with your own certificate.

Log in with the username admin and the admin_password value from the credentials file.

Dashboards opens on Discover with the sac-logs-* index pattern already selected, because first boot created it for you. The histogram shows record volume over time. The table below it lists the individual records already collected from this machine.

Type a query into the search bar to filter. Every record collected by this appliance carries an appliance field, so this query matches everything the stack has ingested:

appliance: "sac-opensearch-log-stack"

The index pattern lists every field OpenSearch has mapped, including the journald metadata and the fields Fluent Bit adds:

7

Ship your own logs

Anything written to a .log file under /var/log/sac/sac-logs/ is picked up automatically. Fluent Bit rescans that directory every five seconds, so a file created after boot is collected without restarting anything.

echo "$(date -Is) payment-service order failed id=42" | sudo tee -a /var/log/sac/sac-logs/app.log

Wait a few seconds, then search for the line in Discover, or from the command line as shown in Step 9.

To ship logs from other machines, point any Fluent Bit, Fluentd or syslog forwarder at this host and write into the same index prefix. Alternatively, mount or sync the remote log files into /var/log/sac/sac-logs/.

8

Prove the pipeline end to end

The image ships a probe that emits a uniquely tagged line and then retrieves that exact line from OpenSearch. It is the fastest way to confirm that collection, indexing and search all work:

sudo /usr/local/sbin/opensearch-fluent-bit-e2e.sh

Expected output:

E2E: emitted probe token SACE2E7fdf3040b36adaab3a985421 into /var/log/sac/sac-logs/sac-e2e-probe.log E2E PASS: token SACE2E7fdf3040b36adaab3a985421 emitted by Fluent Bit and retrieved from OpenSearch after 1 poll(s) {"log":"2026-08-07T12:58:10+00:00 sac end-to-end pipeline probe token=SACE2E7fdf3040b36adaab3a985421","hostname":"ubuntu-opensearch-fluent-bit-build-08071020-6c95e9","appliance":"sac-opensearch-log-stack","@timestamp":"2026-08-07T12:58:10.003Z"}

The token is different on every run. If the probe cannot retrieve its own line, it exits with a non-zero status and says so instead of reporting success.

9

Search from the REST API

The OpenSearch API is on loopback. Run these commands on the machine itself, or forward the port over SSH with ssh -L 9200:127.0.0.1:9200 azureuser@<vm-ip>.

List the daily indices:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/sac/opensearch-root-ca.pem -u "admin:$PW" \
  'https://127.0.0.1:9200/_cat/indices/sac-logs-*?v&h=health,index,docs.count,store.size'

Search for a term:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/sac/opensearch-root-ca.pem -u "admin:$PW" \
  -H 'Content-Type: application/json' \
  'https://127.0.0.1:9200/sac-logs-*/_search?size=1' \
  -d '{"query":{"match":{"log":"payment-service"}}}'

Check cluster health:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/sac/opensearch-root-ca.pem -u "admin:$PW" \
  'https://127.0.0.1:9200/_cluster/health?pretty'

A single-node cluster reports green because the shipped index template creates one shard with no replica.

Server Components

Component Version Purpose

OpenSearch 3.9.0 Search and analytics engine storing the log records

OpenSearch Dashboards 3.9.0 Web interface for search, filtering and visualisation

OpenSearch Security plugin bundled TLS and authentication on the REST and transport layers

Fluent Bit 5.1.3 Log collector and forwarder

OpenJDK 21 Java runtime, from Ubuntu, security maintained

Ubuntu 24.04 LTS Base operating system

Filesystem Layout

Path Purpose

/usr/share/opensearch OpenSearch installation

/etc/opensearch/opensearch.yml Cluster and security configuration

/etc/opensearch/certs/ This machine's certificate authority, node and admin certificates

/etc/opensearch/opensearch-security/ Account database, roles and role mappings

/var/lib/opensearch Index data

/var/log/opensearch/ OpenSearch logs

/etc/opensearch-dashboards/opensearch_dashboards.yml Dashboards configuration

/etc/fluent-bit/fluent-bit.conf The collection pipeline

/etc/fluent-bit/opensearch.env Fluent Bit's OpenSearch credential, root-only

/var/log/sac/sac-logs/ Drop .log files here to have them collected

/etc/sac/opensearch-root-ca.pem Certificate authority, readable for verifying connections

/root/opensearch-fluent-bit-credentials.txt Per-machine credentials, mode 0600

Managing the Services

sudo systemctl status opensearch
sudo systemctl status opensearch-dashboards
sudo systemctl status fluent-bit

Restart the stack in dependency order:

sudo systemctl restart opensearch
sudo systemctl restart opensearch-dashboards
sudo systemctl restart fluent-bit

OpenSearch takes several minutes to become available on a 2 vCPU machine because of the number of plugins it loads. Until OpenSearch is answering, Dashboards reports HTTP 503 with "server is not ready yet". This is expected during that window.

Scripts and Log Files

Path Purpose

/usr/local/sbin/opensearch-fluent-bit-e2e.sh End-to-end pipeline probe

/usr/local/sbin/opensearch-fluent-bit-bootstrap.sh Generates this machine's certificates and passwords

/usr/local/sbin/opensearch-fluent-bit-credguard.sh Refuses to start OpenSearch if a published credential is in effect

/usr/local/sbin/opensearch-fluent-bit-seed-dashboards.sh Creates the sac-logs-* index pattern

/var/log/sac/sac-firstboot.log First boot log

/var/log/opensearch/sac-logs.log OpenSearch server log

/stage/scripts/initial_boot_update.sh Standard SecureAnyCloud first-boot update hook

On Startup

opensearch-fluent-bit-firstboot.service runs once on the first boot of each machine. It generates a private certificate authority, a node certificate and an admin client certificate. It then creates three passwords, writes them into the account database and applies that configuration to the cluster. Finally, it starts the three services and creates the Dashboards index pattern.

The service then writes /var/lib/sac/data/opensearch-fluent-bit-firstboot.done and disables itself, so later reboots are ordinary starts.

The three product services do not start until first boot has published /var/lib/sac/data/opensearch-fluent-bit-bootstrap.ready. This guarantees that no service can run against credentials that have not been generated for this machine.

Troubleshooting

Dashboards shows "server is not ready yet". OpenSearch has not finished starting. Watch sudo journalctl -u opensearch -f and wait, then reload. A cold start takes several minutes on 2 vCPU.

The dashboard does not load at all. Confirm the network security group allows inbound TCP 5601 from your address and that the service is listening:

sudo ss -ltnp | grep 5601

No records in Discover. Confirm Fluent Bit is running and delivering:

sudo systemctl status fluent-bit --no-pager | head -5 sudo curl -s http://127.0.0.1:2020/api/v1/metrics | head -c 400

Then run the end-to-end probe from Step 8. It reports exactly where delivery stops.

Your own log file is not collected. It must end in .log and live directly in /var/log/sac/sac-logs/. Confirm Fluent Bit can read it:

ls -l /var/log/sac/sac-logs/

Forgotten admin password. It exists only in /root/opensearch-fluent-bit-credentials.txt on the machine. Read it with sudo. It cannot be recovered from the image.

Disk usage grows as logs accumulate. Log indices grow over time. Check disk usage with:

df -h /
sudo du -sh /var/lib/opensearch

List the daily indices, then delete the ones you no longer need:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/sac/opensearch-root-ca.pem -u "admin:$PW" \
  'https://127.0.0.1:9200/_cat/indices/sac-logs-*?v&h=index,docs.count,store.size'

Security Recommendations

  • Keep 9200 and 2020 on loopback. The image binds them there deliberately. Exposing the search API to a network removes the boundary that makes Dashboards the single, authenticated front door.
  • Restrict inbound 5601 to the networks that need the dashboard, and inbound 22 to your management addresses.
  • Replace the TLS certificate. The certificate authority generated on first boot is private to this machine, which is why browsers warn about it. To use a certificate from your own authority, replace /etc/opensearch-dashboards/certs/node.pem and node-key.pem, keeping the file ownership and mode intact. Then restart Dashboards.
  • Rotate the administrator password if it has been shared. Passwords live in the account database at /etc/opensearch/opensearch-security/internal_users.yml as bcrypt hashes. Apply changes with the security plugin's securityadmin.sh tool.
  • Create per-person accounts instead of sharing admin. OpenSearch Security supports roles and index-level permissions, so read-only users can be given access to the log indices alone.
  • Take backups with the OpenSearch snapshot API if the log history matters to you. The image ships no snapshot repository, so you choose the destination.
  • Keep the machine patched. Unattended security upgrades are enabled, and the OpenSearch and Fluent Bit package repositories remain configured so both keep receiving updates.

Support

SecureAnyCloud provides 24/7 support for this image. Contact support through secureanycloud.com with your Azure subscription id and the virtual machine name.

For questions about the software itself, the upstream projects are OpenSearch and Fluent Bit, both Apache 2.0 licensed.

Deploy on Azure

Launch OpenSearch Log Stack with Fluent Bit on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure