ELK 9

On the Marketplace listing, choose Get it now. The Azure portal then walks you through your subscription, resource group, region and size, your SSH key and the inbound ports.

Allow SSH (22) only for inbound ports, then select Review and create.

The image carries a Marketplace purchase plan, so accept its terms once per subscription and then create the virtual machine. Replace <your-vm-size> with the size you chose:

az vm image terms accept --urn sac1647283583153:elk-9-ubuntu-24-04:default:latest
az vm create \
  --resource-group myResourceGroup \
  --name my-elk-vm \
  --image sac1647283583153:elk-9-ubuntu-24-04:default:latest \
  --size <your-vm-size> \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --nsg my-ssh-only-nsg \
  --public-ip-sku Standard

On your own workstation, find the public IP and connect:

az vm show --resource-group myResourceGroup --name my-elk-vm --show-details --query publicIps -o tsv ssh azureuser@<vm-ip>

First boot takes a few minutes after the VM is running. The first-boot service sizes the heaps, restarts Elasticsearch, sets the per-VM passwords, then restarts Kibana and Logstash. If you connect straight away, wait for it to finish:

sudo cloud-init status --wait

It prints status: done once first boot has finished.

systemctl is-active elasticsearch.service kibana.service logstash.service elk-firstboot.service
sudo ss -Htln '( sport = :9200 or sport = :5601 or sport = :5044 )' | awk '{print $4}'

All four services report active. Elasticsearch listens on 127.0.0.1:9200 only (Java prints it as [::ffff:127.0.0.1]:9200). Kibana listens on 0.0.0.0:5601 and the Logstash Beats input on *:5044.

Your network security group decides who can reach 5601 and 5044. With the SSH-only group from the prerequisites, nobody outside the VM can.

The elastic password is unique to this VM and is written at first boot to a root-only file (mode 0600):

sudo stat -c '%a %U:%G %n' /stage/scripts/elk-credentials.log
sudo grep -E '^(ELASTIC_USER|ELASTICSEARCH_URL|LOGSTASH_BEATS_PORT)=' /stage/scripts/elk-credentials.log

It prints 600 root:root /stage/scripts/elk-credentials.log, then ELASTIC_USER=elastic, ELASTICSEARCH_URL=http://127.0.0.1:9200 and LOGSTASH_BEATS_PORT=5044. The file's KIBANA_URL line holds the VM's private address; from your workstation, use the SSH tunnel in Step 9 instead.

When you need the password, read it into a variable instead of printing it on screen. The commands in this guide use $ES_PW, so set it in each new shell:

export ES_PW="$(sudo sed -n 's/^ELASTIC_PASSWORD=//p' /stage/scripts/elk-credentials.log)"

Store the password safely, because elastic is the superuser.

Elasticsearch refuses a request with no login and one with a wrong login. It accepts the VM's own:

curl -s -o /dev/null -w 'no login -> HTTP %{http_code}\n' http://127.0.0.1:9200/
curl -s -o /dev/null -w 'wrong login -> HTTP %{http_code}\n' -u 'elastic:not-the-real-one' http://127.0.0.1:9200/
curl -s -o /dev/null -w 'VM login -> HTTP %{http_code}\n' -u "elastic:$ES_PW" http://127.0.0.1:9200/

The first two return HTTP 401 and the last HTTP 200. Kibana does the same for its pages: curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' http://127.0.0.1:5601/ prints 302 http://127.0.0.1:5601/login?next=%2F.

curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/_cluster/health?filter_path=cluster_name,status,number_of_nodes&pretty'
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/?filter_path=name,cluster_name,version.number&pretty'

The cluster sac-elk reports "status" : "green" with one node, elk-node-1, and the version (9.3.4 on image version 1.0.20260502).

A single-node cluster cannot place replica shards. Create indices with no replicas so that they stay green:

curl -s -u "elastic:$ES_PW" -X PUT http://127.0.0.1:9200/app-logs \
  -H 'Content-Type: application/json' -d '{"settings":{"number_of_replicas":0}}'
curl -s -u "elastic:$ES_PW" -X POST 'http://127.0.0.1:9200/app-logs/_doc?refresh=true' \
  -H 'Content-Type: application/json' \
  -d '{"@timestamp":"2026-11-02T09:00:00Z","service":"checkout","level":"error","message":"payment gateway timeout after 30s"}'
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/app-logs/_search?q=message:timeout&filter_path=hits.total.value,hits.hits._source.message&pretty'
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/_cat/indices/app-logs?v&h=health,index,docs.count'

The search finds the document ("value" : 1, payment gateway timeout after 30s) and the index is green with one document. An index created without that setting, such as the daily filebeat-* indices Logstash creates, shows yellow on one node. Its data is complete, and only the replica has nowhere to go.

Keep port 5601 closed in the network security group and carry Kibana over SSH. On your workstation, open the tunnel:

ssh -N -L 5601:127.0.0.1:5601 azureuser@<vm-ip>

Leave the tunnel running and open http://localhost:5601 in your browser. The traffic travels inside the encrypted SSH connection.

Sign in with the user elastic and the password from Step 5. Kibana opens, with Discover, Dashboards and its other applications in the main menu.

Logstash's default pipeline takes events from Beats agents on port 5044 and writes them to Elasticsearch as <beat>-<date> indices.

This example runs the whole path on the VM itself, with Filebeat reading one log file. Elastic's 9.x repository is already configured, so install Filebeat and give it a minimal configuration:

sudo apt-get -q update
sudo apt-get install -y filebeat
sudo tee /etc/filebeat/filebeat.yml >/dev/null <<'EOF' filebeat.inputs: - type: filestream id: demo-app paths: - /var/log/demo-app.log output.logstash: hosts: ["127.0.0.1:5044"] EOF
sudo filebeat test config -c /etc/filebeat/filebeat.yml
sudo filebeat test output -c /etc/filebeat/filebeat.yml
sudo systemctl enable --now filebeat

filebeat test config prints Config OK, and filebeat test output ends with talk to server... OK. Write a few lines to the log and count what arrives:

for i in 1 2 3; do echo "$(date -Is) checkout ERROR payment gateway timeout order=$((1040+i))" | sudo tee -a /var/log/demo-app.log >/dev/null; done sleep 15 curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/_cat/indices/filebeat-*?v&h=health,index,docs.count'

The three lines arrive in today's filebeat-* index (for example filebeat-2026.11.02 with docs.count 3).

To ship logs from your other servers, install Filebeat on each one from the same Elastic 9.x repository. Set output.logstash.hosts to this VM's private address and port 5044. Then add a network security group rule that allows TCP 5044 from those servers' private addresses only.

The Beats input on this image accepts plain TCP with no client authentication, so never open 5044 to the internet (see Security notes).

Create a Kibana data view for the Filebeat indices, either in Kibana (Stack Management, Data views) or with Kibana's API:

curl -s -u "elastic:$ES_PW" -X POST http://127.0.0.1:5601/api/data_views/data_view \
  -H 'kbn-xsrf: true' -H 'Content-Type: application/json' \
  -d '{"data_view":{"title":"filebeat-*","name":"Filebeat logs","timeFieldName":"@timestamp"}}'

Open Discover in Kibana and choose the Filebeat logs data view. Search with KQL, for example message : timeout.

The default pipeline is /etc/logstash/conf.d/01-beats-input.conf. Logstash loads every .conf file in /etc/logstash/conf.d/, so add your own filters and outputs there. Restart Logstash after a change:

sudo systemctl restart logstash.service

After a restart, Logstash takes a minute or two before it listens on 5044 again. sudo journalctl -u logstash -f shows Pipelines running when it is ready.

Its Elasticsearch output signs in as elastic, with the password taken from the root-only environment file /etc/default/logstash (mode 0600). A pipeline file never holds the password itself.

The first-boot service sets the Java heap sizes from the VM's memory and records what it chose:

sudo grep 'Adjusting heaps' /var/log/elk-firstboot.log

Elasticsearch takes about 40 percent of memory, and Logstash about 15 percent, up to 1 GB. To change them later, edit /etc/elasticsearch/jvm.options.d/sac-heap.options (keep -Xms and -Xmx equal) or the -Xms/-Xmx lines in /etc/logstash/jvm.options, then restart that service. If you resize the VM, adjust the heaps yourself, because first boot runs only once.

The image was patched when it was built. Ubuntu security updates install automatically (unattended-upgrades is active), but other updates do not. Update a new VM before you put it into service, and regularly afterwards:

sudo apt-get -q update
sudo apt-get -y upgrade

Elasticsearch, Logstash and Kibana are held, so this leaves them alone (apt-mark showhold lists all three). Elastic's 9.x repository carries newer 9.x releases than the image.

To upgrade the stack within 9.x, release the hold and upgrade all three together while keeping SecureAnyCloud's configuration files. Then hold them again and restart:

sudo apt-mark unhold elasticsearch logstash kibana
sudo apt-get install -y -o Dpkg::Options::=--force-confold elasticsearch logstash kibana
sudo apt-mark hold elasticsearch logstash kibana
sudo systemctl daemon-reload
sudo systemctl restart elasticsearch.service kibana.service logstash.service

We ran exactly this upgrade from 9.3.4 to 9.5.5. The elastic password, the indices and the Beats pipeline carried over, and events flowed again once Logstash had restarted.

--force-confold keeps the existing kibana.yml and logstash.yml. The package's new defaults are saved beside them as .dpkg-dist files.

Read Elastic's release notes before you upgrade and take a snapshot of the OS disk first. Reboot if a new kernel was installed.