Elasticsearch 9

In the Azure Portal, open Marketplace, search for Elasticsearch 9, and select the SecureAnyCloud publisher entry. Click Create to start the wizard.

On the Basics tab, choose your subscription, target resource group and region. The region must match the region where your Azure Compute Gallery exposes the image. Set the virtual machine name.

Choose a size of Standard_D4s_v3 or larger, which provides four virtual CPUs and sixteen gigabytes of memory. Elasticsearch is a JVM workload. For production it benefits from a heap of around eight gigabytes, which must not exceed fifty percent of host memory. Kibana uses around seven hundred megabytes of resident memory.

Choose SSH public key as the authentication type, set a username of your choice, and paste your SSH public key.

On the Disks tab, the recommended OS disk type is Premium SSD. Leave the OS disk size at the default. You can attach a separate Premium SSD data disk now if you plan to move the Elasticsearch data directory to it. You can also do that after the cluster is running by following Step 12.

On the Networking tab, select your existing virtual network and subnet. Attach a network security group that allows:

• inbound TCP 22 from your management IP range
• inbound TCP 5601 only from the virtual network CIDR or a jump host
• inbound TCP 9200 only from the application servers that need it

Do not expose 9200 or 5601 to the public internet. The demo self-signed certificates include a node certificate that was generated on first boot and has no external chain of trust. Any exposure beyond a private network requires the certificate replacement described in Step 13.

On the Management, Monitoring and Advanced tabs, the defaults are appropriate. Click Review + create, wait for validation to pass, then click Create.

Deployment takes around three minutes to reach the running state. Elasticsearch and Kibana need a further sixty to ninety seconds to come up inside the virtual machine.

If you prefer the command line, accept the Marketplace terms once, then deploy from the Marketplace image URN:

RG="elasticsearch-prod" LOCATION="eastus" VM_NAME="elasticsearch-node-1" ADMIN_USER="esuser" IMAGE_URN="sac1647283583153:elasticsearch-9-ubuntu-22-04:default:latest" SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" az group create --name "$RG" --location "$LOCATION" az network vnet create \
  --resource-group "$RG" \
  --name elasticsearch-vnet \
  --address-prefix 10.40.0.0/16 \
  --subnet-name elasticsearch-subnet \
  --subnet-prefix 10.40.1.0/24 az network nsg create --resource-group "$RG" --name elasticsearch-nsg az network nsg rule create --resource-group "$RG" --nsg-name elasticsearch-nsg \
  --name allow-ssh --priority 1000 \
  --source-address-prefixes "<your-management-cidr>" \
  --destination-port-ranges 22 --protocol Tcp --access Allow az network nsg rule create --resource-group "$RG" --nsg-name elasticsearch-nsg \
  --name allow-rest --priority 1010 \
  --source-address-prefixes 10.40.0.0/16 \
  --destination-port-ranges 9200 --protocol Tcp --access Allow az network nsg rule create --resource-group "$RG" --nsg-name elasticsearch-nsg \
  --name allow-kibana --priority 1020 \
  --source-address-prefixes 10.40.0.0/16 \
  --destination-port-ranges 5601 --protocol Tcp --access Allow
# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:elasticsearch-9-ubuntu-22-04:default:latest
az vm create \
  --resource-group "$RG" \
  --name "$VM_NAME" \
  --image "$IMAGE_URN" \
  --size Standard_D4s_v3 \
  --admin-username "$ADMIN_USER" \
  --ssh-key-values "$SSH_KEY" \
  --vnet-name elasticsearch-vnet \
  --subnet elasticsearch-subnet \
  --nsg elasticsearch-nsg \
  --public-ip-address "" \
  --os-disk-size-gb 64

The final command creates a virtual machine with no public IP, which is the recommended posture for an Elasticsearch node. Connect through a bastion host, the Azure Bastion service or a virtual private network peered into the same virtual network.

Connect with the admin username you chose during deployment and the private key that matches the public key you pasted into the deployment form:

ssh -i ~/.ssh/id_rsa <admin-username>@<vm-private-ip>

Every later command in this guide runs on the virtual machine as the admin user, escalating with sudo where a command needs root. Do not log in directly as the elasticsearch or kibana service users. Those accounts have no login shell and exist for the daemons only.

On first boot, elasticsearch-firstboot.service resets the elastic superuser password and the kibana_system service account password to fresh random values. It wires the kibana_system credentials into kibana.yml and writes the elastic user credentials to /stage/scripts/elasticsearch-credentials.log. Read the file with sudo:

sudo cat /stage/scripts/elasticsearch-credentials.log

The file contains four lines:

username=elastic password=<generated-password> elasticsearch_url=https://<vm-private-ip>:9200 kibana_url=http://<vm-private-ip>:5601

Copy these values somewhere safe. The password is not recoverable from the image, and Step 10 shows how to rotate it. If your environment requires it, store the file off the virtual machine, then tighten the file mode or remove the file. The file is owned by root with mode 0600 by default.

The image ships the following components:

• Elasticsearch 9.x: distributed search and analytics engine, installed at /usr/share/elasticsearch. It runs as elasticsearch.service under the elasticsearch system user.
• Kibana 9.x: web UI for search, visualisation and cluster management, installed at /usr/share/kibana. It runs as kibana.service under the kibana system user.
• Bundled JDK: Elastic-managed Java runtime shipped under /usr/share/elasticsearch/jdk. No separate JDK package is installed, and Elasticsearch uses the bundled JDK by default.
• xpack security: enabled by default, with built-in authentication, TLS on transport and HTTP, role-based access control and API keys. Demo self-signed certificates are generated on first boot under /etc/elasticsearch/certs/.
• First-boot unit: the oneshot elasticsearch-firstboot.service resets the elastic and kibana_system passwords once and wires Kibana to Elasticsearch. It is gated by /var/lib/elasticsearch/.firstboot-done.
• Base OS: Ubuntu 22.04 LTS, a long-term support release with standard cloud-init and the Azure Linux Agent.

These are the paths that matter for operating the image:

• /usr/share/elasticsearch: Elasticsearch install root (ES_HOME)
• /usr/share/kibana: Kibana install root (KIBANA_HOME)
• /etc/elasticsearch/elasticsearch.yml: cluster config
• /etc/elasticsearch/jvm.options.d/: heap and JVM overrides (sac-heap.options sets a 1g default)
• /etc/elasticsearch/certs/: auto-generated HTTP CA, HTTP cert, transport cert and transport CA
• /etc/kibana/kibana.yml: Kibana config (elasticsearch.hosts, kibana_system credentials)
• /var/lib/elasticsearch: index data (move to an attached disk for production, see Step 12)
• /var/log/elasticsearch/: Elasticsearch daemon log directory
• /var/log/kibana/: Kibana daemon log directory
• /etc/systemd/system/elasticsearch-firstboot.service: first-boot credential generator
• /etc/systemd/system/kibana.service.d/10-wait-for-firstboot.conf: systemd drop-in that orders Kibana after first boot
• /opt/sac/bin/setEnv.sh: sources ES_HOME, KIBANA_HOME and PATH
• /opt/sac/bin/start_all.sh: starts Elasticsearch, then Kibana
• /opt/sac/bin/stop_all.sh: stops Kibana, then Elasticsearch
• /stage/scripts/elasticsearch-credentials.log: elastic user credentials plus URLs (file mode 0600)
• /stage/scripts/BUILD_VERSIONS: package versions captured at image build time

systemd manages the two main services independently. Kibana depends on Elasticsearch being up and on the first-boot unit having completed. When you bring the stack up from a cold stop, always start Elasticsearch first.

sudo systemctl start elasticsearch.service
sudo systemctl status elasticsearch.service
sudo systemctl start kibana.service
sudo systemctl status kibana.service

Or use the helper script to start both services:

sudo /opt/sac/bin/start_all.sh

To stop both services:

sudo /opt/sac/bin/stop_all.sh

Both services are enabled at boot. To stop Kibana starting automatically, run sudo systemctl disable kibana.service.

Log output lives under /var/log/elasticsearch/ and /var/log/kibana/. Follow the Elasticsearch daemon log with:

sudo journalctl -u elasticsearch.service -f

Follow the Kibana log with:

sudo journalctl -u kibana.service -f

Load the elastic password into a shell variable, then query the cluster:

ELASTIC_PW=$(sudo awk -F= '/^password=/ {print $2}' /stage/scripts/elasticsearch-credentials.log)
curl -sku "elastic:${ELASTIC_PW}" https://localhost:9200
curl -sku "elastic:${ELASTIC_PW}" https://localhost:9200/_cluster/health?pretty
curl -sku "elastic:${ELASTIC_PW}" https://localhost:9200/_cat/nodes?v

The -k flag tells curl to skip TLS verification. You need it for as long as the image uses the demo self-signed certificate.

From a remote host, replace localhost with the private IP of the virtual machine and copy the elastic password to that host. Add -H 'Content-Type: application/json' when you send JSON bodies.

The root endpoint returns the version, the cluster name and the Elasticsearch tagline:

{ "name" : "sac-node-1", "cluster_name" : "sac-elasticsearch", "version" : { "number" : "9.x.x", ... }, "tagline" : "You Know, for Search" }

With the official Elasticsearch Python client, the equivalent connection looks like this:

from elasticsearch import Elasticsearch client = Elasticsearch( "https://<vm-private-ip>:9200", basic_auth=("elastic", "<generated-password>"), verify_certs=False, ssl_show_warn=False, ) print(client.info())

For production code, set verify_certs=True and provide your own root CA bundle after you replace the demo certificates (Step 13). Prefer API keys over basic authentication for programmatic access (see Step 11).

Create an index, index a document, run a simple search, then clean up:

curl -sku "elastic:${ELASTIC_PW}" -X PUT \
  https://localhost:9200/products \
  -H 'Content-Type: application/json' \
  -d '{"settings":{"number_of_shards":1,"number_of_replicas":0}}'
curl -sku "elastic:${ELASTIC_PW}" -X POST \
  https://localhost:9200/products/_doc?refresh=true \
  -H 'Content-Type: application/json' \
  -d '{"name":"sac Elasticsearch image","price":0,"tags":["azure","ubuntu"]}'
curl -sku "elastic:${ELASTIC_PW}" -X GET \
  https://localhost:9200/products/_search \
  -H 'Content-Type: application/json' \
  -d '{"query":{"match":{"name":"elasticsearch"}}}' | jq
curl -sku "elastic:${ELASTIC_PW}" -X DELETE https://localhost:9200/products

You can run the same requests in Kibana. Open Dev Tools from the left navigation, paste the requests without the curl wrapper and without the -sku flags, and run them with the play icon. The Dev Tools console uses your logged-in session, so you do not need to supply credentials per request.

Kibana also ships with sample data sets, available from the home page. They are a fast way to populate an index and explore Discover, Dashboards and Lens.

Generate a new password with the bundled tool:

NEW_PW=$(sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password \
  -u elastic -b -s)
echo "New elastic password: ${NEW_PW}"

The -b flag runs in batch mode (no prompt), and -s prints the plaintext password on stdout. The new password takes effect immediately in the security index, so no restart is needed.

Update /stage/scripts/elasticsearch-credentials.log so operators who SSH in later see the current value. To rotate the kibana_system service account password as well, run the same command with -u kibana_system, then update /etc/kibana/kibana.yml and restart Kibana.

For interactive users, create a separate account per analyst in Kibana under Stack Management > Security > Users. Use the elastic superuser only for break-glass access.

Elasticsearch 9 stores users, roles and role mappings in the security index, not in YAML files. Manage them through Kibana or the REST API.

Create a read-only analyst through the REST API:

curl -sku "elastic:${ELASTIC_PW}" -X POST \
  https://localhost:9200/_security/user/analyst \
  -H 'Content-Type: application/json' \
  -d '{ "password": "AnalystPass!234", "roles": ["viewer"], "full_name": "Read only analyst", "email": "analyst@example.com" }'

Create an API key scoped to a single index pattern for an application:

curl -sku "elastic:${ELASTIC_PW}" -X POST \
  https://localhost:9200/_security/api_key \
  -H 'Content-Type: application/json' \
  -d '{ "name": "app-logs-writer", "role_descriptors": { "logs_writer": { "cluster": ["monitor"], "indices": [{ "names": ["logs-*"], "privileges": ["write", "create_index"] }] } } }'

The response contains an encoded field. This is the base64 credential you pass in the Authorization: ApiKey <encoded> header. To rotate an API key, delete it and create a new one, because there is no in-place update.

For finer-grained roles, define a custom role with the _security/role/<name> endpoint, naming specific indices and privilege groups. Then add the role to a user or attach it to a role descriptor on an API key.

By default, /var/lib/elasticsearch lives on the OS disk. For production, move it to a dedicated Premium SSD data disk. Index growth then no longer competes with operating system writes, and you can size storage independently.

Attach a new Premium SSD disk through the portal or the Azure CLI. Then run these commands on the virtual machine:

sudo systemctl stop kibana.service
sudo systemctl stop elasticsearch.service DEVICE=/dev/disk/azure/scsi1/lun0
sudo mkfs.ext4 "${DEVICE}"
sudo mkdir -p /mnt/elasticsearch-data echo "${DEVICE} /mnt/elasticsearch-data ext4 defaults,nofail 0 2" \
  | sudo tee -a /etc/fstab
sudo mount -a sudo rsync -aHAX /var/lib/elasticsearch/ /mnt/elasticsearch-data/
sudo chown -R elasticsearch:elasticsearch /mnt/elasticsearch-data
sudo sed -i 's|^path.data: .*|path.data: /mnt/elasticsearch-data|' \
  /etc/elasticsearch/elasticsearch.yml
sudo systemctl start elasticsearch.service
sudo systemctl start kibana.service

Check the Elasticsearch log for a clean start. Then confirm the index metadata is intact by listing indices through the REST API with GET /_cat/indices?v.

The certificates shipped with this image are the upstream Elastic auto-generated self-signed certificates, written to /etc/elasticsearch/certs/ on first start. They are suitable for evaluation and for private networks where traffic cannot be intercepted. They are not production grade, because an on-node CA with no external trust issued the node certificate.

To replace them:

• Issue a node certificate and a root CA certificate from your own certificate authority, either a private or a commercial one. The node certificate must include the virtual machine's private IP and DNS name in its Subject Alternative Name extension.
• Use elasticsearch-certutil to generate a p12 bundle if you want to keep Elastic's keystore-integrated workflow. Alternatively, drop PEM files directly into /etc/elasticsearch/certs/.
• Update /etc/elasticsearch/elasticsearch.yml with the new filenames under xpack.security.http.ssl.* and xpack.security.transport.ssl.*.
• Restart Elasticsearch, then Kibana.
• Update Kibana with the new CA bundle in elasticsearch.ssl.certificateAuthorities, so Kibana can verify the Elasticsearch HTTP endpoint without verificationMode: none.

See the upstream Elastic documentation on TLS configuration for Elasticsearch for the full list of keys and the exact Subject Alternative Name requirements.

Elasticsearch refuses to start, max virtual memory areas vm.max_map_count [65530] is too low. The image ships /etc/sysctl.d/99-elasticsearch.conf, which sets vm.max_map_count=262144. If a provisioning tool overrides it, re-apply the value with sudo sysctl -w vm.max_map_count=262144. To make it permanent, edit the file.

Elasticsearch refuses to start with a heap size error. The default heap is one gigabyte, which fits evaluation-sized VMs. For production, edit /etc/elasticsearch/jvm.options.d/sac-heap.options and set -Xms and -Xmx to the same value. Use no more than fifty percent of host memory and no more than thirty-one gigabytes, which preserves compressed object pointers. Then restart Elasticsearch.

Kibana shows "Kibana server is not ready yet". This message appears on every start until Elasticsearch is healthy and the security index is reachable. Wait sixty to ninety seconds, then run sudo journalctl -u kibana.service -n 200 and look for the first non-zero exit condition.

A common cause after a cold restart is that Kibana started before Elasticsearch finished bootstrapping. The systemd drop-in that ships with the image orders the services to prevent this. If you started the services out of order manually, stop Kibana, wait for Elasticsearch, then start Kibana.

Kibana login fails with "Invalid credentials". Confirm that the kibana_system password stored in /etc/kibana/kibana.yml matches the one in the cluster. Suppose you rotated the kibana_system password with elasticsearch-reset-password -u kibana_system and forgot to update kibana.yml. Kibana then fails to authenticate on the back channel, even though the elastic user works.

curl fails with "SSL certificate problem". Elasticsearch 9 serves HTTPS by default on port 9200. Either use the -k flag to skip verification against the self-signed certificate, or add the Elastic root CA to your client's trust store. The root CA is at /etc/elasticsearch/certs/http_ca.crt on the virtual machine.

Authentication errors from a remote client. Confirm the elastic password with sudo cat /stage/scripts/elasticsearch-credentials.log. Check that the client sends either Authorization: Basic <base64> with the elastic credentials, or an API key in Authorization: ApiKey <encoded>. After repeated failures, Elasticsearch 9 rate-limits the authentication endpoint. Wait sixty seconds and retry.

Out of memory on a Standard_B2s. The default JVM heap, Kibana and the OS leave no headroom for the page cache on a virtual machine with two vCPUs and four gigabytes of memory. Resize to Standard_D4s_v3 or larger, or stop Kibana on that VM.

Use an isolated network security group. Expose TCP 22 only to your management CIDR, TCP 5601 only to the virtual network CIDR or an explicit client allow list, and TCP 9200 only to the application servers that need the REST API. Do not place a public IP on the virtual machine unless you have already replaced the demo certificates and given the elastic account a strong password.

Replace the demo certificates before any traffic other than evaluation traffic crosses the wire (Step 13). The auto-generated certificates are self-signed on the node and have no external trust chain.

Rotate the elastic password right after your first Kibana login (Step 10). Create separate user accounts for each application and each analyst, and assign least-privilege roles in Kibana Stack Management. Avoid using the elastic superuser as an application credential.

Prefer API keys over basic authentication for any long-lived programmatic access. You can scope API keys to individual indices and privilege sets, and rotate them without touching end-user passwords.

Keep patches current. Subscribe to the Elastic release notes and apply Elasticsearch minor patches within the distribution's patch windows. SecureAnyCloud publishes refreshed images at each minor release.

If your compliance framework requires it, enable the audit log by setting xpack.security.audit.enabled: true in elasticsearch.yml. The audit log writes JSON records under /var/log/elasticsearch/ and captures every privileged operation.

SecureAnyCloud provides 24/7/365 expert technical support for this image, with a guaranteed 24-hour response on all requests and a one-hour average on critical issues. Contact sales@secureanycloud.com for help with deployment, upgrades, certificate replacement or cluster migration.

Elastic distributes Elasticsearch 9 under the AGPL licence, with the Server Side Public Licence and Elastic Licence v2 as alternatives. This image ships the AGPL-licensed binaries. Kibana is distributed under the same three licences.

The image does not include any Elastic commercial subscription features, such as Machine Learning, Graph, cross-cluster replication or the commercial Security features that require an active Elastic subscription. To use those, procure a subscription directly from Elastic.

The image is a repackaged upstream distribution provided by SecureAnyCloud. Additional charges apply for build, maintenance and 24/7 support. The image is not affiliated with or endorsed by Elastic NV.

Full product documentation, versioning notes and upgrade guides are available at https://secureanycloud.com/guides/elasticsearch-9-on-ubuntu-22-04-azure.

Deploy on Azure

Launch Elasticsearch 9 on Ubuntu 22.04 on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com