ELK Stack

• Go to the AWS Marketplace and search for "ELK Stack SecureAnyCloud"
• Choose Continue to Subscribe, accept the terms, then choose Continue to Configuration
• Select your preferred Region and Software Version
• Choose Continue to Launch
• Choose Launch through EC2 for full control over instance configuration
• Select your instance type (t3.large recommended)
• Configure storage: 20 GB gp3 minimum, and 50 GB or more recommended for production log storage
• Configure your Security Group with the inbound rules listed below
• Select your EC2 key pair and launch the instance

Add these inbound rules to the security group:

• Port 22, TCP, from your IP: SSH access
• Port 9200, TCP, from your IP: Elasticsearch API calls over HTTP
• Port 5601, TCP, from your IP: Kibana dashboard

Important: Restrict ports 9200 and 5601 to trusted IP addresses only. These services provide access to your data and should not be exposed to the public internet.

Once your instance is running and has passed both status checks (2/2), connect over SSH:

ssh -i your-key.pem ec2-user@<public-ip-address>

Replace your-key.pem with the path to your EC2 key pair and <public-ip-address> with your instance's public IP.

Important: Wait for the EC2 instance to reach 2/2 successful status checks before you connect. If you connect too early, you may see errors such as:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

To switch to the root user:

sudo su -

The ELK Stack is preconfigured with security enabled. Randomly generated credentials are stored in log files on the instance. Read them as follows.

Elasticsearch credentials:

cat /stage/scripts/elastic_password.log

Kibana credentials:

cat /stage/scripts/kibana_password.log

Important: Make a note of these passwords. You will need them to access Elasticsearch and Kibana.

Check that Elasticsearch is running and responding to API requests:

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic http://localhost:9200

Enter the password from the elastic_password.log file when prompted.

Expected output:

{ "name" : "ip-172-31-89-117.ec2.internal", "cluster_name" : "elasticsearch", "cluster_uuid" : "wdyXEmS-TEearSUr0MEhPA", "version" : { "number" : "8.6.1", "build_flavor" : "default", "build_type" : "rpm", "lucene_version" : "9.4.2", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }

Open your web browser and go to:

http://<public-ip-address>:5601

You will see the Kibana login screen. Sign in with:

• Username: elastic
• Password: The value from /stage/scripts/elastic_password.log

After you sign in, the Kibana home page opens. From there you can add integrations, create dashboards and explore your data.

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for ELK Stack. Select the SecureAnyCloud listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. ELK runs three JVM workloads plus an nginx proxy on the same host, so it benefits from at least two vCPUs and 8 GiB of memory. For production workloads with sustained ingest, m5.xlarge or m5.2xlarge gives the JVMs room to raise their heaps.

Choose your EC2 key pair under Key pair (login). Under Network settings, select your VPC and subnet, then create or select a security group. It must allow inbound port 22 from your management network, inbound port 80 from the network you will reach Kibana from, and inbound port 5044 from the network where your Beats agents run. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialization takes approximately two to three minutes after the instance state becomes Running and the status checks pass. The firstboot service waits for Elasticsearch to be ready before it rotates the credentials.

The following command launches an instance from the SecureAnyCloud ELK Stack Marketplace AMI into an existing subnet and security group.

Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22, 80 and 5044 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":30,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=elk-stack-01}]'

On success the command prints a JSON document. Note the instance ID, then retrieve its public address once it is running with aws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].PublicIpAddress" --output text.

Connect over SSH with the key pair you selected and the public IP address from step 2. The SSH login user depends on the operating system of the AMI variant you launched. For ELK Stack 8 on Ubuntu 24.04, the login user is ubuntu.

The first-boot service runs before Kibana is ready, so the credentials file is always in place by the time the web interface accepts a login. Read it as root:

sudo cat /root/elk-credentials.txt

The plain text file contains the Kibana URL, the Kibana web user (SecureAnyCloud) and the Kibana web password. It also contains the Elasticsearch URL, the Elasticsearch superuser (elastic), the Elasticsearch password and the Logstash Beats endpoint.

Copy these values somewhere secure, such as a password manager or an encrypted vault. Do not commit them to source control.

From the same SSH session, confirm that Elasticsearch is healthy. The cluster health endpoint needs the elastic credentials:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" http://127.0.0.1:9200/_cluster/health | jq

A JSON document with "status": "green" or "status": "yellow" confirms the cluster is up.

A single-node Elasticsearch cluster reports yellow when an index has unallocated replica shards. This is normal, because there is no second node for the replicas to live on.

Open a web browser and go to http://<public-ip>/. nginx prompts you for HTTP basic authentication.

Enter the basic-auth user SecureAnyCloud and the password from the ELK_WEB_PASSWORD line of /root/elk-credentials.txt. Your browser then loads Kibana, which shows its own sign-in form. Sign in to Kibana as the elastic user with the password from the ELASTIC_PASSWORD line of the same file.

On your first successful sign-in, Kibana shows a welcome screen. The navigation hamburger appears at the top left of every page.

The Discover application is the entry point for searching and exploring documents in Elasticsearch. Open the navigation hamburger and select Analytics, then Discover.

A freshly launched instance has no indices yet, so Discover invites you to create a data view. Once data flows through Logstash into Elasticsearch (see step 6), your indices are listed here. A time histogram appears at the top, fields run down the left, and your documents fill the center.

Discover supports the Kibana Query Language (KQL) and the older Lucene syntax for filtering documents.

The image runs Logstash with a pipeline pre-wired to accept Beats traffic on TCP 5044 and forward it to the local Elasticsearch. The simplest way to start sending data is to install Filebeat on a separate host and point it at the Logstash endpoint on your instance.

On the host you want to collect logs from, install Filebeat and point its Logstash output at your instance:

sudo apt-get update && sudo apt-get install -y filebeat
sudo tee /etc/filebeat/filebeat.yml >/dev/null <<'YAML' filebeat.inputs: - type: filestream id: syslog paths: - /var/log/syslog - /var/log/auth.log output.logstash: hosts: ["<your-public-ip>:5044"] YAML
sudo systemctl enable --now filebeat

Replace <your-public-ip> with your ELK Stack instance's public address.

Filebeat reads the log files and ships events to Logstash. Logstash forwards them to Elasticsearch as sac-YYYY.MM.dd indices, and they appear in Kibana Discover within seconds.

You can also send a simple test event from the ELK instance itself with curl:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" \
  -H 'Content-Type: application/json' \
  -X POST 'http://127.0.0.1:9200/sac-test/_doc?refresh=true' \
  -d '{"message":"hello from sac","ts":"2026-05-23T12:00:00Z"}'

This writes a single document to the sac-test index, which then appears in Discover.

ELK indices accumulate over time, so a production deployment defines an index lifecycle management (ILM) policy to age and retire indices automatically. Open the navigation hamburger and select Management, then Index Lifecycle Policies.

Elasticsearch ships several built-in ILM policies (logs, metrics, synthetics), which most Beats data streams use by default. Select Create policy to define your own. A typical policy keeps an index in the hot phase for 7 days, rolls over to warm storage for 30 days, then deletes it.

Apply the policy to a data stream through its index template. The Elastic documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html covers the full workflow.

To see the health of Elasticsearch, Logstash and Kibana on one screen, open the navigation hamburger and select Management, then Stack Monitoring. If monitoring is not yet enabled, Kibana offers to turn it on by enabling xpack.monitoring.collection.enabled on the cluster. Once it is enabled, the Cluster overview card shows the health of each component.

The cluster overview shows Elasticsearch (number of nodes, indices, documents, JVM heap and disk usage), Kibana (request rate, response time and memory usage), and Logstash once metrics are flowing. Click any card to drill into a per-component view.

The Logstash detail page shows throughput in events per second, latency per filter stage, and the input and output plugins in the running configuration. The shipped configuration exposes one pipeline named main with a Beats input on port 5044, a UDP syslog input on 5140, a grok filter for the syslog branch, and an Elasticsearch output.

You can also query the Logstash node API directly from the instance:

curl -fsS http://127.0.0.1:9600/_node/stats | jq .pipelines.main.events

The response shows the number of events the pipeline has received, filtered and emitted since it started.

Each component runs as a dedicated systemd unit. Use systemctl to inspect and control them:

sudo systemctl status elasticsearch.service
sudo systemctl status logstash.service
sudo systemctl status kibana.service
sudo systemctl status nginx.service

Each unit writes its logs to journalctl:

sudo journalctl -u elasticsearch.service -n 100
sudo journalctl -u logstash.service -n 100
sudo journalctl -u kibana.service -n 100

Configuration files live under each package's directory:

• Elasticsearch: /etc/elasticsearch/elasticsearch.yml and /etc/elasticsearch/jvm.options.d/heap.options
• Logstash: /etc/logstash/conf.d/sac.conf and /etc/logstash/jvm.options
• Kibana: /etc/kibana/kibana.yml
• nginx: /etc/nginx/sites-available/kibana and /etc/nginx/kibana.htpasswd

To raise the Elasticsearch heap, edit /etc/elasticsearch/jvm.options.d/heap.options and replace the -Xms and -Xmx lines. Then restart elasticsearch.service.

As a rule of thumb, set the heap to half the instance memory but never above 30 GiB. The same applies to Logstash through /etc/logstash/jvm.options.

Kibana defaults to signing you in as elastic, which is the cluster superuser. For day-to-day use, create a less privileged user. From a shell on the instance, run:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" \
  -H 'Content-Type: application/json' \
  -X POST 'http://127.0.0.1:9200/_security/user/analyst' \
  -d '{"password":"<choose-a-strong-password>","roles":["kibana_admin","viewer"],"full_name":"Analyst"}'

You can then sign in to Kibana as analyst. To rotate the elastic superuser password later, run the following and update any tools that authenticate as elastic:

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password --user elastic --interactive

nginx already serves Kibana on port 80 with HTTP basic authentication. For any deployment beyond a trusted management network, terminate TLS at nginx so that the basic-auth credential and Kibana session cookies cannot be intercepted.

Install certbot and request a Let's Encrypt certificate for your hostname:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d kibana.your-domain.example \
  --non-interactive --agree-tos -m you@your-domain.example --redirect

certbot edits the nginx server block to listen on 443 with the certificate and to redirect plain HTTP to HTTPS. Reload nginx with sudo systemctl reload nginx. Then restrict the instance security group so that port 80 is reachable only from the proxy network.

Elasticsearch data lives on its own EBS volume at /var/lib/elasticsearch, so the simplest backup strategy is an EBS snapshot of that volume. Stop the elasticsearch service first to flush in-memory state. For live snapshots, use the official snapshot and restore API instead:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" \
  -H 'Content-Type: application/json' \
  -X PUT 'http://127.0.0.1:9200/_snapshot/sac-s3' \
  -d '{"type":"s3","settings":{"bucket":"<your-snapshot-bucket>","region":"<your-region>"}}'

The cluster needs the repository-s3 plugin and an IAM role with s3:PutObject on the bucket. The Elastic documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html covers the full workflow.

Ubuntu's unattended-upgrades is enabled by default, so kernel and package security patches apply automatically. To update ELK itself, install the new version with apt:

sudo apt-get update
sudo apt-get install -y elasticsearch logstash kibana
sudo systemctl restart elasticsearch.service logstash.service kibana.service

The Elastic APT repository pins to the 8.x major version. For a major version upgrade, switch the repository source and follow the Elastic upgrade documentation.

For larger deployments, decouple ELK from the single-instance pattern:

• Move Elasticsearch to a multi-node cluster by launching additional EC2 instances, switching discovery.type from single-node to the seed-hosts model, and configuring node.roles so the cluster has dedicated master, data and ingest nodes
• Alternatively, use Amazon OpenSearch Service for a fully managed cluster behind the same Kibana
• Run Logstash on its own fleet of instances with an Application Load Balancer in front, so Beats agents send to a load-balanced endpoint and Logstash can scale horizontally
• Put Kibana behind an Application Load Balancer with sticky sessions for high availability of the web interface
• Use Amazon S3 as a snapshot repository for point-in-time recovery of the Elasticsearch indices

Each of these is documented in the official Elastic documentation at https://www.elastic.co/guide/en/elastic-stack/current/index.html.