Deployment guide

Apache Pinot

Overview

Apache Pinot is an open-source, real-time distributed OLAP datastore built for low-latency, high-concurrency analytics over streaming and batch data at scale. It serves sub-second queries through a columnar store, pluggable indexes and a distributed query engine. It also ships a web console hosted by the Pinot Controller.

This image gives you Apache Pinot 1.4.0 on OpenJDK 17 as a single-VM quickstart topology on Azure. It generates a unique console login on the first boot of every VM and is backed by 24/7 SecureAnyCloud support.

What you get:

Apache Pinot 1.4.0 running on OpenJDK 17, installed at /opt/pinot
A complete single-VM cluster: the bundled Apache ZooKeeper, the Pinot Controller (web console), the Pinot Broker and the Pinot Server, each as its own systemd service and started in dependency order
The Pinot Controller console, which has no native authentication, served behind an nginx reverse proxy with HTTP basic auth on port 80
A per-VM console password for user admin, generated at first boot and stored in a root-only file
A dedicated Azure data disk at /var/lib/pinot that holds ZooKeeper data, the Controller segment store and Server segment storage. It is separate from the OS disk and re-provisioned with every VM.
pinot-zookeeper, pinot-controller, pinot-broker, pinot-server and nginx as systemd units, enabled and active
24/7 SecureAnyCloud support

Prerequisites

You need an active Azure subscription, an SSH key pair, and a VNet and subnet in the target region.

Standard_B4ms (4 vCPU / 16 GiB RAM) is the recommended minimum. Pinot runs several JVM services on one VM and is memory-hungry. Scale up for heavier ingestion and query loads.

For the NSG, allow 22/tcp inbound from your management network and 80/tcp from the networks that need the Pinot console. For public exposure, front the proxy with TLS (see Enabling a trusted certificate).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, and search the Marketplace for Apache Pinot by SecureAnyCloud. Select Create.

On Basics, pick your subscription, resource group, region and size (Standard_B4ms or larger). Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab. Then choose Review + create and Create.

Step 2 - Deploy from the Azure CLI

Accept the Marketplace terms once, then create the VM and open port 80:

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:apache-pinot-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name pinot \
  --image sac1647283583153:apache-pinot-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name pinot --port 80 --priority 1010

Step 3 - Connect to your VM

Connect over SSH with the public IP address of the VM:

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Pinot runs several JVM services and takes one to three minutes to start fully on first boot. Once it is up, all five systemd units report active, and the unauthenticated nginx health endpoint answers 200. Check both:

systemctl is-active pinot-zookeeper.service pinot-controller.service pinot-broker.service pinot-server.service nginx.service curl -s -o /dev/null -w 'nginx health: %{http_code}\n' http://127.0.0.1/health

All services report active, and GET /health returns 200.

Step 5 - Retrieve your console login

The console username is admin. The first boot of your VM generates a unique password and writes it to a root-only file. Read it with:

sudo cat /root/pinot-credentials.txt

The username is admin. The password is the PINOT_ADMIN_PASSWORD value, and PINOT_URL is the address to open in your browser.

Step 6 - Sign in to the Pinot Controller console

Browse to http://<vm-public-ip>/. When the browser prompts for HTTP basic authentication, sign in with admin and the password from Step 5. You land on the Pinot Controller console, where you can browse tables, run Pinot SQL queries, inspect the cluster topology and manage tenants and segments.

The console has no native authentication, so nginx enforces it. A request with no credentials and a request with the wrong password are both rejected with 401. You can confirm this from the shell. The per-VM password is read from the credentials file and used inline. The no-credentials and wrong-password probes return 401, and the correct password returns 200:

echo "no credentials -> $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/)"
echo "wrong password -> $(curl -s -o /dev/null -w '%{http_code}' -u 'admin:wrong' http://127.0.0.1/)"

The first command prints 401, and the second command also prints 401.

Step 7 - Query the Pinot cluster and health APIs

Pinot exposes a REST API through the same authenticated proxy. Read the Controller health and the live cluster instances. Where the example shows <PINOT_ADMIN_PASSWORD>, substitute the admin password from Step 5:

curl -s -u "admin:<PINOT_ADMIN_PASSWORD>" http://127.0.0.1/health
curl -s -u "admin:<PINOT_ADMIN_PASSWORD>" http://127.0.0.1/instances

The health endpoint reports that the Controller is healthy, and instances lists the registered Controller, Broker and Server instances in the cluster.

Step 8 - Load the quickstart data and run a query

Pinot ships quickstart datasets that you can load to explore the console end to end. In the Query view of the console, select a table and run a Pinot SQL statement such as SELECT * FROM baseballStats LIMIT 10. Then choose Run Query to see the rows returned.

You can also create your own schema and table, then ingest batch or streaming data through the Controller APIs.

Step 9 - Confirm persistence on the data disk

All of Pinot's persistent state lives on the dedicated Azure data disk at /var/lib/pinot. Your ZooKeeper metadata, Controller segment store and Server segments therefore survive reboots and ride into any image you capture. Check the mount and its contents:

findmnt /var/lib/pinot ls /var/lib/pinot

findmnt shows /var/lib/pinot mounted from the data disk, and the directory holds the zookeeper, controller and server subdirectories.

Enabling a trusted certificate

The image serves the console over plain HTTP behind nginx basic auth. For production, terminate TLS at the nginx proxy with a real domain that points at the VM's public IP. Install certbot and request a certificate, replacing the domain with your own. Certbot configures nginx to serve HTTPS on port 443:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

After certbot completes, the Pinot console is reachable at https://your-domain.example.com/ with a browser-trusted certificate. The basic-auth prompt is unchanged.

Backup and maintenance

Your Pinot ZooKeeper metadata, Controller segment store and Server segments live on the data disk at /var/lib/pinot. Snapshot that disk to back up your cluster state and ingested data. The Pinot binaries and single-VM configuration are at /opt/pinot/current.

Keep the OS patched with sudo apt update && sudo apt upgrade. The image already applies all available Ubuntu security updates at build time.

The cluster restarts cleanly in dependency order with sudo systemctl restart pinot-zookeeper pinot-controller pinot-broker pinot-server. Allow one to three minutes for the cluster to come back up.

To change the console password, regenerate the nginx basic-auth entry with sudo htpasswd -B /etc/nginx/pinot.htpasswd admin and reload nginx.

Support

This image is backed by 24/7 SecureAnyCloud support. Contact us by email or chat for help with schemas, table configs, ingestion jobs, Pinot SQL, scaling out to a clustered deployment and TLS.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on Azure

Launch Apache Pinot on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

Apache Pinot is a real-time distributed OLAP datastore designed for low-latency, high-concurrency analytics over large event and time-series datasets. This image deploys it as a single-instance cluster on AWS, with a unique console password generated on the first boot of every instance.

What you get:

The bundled Apache ZooKeeper, the Pinot Controller (which hosts the web console), the Pinot Broker (the query endpoint) and the Pinot Server. Each runs as its own systemd service, started in dependency order on boot and restarted on failure.
Pinot installed under /opt/pinot, running as a dedicated unprivileged pinot system account on a headless OpenJDK 17 runtime
A dedicated, independently resizable EBS data volume mounted by filesystem UUID at /var/lib/pinot. It holds the ZooKeeper data, the Controller data directory and segment store, and the Server's segment storage. Your ingested data and cluster metadata survive instance replacement, and the volume is captured into the image layout.
The Pinot web console served on port 80 by an nginx reverse proxy with HTTP basic authentication

The Pinot Controller console and the Broker query endpoint have no native authentication. This image therefore keeps every Pinot component and ZooKeeper off the public network. The security group opens only port 22 (SSH) and port 80 (the console), and an authenticated nginx reverse proxy fronts the console.

The console password is generated on the first boot of every deployed instance, so two instances launched from the same AMI never share a password. It is written to /root/pinot-credentials.txt with mode 0600, so only the root user can read it.

Prerequisites

Before you deploy this image, you need:

An Amazon Web Services account where you can launch EC2 instances
IAM permissions to launch instances, create security groups and subscribe to AWS Marketplace products
An EC2 key pair in the target Region for SSH access to the instance
A VPC and subnet in the target Region, with a security group that allows inbound port 22 from your management network and port 80 for the web console
The AWS CLI (version 2) installed locally, if you plan to deploy from the command line

Launch the instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and choose Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Apache Pinot. Select the SecureAnyCloud listing and choose Select, then choose Continue on the subscription summary.

Pick an instance type of m5.large or larger. Pinot runs several JVMs, so size the instance for your data volume and query concurrency. Choose your EC2 key pair under Key pair (login).

Under Network settings, select your VPC and subnet. Create or select a security group that opens port 22 from your management network and port 80 for the web console. Leave the root volume at the default size or larger, and keep the attached data volume that holds the Pinot cluster state.

Choose Launch instance. First-boot initialisation runs immediately. After the instance state becomes Running and the status checks pass, the Pinot cluster takes about a minute to come up in dependency order.

Launch the instance from the AWS CLI

The following command launches an instance from the SecureAnyCloud Apache Pinot Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing. Replace <key-name> with your EC2 key pair name and <subnet-id> with your subnet ID. Replace <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-pinot}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name. You can read it from the EC2 console or with aws ec2 describe-instances.

Connect to your instance

Connect over SSH with your key pair and the login user for your operating system variant.

OS variant SSH login user

Ubuntu 24.04 ubuntu

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Retrieve the console password

The Pinot console password is unique to your instance and was generated on first boot. Read it as root:

sudo cat /root/pinot-credentials.txt

The file lists the console URL, the administrator username admin and the generated password. It is mode 0600 and owned by root, so only the root user can read it. Every command below that needs the password reads it straight from this file into a shell variable, so you never have to copy or paste it.

Confirm the services are running

The Pinot cluster and nginx run as systemd services. Confirm that they are all active:

systemctl is-active pinot-zookeeper.service pinot-controller.service pinot-broker.service pinot-server.service nginx.service

nginx serves an unauthenticated health endpoint on port 80, which suits load-balancer health checks. Confirm that it answers:

curl -fsS http://127.0.0.1/health

Verify the authenticated console

The console itself requires the per-instance password. First, confirm that an unauthenticated request is rejected:

curl -s -o /dev/null -w 'unauthenticated console: HTTP %{http_code}\n' http://127.0.0.1/

The output shows HTTP 401. Now read the password from the credentials file into a shell variable, which never prints it. Confirm that the authenticated console returns HTTP 200:

PW=$(sudo awk -F= '/^PINOT_ADMIN_PASSWORD=/{print $2}' /root/pinot-credentials.txt)
curl -sL -o /dev/null -w 'authenticated console: HTTP %{http_code}\n' -u "admin:$PW" http://127.0.0.1/

Open the web console

Browse to http://<public-ip>/ and sign in as user admin with the password from Step 4. The console opens on the Cluster Manager, which shows the live Controller, Broker and Server instances and the tenant and table counts.

The Query Console gives you an interactive SQL editor. Pick a table on the left and run an aggregation over your data. Ranked results come back in milliseconds, with per-query response statistics above the result grid.

Select a table name to open its detail view. It shows the table configuration, the schema (dimension, metric and date-time columns) and the segments loaded on each Server instance.

Inspect the cluster and tables from the command line

The authenticated proxy gives you access to the Pinot Controller REST API. List the tables in the cluster. The command reads the password from the credentials file, so it is never printed:

PW=$(sudo awk -F= '/^PINOT_ADMIN_PASSWORD=/{print $2}' /root/pinot-credentials.txt)
curl -s -u "admin:$PW" http://127.0.0.1/tables

Confirm that the Controller reports itself healthy through the same authenticated proxy:

PW=$(sudo awk -F= '/^PINOT_ADMIN_PASSWORD=/{print $2}' /root/pinot-credentials.txt)
curl -s -u "admin:$PW" http://127.0.0.1/health

Create a table and ingest data

Pinot ingests data into a table that is described by a schema. The following worked example creates a small offline table, batch-ingests five rows and queries them. Run it on the instance. It reads the console password from the credentials file.

The example uses the Controller REST API for schema and table creation, the bundled pinot-admin.sh batch ingestion job for loading data, and the Broker query endpoint for the query.

PW=$(sudo awk -F= '/^PINOT_ADMIN_PASSWORD=/{print $2}' /root/pinot-credentials.txt)
sudo install -d -o pinot -g pinot /var/lib/pinot/example/rawdata /var/lib/pinot/example/segments
# 1. Define the schema (two dimensions, one metric, one time column).
cat <<'JSON' | sudo -u pinot tee /var/lib/pinot/example/schema.json >/dev/null {"schemaName":"events", "dimensionFieldSpecs":[{"name":"id","dataType":"INT"},{"name":"city","dataType":"STRING"}], "metricFieldSpecs":[{"name":"amount","dataType":"LONG"}], "dateTimeFieldSpecs":[{"name":"ts","dataType":"LONG","format":"1:MILLISECONDS:EPOCH","granularity":"1:MILLISECONDS"}]} JSON
# 2. Define the offline table.
cat <<'JSON' | sudo -u pinot tee /var/lib/pinot/example/table.json >/dev/null {"tableName":"events","tableType":"OFFLINE", "segmentsConfig":{"timeColumnName":"ts","timeType":"MILLISECONDS","replication":"1"}, "tenants":{},"tableIndexConfig":{"loadMode":"MMAP"},"metadata":{}} JSON curl -s -u "admin:$PW" -X POST -H 'Content-Type: application/json' \
  --data-binary @/var/lib/pinot/example/schema.json http://<public-ip>/schemas curl -s -u "admin:$PW" -X POST -H 'Content-Type: application/json' \
  --data-binary @/var/lib/pinot/example/table.json http://<public-ip>/tables
# 3. Provide the data and a batch ingestion job spec.
cat <<'CSV' | sudo -u pinot tee /var/lib/pinot/example/rawdata/data.csv >/dev/null id,city,amount,ts 1,london,10,1700000000000 2,paris,20,1700000001000 3,london,30,1700000002000 CSV
cat <<'YAML' | sudo -u pinot tee /var/lib/pinot/example/job.yaml >/dev/null executionFrameworkSpec: {name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner, segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner} jobType: SegmentCreationAndTarPush inputDirURI: /var/lib/pinot/example/rawdata includeFileNamePattern: 'glob:**/*.csv' outputDirURI: /var/lib/pinot/example/segments overwriteOutput: true pinotFSSpecs: [{scheme: file, className: org.apache.pinot.spi.filesystem.LocalPinotFS}] recordReaderSpec: {dataFormat: csv, className: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader, configClassName: org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig} tableSpec: {tableName: events} pinotClusterSpecs: [{controllerURI: 'http://localhost:9000'}] pushJobSpec: {pushParallelism: 1, pushAttempts: 2, pushRetryIntervalMillis: 1000} YAML
# 4. Run the batch ingestion job.
sudo -u pinot env JAVA_OPTS='-Xms256M -Xmx1G' /opt/pinot/current/bin/pinot-admin.sh \
  LaunchDataIngestionJob -jobSpecFile /var/lib/pinot/example/job.yaml
# 5. Query the data through the Broker (a few seconds after ingestion).
curl -s -u "admin:$PW" -H 'Content-Type: application/json' \
  -XPOST http://<public-ip>/sql -d '{"sql":"SELECT city, SUM(amount) FROM events GROUP BY city"}'

You can also use the console's Query Console to run SQL interactively. To drive schema and table creation from your own applications, use the same Controller REST API.

Data persistence and the dedicated data volume

All of Pinot's persistent state lives on a dedicated EBS data volume mounted at /var/lib/pinot. This covers the bundled ZooKeeper data, the Controller data directory and segment store, and the Server's segment storage.

The volume is mounted by filesystem UUID via /etc/fstab, so it re-attaches reliably across reboots and instance replacement. It is captured into the AMI, so every launched instance gets the same layout. Because the tier is a separate volume, you can resize it independently for capacity and IOPS, and snapshot it on its own schedule. Check the mount with:

findmnt /var/lib/pinot

Security model

The Pinot Controller console and the Broker query endpoint ship with no native authentication. This image is hardened so that nothing Pinot-related is reachable off the instance except the authenticated console:

Network isolation. The security group opens only port 22 (SSH) and port 80 (the console). The ZooKeeper port (2181), the Controller (9000), the Broker (8099) and the Server (8098) are never opened to the network, so they are unreachable from off the instance.
Authenticated reverse proxy. nginx is the single public application entry point on port 80. It enforces HTTP basic authentication in front of the Controller console. Unauthenticated requests, wrong passwords and the password used at build time are all rejected with HTTP 401.
Per-instance credentials. The console password is generated uniquely on each instance's first boot and stored in a root-only file. No shared or default console credential ships in the image.

For production, restrict port 80 in your security group to your own CIDR ranges. Terminate TLS at an Application Load Balancer or a reverse proxy in front of the instance, so console traffic is encrypted in transit. Deploy the instance in a private subnet where possible.

Maintenance

Service control: manage the cluster with sudo systemctl restart pinot-controller.service (and the pinot-zookeeper, pinot-broker, pinot-server and nginx units). The services start in dependency order on boot.
Logs: inspect a component with sudo journalctl -u pinot-controller.service -n 100 --no-pager. Pinot's own logs are under /var/lib/pinot/logs.
OS updates: apply security updates with sudo apt-get update && sudo apt-get upgrade, and reboot during a maintenance window.

Support

SecureAnyCloud provides 24/7 technical support for this image by email and live chat. Contact sales@secureanycloud.com for deployment and instance sizing guidance, schema and table design, ingestion configuration, query tuning, and console authentication and TLS setup. You can also contact it for any other issue, including refund requests.

Deploy on AWS

Launch Apache Pinot on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS