Deployment guide

Apache Hive

Overview

Apache Hive is an open-source data warehouse and SQL query engine. It lets you read, write and manage large datasets stored in distributed storage using familiar SQL (HiveQL). The SecureAnyCloud image installs Hive 4.0.1 on OpenJDK 11 at /opt/hive, on top of a single-node Apache Hadoop 3.3.6 HDFS cluster, with a production-grade MariaDB metastore backend instead of embedded Derby.

What you get:

Apache Hive 4.0.1 data warehouse on OpenJDK 11 at /opt/hive
Apache Hadoop 3.3.6 single-node HDFS (NameNode + DataNode) hosting the Hive warehouse
A production MariaDB metastore, with the schema initialised by schematool -initSchema -dbType mysql
HiveServer2 JDBC over Thrift on 10000, web UI on 10002 and Metastore Thrift on 9083, all bound to loopback
An nginx reverse proxy on :80 in front of the HiveServer2 web UI, with an unauthenticated /health endpoint
A dedicated Azure data disk at /var/lib/hive holding the HDFS data, the MariaDB metastore and logs, separate from the OS disk and re-provisioned with every VM
A per-VM Hive password (HIVE_ADMIN_PASSWORD), generated on the first boot of every VM and kept in a root-only file
hadoop.service, mariadb.service, hive-metastore.service, hiveserver2.service and nginx.service as systemd units, enabled and active
24/7 SecureAnyCloud support

Prerequisites

Before you deploy this image, make sure you have:

An active Azure subscription, an SSH key pair, and a VNet and subnet in the target region.
A VM size of Standard_B4ms (4 vCPU / 16 GiB RAM) or larger. This is a good starting point, because the Hadoop, Metastore, HiveServer2 and MariaDB JVMs together are memory hungry. Scale up for larger warehouses and concurrency.
NSG inbound rules that allow 22/tcp from your management network and 80/tcp for the web UI. Front the proxy with TLS for public exposure (see Enabling HTTPS).

The JDBC port 10000 is bound to loopback by default. Expose it through an SSH tunnel or a private endpoint rather than opening it to the internet.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Apache Hive by SecureAnyCloud, and select Create.

On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80).

Review the dedicated data disk on the Disks tab, then choose Review + create and Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:apache-hive-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name hive \
  --image sac1647283583153:apache-hive-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name hive --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active hadoop.service mariadb.service hive-metastore.service hiveserver2.service nginx.service

Each service reports active. On the first boot, the stack can take a couple of minutes to come fully up while HDFS leaves safe mode and HiveServer2 warms up.

Step 5 - Retrieve your Hive password

The Hive password is generated uniquely on the first boot of your VM and written to a root-only file:

sudo cat /root/hive-credentials.txt

The HIVE_ADMIN_PASSWORD value is the password. Connect to HiveServer2 over JDBC as user hive with it, and sign in to the web UI as hive with the same password.

Step 6 - Run SQL with beeline

Hive ships beeline, the JDBC command-line client. Connect to HiveServer2 on the VM and run a CREATE / INSERT / SELECT round trip against the HDFS warehouse. Replace <HIVE_ADMIN_PASSWORD> with the value from Step 5:

sudo -u hive /opt/hive/bin/beeline \
  -u 'jdbc:hive2://127.0.0.1:10000/default' \
  -n hive -p '<HIVE_ADMIN_PASSWORD>' \
  -e "CREATE TABLE IF NOT EXISTS demo(id int, v string); INSERT INTO demo VALUES (1,'sac'); SELECT v FROM demo WHERE id=1;"

beeline connects, runs each statement, and prints the row SecureAnyCloud from the SELECT. The table is stored as a managed Hive table in the HDFS warehouse under /user/hive/warehouse, with its schema recorded in the MariaDB metastore.

Step 7 - Inspect the metastore and warehouse

List databases and tables to confirm that the metastore is serving schema, and look at the warehouse directory on HDFS:

export
JAVA_HOME=$(dirname "$(dirname "$(readlink -f "$(command -v java)")")")
sudo -u hive JAVA_HOME="$JAVA_HOME" HADOOP_HOME=/opt/hadoop \
  /opt/hadoop/bin/hdfs dfs -ls /user/hive/warehouse

The HDFS listing shows the warehouse directory and any managed table directories you have created. The schema metadata itself lives in the MariaDB metastore database on the dedicated data disk.

Step 8 - Open the HiveServer2 web UI

Browse to http://<vm-public-ip>/ and sign in as hive with the password from Step 5. The HiveServer2 web UI shows active sessions, open and historical queries, the configuration in effect, and software attributes. It is useful for watching query execution and diagnosing sessions.

Step 9 - Connect from your applications

Point JDBC clients (BI tools, ETL jobs, beeline from another host) at jdbc:hive2://<vm-public-ip>:10000/default with user hive and the per-VM password.

Port 10000 is bound to loopback by default. Reach it over an SSH tunnel (ssh -L 10000:127.0.0.1:10000 azureuser@<vm-public-ip>) or a private endpoint rather than opening it to the internet.

Enabling HTTPS

For production, terminate TLS at nginx with a real domain pointed at the VM's public IP. Install certbot and request a certificate, replacing the domain with yours:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

Backup and maintenance

The HDFS data, the MariaDB metastore and Hive logs live on the dedicated data disk at /var/lib/hive. Snapshot that disk in Azure to back up your warehouse and schema.

Keep the OS patched with sudo apt update && sudo apt upgrade. The stack restarts cleanly with sudo systemctl restart hadoop hive-metastore hiveserver2.

Support

This image is backed by 24/7 SecureAnyCloud support. Contact us by email or chat for help with Hive and Hadoop configuration, the metastore, HiveQL, query tuning, TLS and backups.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on Azure

Launch Apache Hive on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure

Overview

This image runs Apache Hive, the open-source data warehouse software that lets you read, write and manage large datasets in distributed storage using familiar SQL (HiveQL). You define tables over your data and query them with standard SQL, while Hive translates each query into a distributed execution plan over HDFS.

What you get:

Apache Hive 4.0.1 installed under /opt/hive and running as a dedicated unprivileged hive system account, on a headless OpenJDK 11 runtime.
A single-node Apache Hadoop 3.3.6 cluster running HDFS in pseudo-distributed mode (a NameNode and a DataNode), so the Hive warehouse lives on HDFS at /user/hive/warehouse.
A production-grade MariaDB metastore, not the embedded Derby database, holding table and partition metadata. The schema is initialised by schematool.
HiveServer2 with a JDBC and Thrift interface on 127.0.0.1:10000 and a read-only web UI on 127.0.0.1:10002. The Hive Metastore Thrift service runs on 127.0.0.1:9083. Every one of these ports, along with HDFS and MariaDB, binds to loopback only and is never exposed on the network.
A dedicated, independently resizable EBS data volume mounted at /var/lib/hive. The HDFS data directories, the MariaDB metastore datadir and the Hadoop and Hive logs all live there, so your warehouse and metadata survive instance replacement.
HiveServer2 client authentication through a custom per-instance credential provider, because Apache HiveServer2 accepts unauthenticated connections by default. An nginx reverse proxy on port 80 fronts the read-only web UI and protects it with HTTP Basic authentication.
An admin password generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. The loopback-only MariaDB metastore password is rotated per instance as well. The admin password is written to /root/hive-credentials.txt with mode 0600, so only the root user can read it. No shared or default credentials ship in the image.

The Hadoop, metastore and HiveServer2 JVMs take a short time to start after first boot before HiveServer2 answers.

Prerequisites

Before you deploy this image, make sure you have:

An Amazon Web Services account where you can launch EC2 instances
IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
An EC2 key pair in the target Region for SSH access to the instance
A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the web UI
The AWS CLI (version 2) installed locally, if you plan to deploy from the command line

The Hadoop, Metastore, HiveServer2 and MariaDB JVMs together are memory hungry, so choose an instance with adequate RAM. m5.large is a good starting point. Scale up for larger warehouses and higher concurrency.

Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and choose Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Apache Hive. Select the SecureAnyCloud listing and choose Select, then choose Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login).

Under Network settings, select your VPC and subnet. Create or select a security group that opens port 22 from your management network and port 80 for the web UI. Leave the root volume at the default size or larger. A dedicated data volume for the warehouse and metastore is created automatically.

Choose Launch instance. First-boot initialisation generates the admin password and rotates the metastore password. After the instance state becomes Running and the status checks pass, the JVM stack takes a short time to become ready.

Launch the Instance from the AWS CLI

The following block launches an instance from the SecureAnyCloud Apache Hive Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing and <key-name> with your EC2 key pair name. Replace <subnet-id> with your subnet ID and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-hive}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name. You can find them in the EC2 console or with aws ec2 describe-instances.

Connect to Your Instance

Connect over SSH with your key pair and the login user for your operating system variant.

OS variant SSH login user

Ubuntu 24.04 ubuntu

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Confirm the Services Are Running

The data warehouse stack runs as five systemd units: hadoop.service (HDFS), mariadb.service (the metastore database), hive-metastore.service, hiveserver2.service and nginx.service (the web UI proxy). Confirm that they are all active:

systemctl is-active hadoop.service mariadb.service hive-metastore.service hiveserver2.service nginx.service

Each service reports active. On the first boot, the stack can take a couple of minutes to come fully up while HDFS leaves safe mode and HiveServer2 warms up.

Retrieve Your Hive Password

The Hive admin password is generated uniquely on the first boot of your instance and written to a root-only file:

sudo cat /root/hive-credentials.txt

The HIVE_ADMIN_PASSWORD value is the password. Connect to HiveServer2 over JDBC as user hive with it, and sign in to the web UI as hive with the same password.

Run SQL with beeline

Hive ships beeline, the JDBC command-line client. The block below reads your per-instance password from the credentials file and runs a CREATE / INSERT / SELECT round trip against the HDFS warehouse. It then drops the demo table:

export
JAVA_HOME=$(dirname "$(dirname "$(readlink -f "$(command -v java)")")")
PW=$(sudo grep '^HIVE_ADMIN_PASSWORD=' /root/hive-credentials.txt | cut -d= -f2-) sudo -u hive JAVA_HOME="$JAVA_HOME" HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline \
  -u 'jdbc:hive2://127.0.0.1:10000/default' -n hive -p "$PW" \
  -e "CREATE TABLE IF NOT EXISTS guide_demo(id int, v string); INSERT INTO guide_demo VALUES (1,'sac'); SELECT v FROM guide_demo WHERE id=1; DROP TABLE guide_demo;"

beeline connects, runs each statement, and prints the row SecureAnyCloud from the SELECT. Managed tables are stored in the HDFS warehouse under /user/hive/warehouse, with their schema recorded in the MariaDB metastore.

To connect interactively, run the same beeline -u ... -n hive -p '<HIVE_ADMIN_PASSWORD>' command without -e and enter statements at the prompt.

Inspect the Metastore and Warehouse

List the HDFS warehouse directory to see your managed table directories. The schema metadata itself lives in the MariaDB metastore database on the dedicated data volume.

export
JAVA_HOME=$(dirname "$(dirname "$(readlink -f "$(command -v java)")")")
sudo -u hive JAVA_HOME="$JAVA_HOME" HADOOP_HOME=/opt/hadoop \
  /opt/hadoop/bin/hdfs dfs -ls /user/hive/warehouse

Open the HiveServer2 Web UI

The read-only HiveServer2 web UI is published on port 80 through nginx and protected by HTTP Basic authentication. Browse to http://<public-ip>/ and sign in as hive with the password from Step 5. The web UI shows active sessions, open and historical queries, the query execution stages, the configuration in effect, and software attributes.

An unauthenticated http://<public-ip>/health endpoint returns ok for liveness checks:

curl -s -o /dev/null -w 'health check: HTTP %{http_code}\n' http://127.0.0.1/health

Connect from Your Applications

Point JDBC clients (BI tools, ETL jobs, beeline from another host) at jdbc:hive2://<public-ip>:10000/default with user hive and the per-instance password. Port 10000 is bound to loopback by default, so reach it over an SSH tunnel or a private endpoint rather than opening it to the internet:

ssh -i /path/to/your-key.pem -L 10000:127.0.0.1:10000 ubuntu@<public-ip>

With the tunnel open, connect a local JDBC client to jdbc:hive2://127.0.0.1:10000/default.

Enabling HTTPS

For production, terminate TLS at nginx with a real domain pointed at the instance's public IP, and restrict the security group so that ports 80 and 443 are reachable only from your network. Install certbot and request a certificate, replacing the domain with yours:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

Backup and Maintenance

The HDFS data, the MariaDB metastore and the Hive logs all live on the dedicated data volume mounted at /var/lib/hive. Snapshot that EBS volume in AWS to back up your warehouse and schema. Keep the OS patched with sudo apt update && sudo apt upgrade. The stack restarts cleanly with:

sudo systemctl restart hadoop hive-metastore hiveserver2 nginx

Support

This image is backed by 24/7 SecureAnyCloud support. Contact us by email or chat for help with Hive and Hadoop configuration, the metastore, HiveQL authoring and query tuning, JDBC and ODBC connectivity, TLS and backups.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Launch Apache Hive on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS