Deployment guide

Hadoop

Overview

Apache Hadoop is an open-source framework for distributed storage (HDFS) and distributed processing (YARN and MapReduce) of large data sets. This SecureAnyCloud image runs a ready-to-use, single-node, pseudo-distributed Hadoop 3.4.1 cluster on Azure.

The HDFS NameNode and DataNode, and the YARN ResourceManager and NodeManager, each run as a hardened systemd service. The cluster starts automatically on first boot.

What you get:

Apache Hadoop 3.4.1 (HDFS + YARN + MapReduce) on OpenJDK 11, installed at /opt/hadoop
Four systemd services: hadoop-namenode, hadoop-datanode, hadoop-resourcemanager, hadoop-nodemanager, all running as a dedicated hadoop user
The NameNode web UI on :9870 and the ResourceManager web UI on :8088
A dedicated Azure data disk at /var/lib/hadoop holding the HDFS NameNode and DataNode directories. It is separate from the OS disk and is re-provisioned with every VM
A per-VM /root/hadoop-info.txt, written at first boot, with the UI URLs
24/7 SecureAnyCloud support

Prerequisites

You need an active Azure subscription, an SSH key pair, and a virtual network and subnet in the target region.

Standard_B2ms (2 vCPU, 8 GiB RAM) is a sensible minimum for Hadoop and YARN. Use a larger size for bigger workloads.

In the network security group (NSG), allow inbound 22/tcp from your management network. Also allow 9870/tcp and 8088/tcp from the networks that need the web dashboards.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, select Create a resource, and search the Marketplace for Apache Hadoop by SecureAnyCloud. Select Create.

On the Basics tab, choose your subscription, resource group, region and size (Standard_B2ms or larger). Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22).

On the Disks tab, review the dedicated data disk. Then select Review + create, and then Create.

Add rules for ports 9870 and 8088 afterwards. Step 2 shows the CLI commands.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:hadoop-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name hadoop \
  --image sac1647283583153:hadoop-ubuntu-24-04:default:latest \
  --size Standard_B2ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name hadoop --port 9870 --priority 1010
az vm open-port --resource-group <your-rg> --name hadoop --port 8088 --priority 1020

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm that the cluster is running

systemctl is-active hadoop-namenode hadoop-datanode hadoop-resourcemanager hadoop-nodemanager curl -s -o /dev/null -w 'NameNode UI: %{http_code}\n' http://127.0.0.1:9870/ curl -s -o /dev/null -w 'ResourceManager UI: %{http_code}\n' http://127.0.0.1:8088/

All four services report active, and both web UIs return 200.

Step 5 - Retrieve your cluster details

First boot writes the UI URLs for this VM to a file. Read it with:

sudo cat /root/hadoop-info.txt

When the NSG ports are open, browse to the NameNode UI at http://<vm-public-ip>:9870/ and the ResourceManager UI at http://<vm-public-ip>:8088/.

Step 6 - Use HDFS

Run HDFS commands as the hadoop user. The commands below create a directory, store a file and read it back:

sudo -u hadoop /opt/hadoop/bin/hdfs dfs -mkdir -p /demo echo 'hello hadoop' | sudo -u hadoop /opt/hadoop/bin/hdfs dfs -put -f - /demo/hello.txt
sudo -u hadoop /opt/hadoop/bin/hdfs dfs -ls /demo
sudo -u hadoop /opt/hadoop/bin/hdfs dfs -cat /demo/hello.txt

Step 7 - Run a MapReduce job

Submit the bundled example MapReduce job, which estimates Pi, to YARN. This confirms that the processing layer works:

sudo -u hadoop /opt/hadoop/bin/hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 2 5

The ResourceManager accepts the job, and you can follow it at :8088. When the job completes, it prints an estimated value of Pi.

Step 8 - Check the HDFS report

Run the report command:

sudo -u hadoop /opt/hadoop/bin/hdfs dfsadmin -report | head -20

The report shows the live DataNode, the configured capacity and the usage.

Backup and maintenance

All HDFS data lives on the dedicated data disk at /var/lib/hadoop. To back up your filesystem, snapshot that disk in Azure.

Keep the OS patched with sudo apt update && sudo apt upgrade.

Restart the daemons with sudo systemctl restart hadoop-namenode hadoop-datanode hadoop-resourcemanager hadoop-nodemanager.

Do not reformat the NameNode unless you intend to erase HDFS.

Support

24/7 SecureAnyCloud support is included with this image. Contact us by email or chat for help with HDFS, YARN, MapReduce, scaling out to multiple nodes, and tuning.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on Azure

Launch Apache Hadoop on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure