SecureAnyCloud
Deployment guide

Hadoop Big Data Stack

Overview

The Hadoop Big Data Stack AMI by SecureAnyCloud provides a fully preconfigured Apache Hadoop ecosystem, ready to run on Amazon EC2. This single-node deployment includes Hadoop, Hive, HBase, Pig, Spark and Zookeeper. It gives you a complete big data processing and analytics platform that is operational within minutes of launch.

The AMI is ideal for development, testing, prototyping and learning environments where you need the full Hadoop stack without configuring each component from scratch.

What you get:

  • Hadoop, Hive, HBase, Pig, Spark and Zookeeper, preconfigured for single-node operation
  • All components installed under a dedicated /apps mount point, with their own volumes
  • Hadoop services managed through convenient start and stop scripts

This guide walks you through connecting to the instance, initializing the Hadoop services, accessing the web management interfaces, and working with each component in the stack.

Visit secureanycloud.com to explore the full catalogue of preconfigured AMIs available on AWS Marketplace.

Prerequisites

Before you launch the Hadoop Big Data Stack AMI, make sure you have the following in place.

  • AWS account: You need an active AWS account with permissions to launch EC2 instances and manage security groups.
  • EC2 key pair: Create or select an existing EC2 key pair in the Region where you plan to launch the instance. SSH authentication requires this key pair.
  • Security group: Your security group must allow inbound traffic on the ports listed below.

The required inbound ports are:

  • TCP 22: SSH connectivity for remote administration
  • TCP 8088: Hadoop ResourceManager UI
  • TCP 8042: Hadoop NodeManager UI

For additional Hadoop ecosystem web interfaces, you may also want to open these ports:

  • TCP 9870: HDFS NameNode UI
  • TCP 16010: HBase Master UI
  • TCP 8080: Spark Master UI (if configured)
  • TCP 2181: Zookeeper client port

Restrict all ports to your own IP address or a trusted CIDR range. Never expose these management interfaces to the public internet without additional authentication.

Minimum system requirements: 1 vCPU, 1 GB RAM and 20 GB of disk space.

Although the minimum requirements are low, big data workloads benefit significantly from more resources. For anything beyond basic testing, consider an m5.xlarge or larger instance type with at least 16 GB of RAM.

Step-by-step setup

Step 1: Launch the Instance

  • Open the AWS Marketplace listing for Hadoop Big Data Stack by SecureAnyCloud.
  • Choose Continue to Subscribe, then Continue to Configuration.
  • Select your preferred AWS Region and instance type.
  • On the launch page, choose your VPC and subnet, and assign the security group that you prepared above.
  • Select your EC2 key pair and launch the instance.

Step 2: Wait for Status Checks

Wait until the EC2 instance reaches 2/2 status checks passed before you connect.

The instance runs an initial boot update script that applies the latest operating system patches. The first boot may therefore take a few minutes longer than usual.

If you connect before both status checks have passed, you may see errors such as:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic). ec2-user@your-instance-ip's password:

These errors are expected during early boot. Wait for the status checks to complete, then try again.

Step 3: Connect via SSH

Connect to the instance using your private key:

ssh -i /path/to/your-key.pem ec2-user@<PUBLIC_IP>

Replace <PUBLIC_IP> with the public IP address or public DNS name shown in the EC2 console.

Step 4: Switch to the Root User

Once you are connected as ec2-user, switch to the root user:

sudo su -

Step 5: Configure Hadoop SSH Keys

Before you start Hadoop services for the first time, run the SSH key setup script as the root user. Hadoop requires SSH key-based communication between its components (NameNode, DataNode, TaskTracker, JobTracker), even on a single-node deployment:

./stage/scripts/setup_ssh_hadoop_user.sh

The script configures the hadoop OS user with the SSH keys needed for passwordless authentication between Hadoop components.

Step 6: Start All Hadoop Services

Switch to the hadoop user and start all services with the provided convenience script:

sudo su - hadoop cd $HOME . ./start-all.sh

The script starts HDFS (NameNode, DataNode), YARN (ResourceManager, NodeManager) and all supporting services.

Step 7: Verify the ResourceManager UI

Open a web browser and go to:

http://<PUBLIC_IP>:8088

The Hadoop ResourceManager web interface shows the YARN cluster status, including node information, application statistics and scheduler metrics.

Step 8: Verify the NodeManager UI

Go to the NodeManager interface:

http://<PUBLIC_IP>:8042

This page displays information about the local node, including resource usage and container logs.

Server Components

The Hadoop Big Data Stack AMI includes the following preconfigured components. All big data components are installed under the /apps directory on a dedicated volume.

  • Java 1.8 (software home /apps/java): the Java Runtime Environment required by all Hadoop ecosystem components
  • Hadoop 3.3.4 (software home /apps/hadoop): the distributed storage (HDFS) and processing (MapReduce/YARN) framework
  • Apache Hive 3.1.3 (software home /apps/apache-hive): data warehouse infrastructure for querying data stored in HDFS
  • Apache HBase 2.4.15 (software home /apps/apache-hbase): a distributed, scalable NoSQL database built on top of HDFS
  • Apache Pig 0.17 (software home /apps/apache-pig): a high-level platform for creating MapReduce programs using Pig Latin
  • Apache Spark 3.3.2 (software home /apps/apache-spark): a unified analytics engine for large-scale data processing
  • Apache Zookeeper 3.7.1 (software home /apps/apache-zookeeper): a centralized service for configuration, synchronization and naming

Each component keeps its configuration files in a conf subdirectory within its installation directory.

Versions are subject to change on initial boot if the update script finds newer packages.

Filesystem Layout

The AMI uses dedicated mount points to separate the operating system from the big data components:

  • /: the root filesystem, containing the operating system
  • /boot: the operating system kernel files
  • /apps: the installation directory for big data components, on a dedicated volume

Key directories and their purposes:

  • /apps/java: the Java installation directory (JAVA_HOME)
  • /apps/hadoop: the Hadoop installation, including HDFS and YARN binaries
  • /apps/hadoop/etc/hadoop: the Hadoop configuration files (core-site.xml, hdfs-site.xml, yarn-site.xml)
  • /apps/apache-hive: the Hive installation directory
  • /apps/apache-hive/conf: the Hive configuration files
  • /apps/apache-hbase: the HBase installation directory
  • /apps/apache-hbase/conf: the HBase configuration files
  • /apps/apache-pig: the Pig installation directory
  • /apps/apache-pig/conf: the Pig configuration files
  • /apps/apache-spark: the Spark installation directory
  • /apps/apache-spark/conf: the Spark configuration files
  • /apps/apache-zookeeper: the Zookeeper installation directory
  • /apps/apache-zookeeper/conf: the Zookeeper configuration files
  • /home/hadoop: the home directory for the hadoop OS user, which contains the start and stop scripts
  • /stage/scripts: SecureAnyCloud provisioning scripts and log files

The /apps volume is mounted on a separate EBS volume. You can therefore resize the storage for your big data components and data independently, without affecting the root filesystem.

Managing Services

You manage all Hadoop ecosystem services through the hadoop OS user. The AMI provides convenient start and stop scripts in the hadoop user's home directory.

Starting All Hadoop Services

sudo su - hadoop cd $HOME . ./start-all.sh

This starts HDFS, YARN and all related daemons for single-node operation.

Stopping All Hadoop Services

sudo su - hadoop cd $HOME . ./stop-all.sh

Checking Individual Service Status

Verify the running Java processes, which include all Hadoop daemons, with:

sudo su - hadoop jps

The jps command should list processes such as NameNode, DataNode, ResourceManager, NodeManager and any other active Hadoop daemons.

Verifying the Java Version

java -version

Using System Components

Apache Hive

Apache Hive provides a SQL-like interface for querying data stored in HDFS. Type the following command manually (do not copy and paste) to verify the installed version:

hive --version

To start the Hive interactive shell:

sudo su - hadoop hive

Configuration files are located at /apps/apache-hive/conf. To customize Hive behavior, edit hive-site.xml in that directory.

Apache HBase

Apache HBase is a distributed NoSQL database. Type the following command manually (do not copy and paste) to launch the HBase shell:

hbase shell

From the HBase shell, you can create tables, insert data and run scans. Configuration files are located at /apps/apache-hbase/conf.

Apache Pig

Apache Pig provides a high-level language (Pig Latin) for expressing data analysis programs. Type the following command manually (do not copy and paste) to verify the installed version:

pig -version

To start the Pig interactive shell (Grunt):

sudo su - hadoop pig

Configuration files are located at /apps/apache-pig/conf.

Apache Spark

Apache Spark is a unified analytics engine for large-scale data processing. Type the following command manually (do not copy and paste) to verify the installed version:

spark-submit --version

To launch the Spark interactive shell:

sudo su - hadoop spark-shell

For PySpark (Python interface):

pyspark

Configuration files are located at /apps/apache-spark/conf.

Apache Zookeeper

Apache Zookeeper provides distributed coordination services. Type the following command manually (do not copy and paste) to verify the installed version:

zkServer.sh version

To check the Zookeeper service status:

zkServer.sh status

Configuration files are located at /apps/apache-zookeeper/conf.

Scripts and Log Files

The AMI includes several scripts and log files created by SecureAnyCloud to streamline provisioning and service management:

  • initial_boot_update.sh (in /stage/scripts): updates the operating system with the latest available patches on first boot
  • initial_boot_update.log (in /stage/scripts): the output log from the initial boot update script
  • setup_ssh_hadoop_user.sh (in /stage/scripts): configures SSH keys for the hadoop OS user, and must be run as root before you start services
  • start_all.sh (in /home/hadoop): starts all Hadoop services for single-node operation
  • stop_all.sh (in /home/hadoop): stops all Hadoop services for single-node operation

Disabling the Initial Boot Update Script

The OS update script runs automatically on every reboot via crontab. If you prefer to manage updates manually, disable it:

rm -f /stage/scripts/initial_boot_update.sh crontab -e
# DELETE THE BELOW LINE, SAVE AND EXIT THE FILE: @reboot /stage/scripts/initial_boot_update.sh

Web Management Interfaces

The Hadoop ecosystem provides several web interfaces for monitoring and management. All URLs use the public or private IP of your EC2 instance.

  • ResourceManager UI at http://PUBLIC_IP:8088: the YARN cluster overview, application status, node information and scheduler metrics
  • NodeManager UI at http://PUBLIC_IP:8042: individual node details, including resource usage and container logs

After you start all Hadoop services with the start-all.sh script, these interfaces are accessible from your browser immediately, provided the corresponding ports are open in your security group.

Troubleshooting

Cannot connect via SSH

  • Confirm that the instance has reached 2/2 status checks in the EC2 console.
  • Verify that your security group allows inbound TCP traffic on port 22 from your IP address.
  • Make sure you are using the correct key pair and connecting as ec2-user.
  • Check that the key file permissions are set correctly: chmod 400 /path/to/your-key.pem.

Hadoop services fail to start

  • Make sure you ran the SSH key setup script as root first: ./stage/scripts/setup_ssh_hadoop_user.sh.
  • Verify that you are starting services as the hadoop user, not as root or ec2-user.
  • Check that you source the start script correctly with . ./start-all.sh (note the leading dot and space).
  • Review the Hadoop logs in /apps/hadoop/logs/ for specific error messages.

ResourceManager or NodeManager UI not accessible

  • Confirm that Hadoop services are running by checking the jps output as the hadoop user.
  • Verify that your security group allows inbound traffic on ports 8088 and 8042.
  • Check that services are bound to the correct network interface and not just localhost.

Hive, HBase, or other component commands not found

  • Make sure you are running as the hadoop user: sudo su - hadoop.
  • Verify that the PATH includes the relevant component bin directories.
  • Check that the component installation exists under /apps/.

Out of memory errors during job execution

  • This AMI supports a minimum of 1 GB RAM, but big data workloads typically need much more.
  • Consider resizing the EC2 instance to a type with more memory (m5.xlarge or larger).
  • Adjust the YARN memory settings in /apps/hadoop/etc/hadoop/yarn-site.xml.

HDFS reports no space available

  • Check disk usage with df -h and verify that the /apps volume has available space.
  • Consider resizing the EBS volume attached to /apps if you need more storage.
  • Clean up temporary files or old job outputs in HDFS with hdfs dfs -rm -r /path/to/old/data.

Security Recommendations

Restrict Web Interface Access

The Hadoop web interfaces (ports 8088 and 8042) do not have built-in authentication by default. Open these ports only to trusted IP addresses in your security group. Never expose them to 0.0.0.0/0.

Use SSH Tunneling for Web Interfaces

Instead of opening Hadoop ports directly, use SSH tunneling to access the web UIs securely:

ssh -i /path/to/your-key.pem -L 8088:localhost:8088 -L 8042:localhost:8042 ec2-user@<PUBLIC_IP>

Then open the interfaces at http://localhost:8088 and http://localhost:8042 from your local browser.

Restrict SSH Access

Limit SSH (port 22) to specific trusted IP addresses. Consider using AWS Systems Manager Session Manager as an alternative to direct SSH access.

Apply Security Updates Regularly

Keep the operating system and all packages up to date:

sudo yum update -y

Protect the Hadoop User

The hadoop OS user owns all the big data components and services. Make sure that only authorized administrators can switch to this user, and review the sudoers configuration to limit access.

Enable Hadoop Security Features

For production environments, consider enabling Kerberos authentication for the Hadoop cluster. Kerberos adds strong authentication between all Hadoop components and prevents unauthorized access to HDFS data.

Back Up HDFS Data

Regularly back up important HDFS data. Use distcp to copy data to Amazon S3:

hadoop distcp hdfs:///important-data s3a://your-backup-bucket/hadoop-backup/

You can also use AWS EBS snapshots for volume-level backups of the /apps partition.

Support

If you run into an issue that this guide does not cover, or you need further help, the SecureAnyCloud support team is available 24/7.

  • Email: sales@secureanycloud.com
  • Phone: (+44) 02045382725
  • Website: secureanycloud.com
  • Address: 3rd Floor, 86 90 Paul Street, London, EC2A 4NE

When you contact support, include your EC2 instance ID, the AWS Region, and a description of the issue along with any relevant log output.

Deploy on AWS

Launch Hadoop Big Data Stack with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS

Launch Hadoop Big Data Stack with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com