Apache Hop 2.20.0

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Apache Hop 2.20.0 on Ubuntu 24.04 as the software version and select your Region, then choose Launch through EC2.

Choose m5.large, your key pair, and a security group as described above. Hop Web keeps each signed-in user's designer in server memory, so use a larger type for several designers or large pipelines. The AMI attaches the 20 GiB data volume for /var/lib/hop automatically. If your projects will hold large files, increase its size in the storage section of the launch wizard.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-hop}]'

Connect over SSH with your key pair. The login user depends on the version you launched:

• Apache Hop 2.20.0 on Ubuntu 24.04: log in as ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot generates the certificate, the admin password and the encryption key, and creates Hop's configuration and projects. The egress firewall, Hop Web and nginx then start. Hop Web takes about 40 seconds to start:

test -f /var/lib/sac/data/hop-firstboot.done && echo "first boot complete" systemctl is-active sac-hop-egress hop-web nginx

first boot complete active active active

Only root can read the credentials file. The following command prints it with the password hidden. To show the password, run sudo cat /root/hop-credentials.txt:

sudo sed -E 's/^(HOP_[A-Z_]*PASSWORD=).*/\1<hidden>/' /root/hop-credentials.txt

# Apache Hop 2.20.0 (Hop Web) - generated on first boot by hop-firstboot.service
# Every value below is unique to this instance; none was baked into the image. Keep this file secret. HOP_URL=https://ec2-54-84-58-220.compute-1.amazonaws.com/ui HOP_HOST=ec2-54-84-58-220.compute-1.amazonaws.com HOP_ADMIN_USER=admin HOP_ADMIN_PASSWORD=<hidden> HOP_CA_CERT=/etc/hop/tls/hop.crt HOP_CERT_SHA256=7157bc4250dfef8b08a3c9f61febf8a41d6129aeedd282c55f7badb9e31093d5
#
# Hop Web: https://<HOP_HOST>/ui, sign in as HOP_ADMIN_USER / HOP_ADMIN_PASSWORD (self-signed certificate; see the guide).
# New admin password / locked out:
sudo hop-reset-admin
# Run a pipeline from the shell:
sudo hop-admin hop-run --project samples --file '<pipeline.hpl>'
# Encrypt a connection password:
sudo hop-admin hop-encrypt -hop '<password>'
# Change the name it serves as:
sudo hop-set-hostname <your-dns-name>
# Allow a private database host:
sudo hop-egress-allow add <cidr>
# User guide: https://secureanycloud.com/stacks/apache-hop-2-20-0/

Your instance shows its own DNS name and certificate fingerprint. HOP_CERT_SHA256 is the SHA-256 fingerprint of the certificate. Compare it with the one your browser shows before you accept the certificate for the first time.

Browse to the HOP_URL from Step 4. The certificate is self-signed, so the browser warns once. Check that the fingerprint matches HOP_CERT_SHA256, then continue. Use the instance's DNS name as shown, because a request for any other host name (such as the bare IP address) is dropped. To serve Hop under your own name, see Step 10.

Sign in as admin with HOP_ADMIN_PASSWORD.

Hop Web opens with its welcome dialog over the file explorer of the default project. The project selector in the toolbar switches between default and samples. The samples project holds Apache's example pipelines and workflows, which show how transforms fit together.

Give your own projects a home folder under /var/lib/hop (for example /var/lib/hop/projects/sales) so that they are on the data volume. Step 6 shows how to create one from the command line.

sudo hop-admin runs Hop's command line tools as the hop service account, against the same configuration and projects as Hop Web and inside the same egress firewall. Run the generate-rows example from the samples project:

sudo hop-admin hop-run --project samples --file '${PROJECT_HOME}/transforms/generate-rows.hpl' --runconfig local --level Basic 2>&1 | grep -E 'Finished processing|Pipeline duration|Execution finished' | sed -E 's/^[0-9/: .]+- //'

generate 10 rows.0 - Finished processing (I=0, O=0, R=0, W=10, U=0, E=0) generate-rows - Pipeline duration : 0.126 seconds [ 0.126" ] generate-rows - Execution finished on a local pipeline engine with run configuration 'local'

${PROJECT_HOME} is in single quotes so that Hop, not the shell, expands it.

To create a project of your own on the data volume, first make sure the folder exists and belongs to the hop user. The project then appears in Hop Web's project selector. For a project called sales:

sudo install -d -o hop -g hop -m 0750 /var/lib/hop/projects /var/lib/hop/projects/sales
sudo hop-admin hop-conf --project-create --project=sales --project-home=/var/lib/hop/projects/sales

To schedule a pipeline or workflow, use cron or a systemd timer that calls hop-admin hop-run (for workflows, --file points at a .hwf file). The other tools are hop-conf (projects and environments), hop-encrypt, hop-search and hop-import. Run sudo hop-admin hop-run --help to list the options.

Hop Web includes the Hop Server API under /hop/ (pipeline and workflow status, remote execution), behind the same sign-in. The following command fetches the status page as XML. It passes the password to curl on standard input instead of on the command line:

H=$(sudo sed -n 's/^HOP_HOST=//p' /root/hop-credentials.txt)
sudo sh -c 'printf "user = \"admin:%s\"\n" "$(sed -n "s/^HOP_ADMIN_PASSWORD=//p" /root/hop-credentials.txt)"' \
  | sudo curl -s --cacert /etc/hop/tls/hop.crt -K - "https://$H/hop/status/?xml=Y" | grep -oE '<(statusdesc|cpu_cores|os_name)>[^<]*'

<statusdesc>Online <cpu_cores>2 <os_name>Linux

Hop ships JDBC drivers for PostgreSQL, Microsoft SQL Server, Amazon Redshift, IBM Db2 for i (jt400), MonetDB, CrateDB, H2, HSQLDB, Apache Derby, SQLite and Microsoft Access in /opt/hop/lib/jdbc. Hop does not bundle some drivers for licence reasons, such as MySQL and Oracle. To add one, copy its jar there as root (sudo install -m 0644 -o root -g root <driver>.jar /opt/hop/lib/jdbc/) and restart Hop Web with sudo systemctl restart hop-web.

Passwords in connections are encrypted. This image sets Hop's AES2 password encoder with a key generated on your instance's first boot (/etc/hop/encoder.key, readable only by root and the hop service). Passwords you type into a connection in Hop Web are stored encrypted with it. To put a password into a metadata file or an environment variable yourself, encrypt it first:

sudo hop-admin hop-encrypt -hop '<database-password>'

The output is an AES2 ... value to paste in place of the password. Keep a copy of /etc/hop/encoder.key with your backups. Projects copied to another server can decrypt their passwords only with the same key.

Databases in your VPC are blocked until you allow them. The egress firewall refuses connections from Hop to private addresses, so a connection to an RDS instance or a database on 10.x, 172.16-31.x or 192.168.x fails with "connection refused" until you allow its network. Allow the database's address as a /32, or its whole subnet, such as 10.0.5.0/24:

sudo hop-egress-allow add <private-ip>/32

This command opens the range in both firewall layers, keeps it across reboots and restarts Hop Web. sudo hop-egress-allow list shows what is open, and sudo hop-egress-allow remove <cidr> closes it again. Only private ranges (inside 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 or fc00::/7) are accepted. The instance metadata service and link-local addresses can never be allowed. Public endpoints (SaaS APIs, public S3 URLs, Maven Central) need no change.

Instance role credentials are not available to Hop. Because Hop cannot reach the instance metadata service, it cannot pick up the EC2 instance role. For Amazon S3, give Hop's S3 configuration an access key for an IAM user (or role) that is limited to the buckets it needs.

Run as the hop user, the metadata service and a private address are refused, and a public endpoint works:

for u in http://169.254.169.254/latest/meta-data/ http://10.0.0.1/ https://repo1.maven.org/maven2/; do
sudo setpriv --reuid=hop --regid=hop --clear-groups -- curl -s -m 20 -o /dev/null -w "%{http_code} $u\n" "$u" || true done

000 http://169.254.169.254/latest/meta-data/ 000 http://10.0.0.1/ 200 https://repo1.maven.org/maven2/

000 means the connection was refused before any HTTP was exchanged. The rules themselves:

sudo iptables -w -S SAC-HOP-EGRESS | grep -E -- '-d (169.254|10\.|172\.16|192\.168)' | grep tcp

-A SAC-HOP-EGRESS -d 169.254.0.0/16 -p tcp -j REJECT --reject-with tcp-reset -A SAC-HOP-EGRESS -d 10.0.0.0/8 -p tcp -j REJECT --reject-with tcp-reset -A SAC-HOP-EGRESS -d 172.16.0.0/12 -p tcp -j REJECT --reject-with tcp-reset -A SAC-HOP-EGRESS -d 192.168.0.0/16 -p tcp -j REJECT --reject-with tcp-reset

Everything except the health check needs a password. A cross-site request is refused, a request for a host name the instance does not serve is dropped without an answer, and nothing answers on port 80:

H=$(sudo sed -n 's/^HOP_HOST=//p' /root/hop-credentials.txt) C=/etc/hop/tls/hop.crt curl -s --cacert $C -o /dev/null -w 'Hop Web, no password -> HTTP %{http_code}\n' -H 'Accept: application/json' https://$H/ui curl -s --cacert $C -o /dev/null -w 'Hop Server API, no password -> HTTP %{http_code}\n' -H 'Accept: application/json' https://$H/hop/status/ printf 'user = "admin:not-the-password"\n' | curl -s --cacert $C -K - -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' -H 'Accept: application/json' https://$H/hop/status/ curl -s --cacert $C -o /dev/null -w 'cross-site request -> HTTP %{http_code}\n' -H 'Origin: https://other.example' https://$H/ui curl -s --cacert $C -o /dev/null -w 'health check -> HTTP %{http_code}\n' https://$H/healthz curl -sk -H 'Host: other.example' -o /dev/null -w 'unknown host -> %{http_code}\n' https://127.0.0.1/healthz || true curl -s -o /dev/null -w 'plain http :80 -> %{http_code}\n' --connect-timeout 3 http://127.0.0.1/ || true

Hop Web, no password -> HTTP 401 Hop Server API, no password -> HTTP 401 wrong password -> HTTP 401 cross-site request -> HTTP 403 health check -> HTTP 200 unknown host -> 000 plain http :80 -> 000

A browser without a session is sent to the sign-in page instead of receiving the 401. The health check (/healthz) returns only ok and is meant for load balancers.

Hop Web listens on loopback only, and nginx is the only service on 443:

sudo ss -Htlnp | awk '{print $4, $6}' | grep -E ':(443|8080) ' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u

0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::ffff:127.0.0.1]:8080 users:(("java"))

sudo hop-rotate-password gives admin a new random password, writes it to /root/hop-credentials.txt and restarts Hop Web. The old password stops working at once, and open sessions end.

sudo hop-rotate-password

If you lose the password, or Hop's user store (/var/lib/hop/config/security/users.json) no longer has a working admin (disabled, deleted or without the admin role), sudo hop-reset-admin restores admin with the admin role and a new password, and forces Hop Web's sign-in back on. Other users are kept as they are. Neither command prints the password, so read it from the credentials file.

sudo hop-reset-admin

To use your own DNS name, point a DNS record at the instance, then set the name it is served as. This creates a new self-signed certificate for the name (and the instance's IP addresses), updates nginx and the URL in the credentials file, and keeps the name across reboots. The fingerprint changes, so check the new HOP_CERT_SHA256 in your browser.

sudo hop-set-hostname <your-domain>

To use a certificate from your own CA instead, replace /etc/hop/tls/hop.crt and /etc/hop/tls/hop.key (mode 0600, owner root) with yours for the same name and run sudo systemctl reload nginx. Alternatively, put an Application Load Balancer with an ACM certificate in front of port 443.

• Apache Hop (command line tools, plugins, JDBC drivers): /opt/hop (read-only, owner root)
• Hop Web application and Tomcat: /opt/hop-web (Tomcat base), /opt/tomcat (read-only)
• Configuration, users, projects, audit log, Tomcat work files: /var/lib/hop (data volume)
• Licences and build provenance: /usr/share/doc/apache-hop/
• Service units: hop-web.service, sac-hop-egress.service, hop-firstboot.service in /etc/systemd/system/
• Egress rules: /usr/local/sbin/hop-egress-apply, with allowed networks in /etc/sac/hop-egress-allow
• nginx site: /etc/nginx/sites-available/sac-hop
• Certificate, key and password encryption key: /etc/hop/tls/, /etc/hop/encoder.key
• Helpers: /usr/local/sbin/hop-admin, hop-rotate-password, hop-reset-admin, hop-set-hostname, hop-egress-allow, hop-guard
• Credentials: /root/hop-credentials.txt (mode 0600)

sudo hop-admin hop-run --version 2>/dev/null | tail -1 java -cp /opt/tomcat/lib/catalina.jar org.apache.catalina.util.ServerInfo | head -1 grep -E '^(hop|tomcat|removed):' /usr/share/doc/apache-hop/BUILD-PROVENANCE.txt | cut -c1-100

2.20.0 Server version: Apache Tomcat/10.1.61 hop: https://downloads.apache.org/hop/2.20.0/apache-hop-client-2.20.0.zip removed: plugins/transforms/propertyinput (Property Input transform; its ini4j 0.5.4 has CVE-2022-4 tomcat: https://downloads.apache.org/tomcat/tomcat-10/v10.1.61/bin/apache-tomcat-10.1.61.tar.gz

Not included: the Property Input transform. This image does not ship Hop's optional Property Input transform (plugins/transforms/propertyinput), which reads Java .properties and INI files into a pipeline. It is the only part of Hop that uses the ini4j 0.5.4 library, and ini4j has a published vulnerability, CVE-2022-41404, for which no fixed release exists upstream, so the transform and ini4j were removed rather than shipped with a known unfixed advisory. Every other transform is present and Hop starts without it. Pipelines that use Property Input will report the transform as missing; read such files with the Text File Input transform instead, or add the plugin yourself from the upstream Hop 2.20.0 release if you accept that risk.

Logs: sudo journalctl -u hop-web for Hop Web and Tomcat, /var/log/nginx/access.log for requests (with the real client address).

• Allow port 22 in the security group from your own networks only, and port 443 only from the networks your designers and API clients use.
• Treat HOP_ADMIN_PASSWORD like a root password: anyone who signs in to Hop Web can run pipelines and workflows, including shell actions, as the hop user.
• Open private networks with hop-egress-allow one subnet at a time, only for the databases and services your pipelines need.
• Keep /etc/hop/encoder.key and EBS snapshots of the data volume with your backups. The snapshots hold your projects, and the key decrypts the passwords in them.
• Use your own DNS name (Step 10), and rotate the password when someone leaves or it may have been exposed.
• Keep the operating system patched. Unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.