SecureAnyCloud
Deployment guide

Manifoldcf

Overview

Apache ManifoldCF is an open-source content ingestion framework. Repository connectors pull documents out of the systems they live in (filesystems, web sites, SharePoint, RSS feeds, relational databases, email stores and document management systems), and output connectors push them into a search engine. Between the two sits a crawl scheduler that records per-document state, so a repeat run reprocesses only what actually changed instead of re-indexing everything from scratch.

This guide covers the deployment and operation of Apache ManifoldCF on Ubuntu 24.04 on Azure, using the SecureAnyCloud images on the Azure Marketplace.

What distinguishes ManifoldCF from a plain crawler is its security model. Its authority services translate a user identity into the access tokens that user holds in the source repository, and those tokens are indexed alongside each document. A search front end can then return only the documents a given person is entitled to open. This matters in front of SharePoint or filesystem content that carries real access control lists, because showing someone a document title they are not cleared for is itself a disclosure.

The SecureAnyCloud image ships the free and open-source, Apache-2.0 licensed ManifoldCF 2.30, and does the part that is normally a day of work: a complete, wired-up pipeline. Apache Solr 9.10.1 is installed locally as a working output target. PostgreSQL 16 backs the crawl database, in place of the evaluation-grade embedded store that upstream's examples use. A small self-authored sample corpus plus a one-command round-trip script let you prove the whole path (crawl, extract, index, retrieve) before you point it at anything of your own. Backed by 24/7 SecureAnyCloud support.

Apache, Apache ManifoldCF, Apache Solr and their logos are trademarks of The Apache Software Foundation. This image is produced by SecureAnyCloud and is not affiliated with, endorsed by, or sponsored by the ASF. It ships the free and open source software, unmodified.

What you get:

  • Apache ManifoldCF 2.30 (Apache-2.0), installed from the official Apache CDN and SHA-512 verified at build time.
  • Apache Solr 9.10.1 (Apache-2.0) as a local, ready-to-use output target, with a SecureAnyCloud core already created.
  • PostgreSQL 16 as the crawl database, with the schema initialised and every connector registered.
  • OpenJDK 21, which is required and not merely supported, because ManifoldCF 2.30's classes are Java 21 bytecode.
  • nginx as the only public surface, publishing the crawler console and the JSON API and nothing else.
  • The Tika transformation stage wired into the crawl pipeline, so that document text is extracted and indexed, not just filenames.
  • A six-document sample corpus and manifoldcf-crawl-roundtrip.sh, a one-command, end-to-end proof.
  • Four secrets generated per VM at first boot: console login, API credential, database password and Jetty shutdown token.
  • Ubuntu 24.04 LTS base, fully patched at build time, with unattended security upgrades enabled.

How the pieces fit together

It is worth understanding the topology before you deploy, because it explains several of the choices below. Only nginx is reachable from outside the VM.

  • nginx: port 80, bound to all interfaces. This is the only public surface.
  • ManifoldCF (Jetty): port 8345, bound to 127.0.0.1. Not reachable from outside.
  • Apache Solr: port 8983, bound to 127.0.0.1. Not reachable from outside.
  • PostgreSQL: port 5432, bound to 127.0.0.1. Not reachable from outside.

ManifoldCF runs as a single JVM hosting the agents daemon in process, alongside the three standard web contexts (/mcf-crawler-ui, /mcf-api-service, /mcf-authority-service) at their normal paths. Upstream documentation and tutorials therefore apply to this image unchanged.

All three contexts share one Jetty connector, and the authority service has no credential mechanism upstream, so publishing Jetty directly would expose an unauthenticated service. That is why Jetty is held on loopback, and nginx republishes only the two contexts that can be protected, returning 403 for the authority service explicitly.

If you outgrow a single node, ManifoldCF supports multiprocess and ZooKeeper topologies. That is a migration away from this image's layout rather than a setting, and upstream documents it.

Prerequisites

  • An Azure subscription with permission to create VMs and edit network security groups.
  • The Azure CLI installed and signed in, if you are deploying from the command line.
  • An SSH key pair for administrative access to the VM.

Sizing

The recommended size for this image is Standard_B2s (2 vCPU, 4 GiB). The image is tuned to fit that size: the ManifoldCF JVM is capped at a 768 MB heap and Solr at 512 MB, leaving room for PostgreSQL and the OS. First boot recomputes both heaps from the memory actually present, so a larger VM benefits automatically without you editing anything.

Choose a larger size when your crawl is larger. Two things push you up: corpus size (Solr's index and its caches) and crawl concurrency.

The image ships crawler.threads=10 and a database handle pool of 20 against a PostgreSQL max_connections of 50. This is a deliberate reduction from upstream's defaults of 50 threads and 100 handles, which would exhaust PostgreSQL's connection limit on any machine this size. If you raise the thread count, raise the pool and max_connections with it. Keep upstream's documented constraint: workers + deletes + expirations + cleanup + 10 must stay below the maximum handle count.

1

Deploy from the Azure Portal

  • In the Azure Portal, search the Marketplace for Apache ManifoldCF on Ubuntu 24.04 LTS by SecureAnyCloud and select Create.
  • Choose your subscription, resource group and region.
  • Set the VM size to Standard_B2s or larger.
  • Provide an administrator username and SSH public key.
  • On the Networking tab, allow inbound HTTP (80) and SSH (22). Restrict the source to your own address range rather than leaving either open to the internet.
  • Select Review + create.
2

Deploy from the Azure CLI

Run these commands on your own machine. Replace the resource group, name and region to suit.

az group create --name manifoldcf-rg --location eastus
az vm image terms accept --urn sac1647283583153:manifoldcf-ubuntu-24-04:default:latest
az vm create \
  --resource-group manifoldcf-rg \
  --name manifoldcf-vm \
  --image sac1647283583153:manifoldcf-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
az vm open-port \
  --resource-group manifoldcf-rg \
  --name manifoldcf-vm \
  --port 80 --priority 1010

The az vm open-port call publishes the console port. Afterwards, restrict its source to your own network in the network security group, rather than leaving port 80 open to the internet.

3

Connect to your VM

ssh azureuser@<vm-ip>

First boot generates this VM's secrets, initialises the database schema and starts the crawler. This takes a minute or two on a Standard_B2s. The steps below confirm that it finished.

4

Confirm the services are running

Four services make up the appliance. All four should report active, and all four are enabled at boot:

systemctl is-active postgresql solr manifoldcf nginx
active active active active
systemctl is-enabled postgresql solr manifoldcf nginx
enabled enabled enabled enabled

Confirm that the JVM is OpenJDK 21. This is not cosmetic. ManifoldCF 2.30's class files are Java 21 bytecode, and an older JVM cannot load them:

java -version 2>&1 | head -1
openjdk version "21.0.11" 2026-04-21
5

Read the per-instance credentials

This image ships no default password. Four secrets are generated on the first boot of every VM and written to a file that only root can read.

sudo ls -l /root/manifoldcf-credentials.txt
-rw------- 1 root root 659 Jul 20 05:39 /root/manifoldcf-credentials.txt
sudo cat /root/manifoldcf-credentials.txt

The file records the console URL, the API URL and the following secrets:

  • MCF_ADMIN_USER / MCF_ADMIN_PASSWORD: the crawler console login.
  • MCF_API_USER / MCF_API_PASSWORD: HTTP Basic on the JSON API, enforced by nginx.
  • MCF_DB_USER / MCF_DB_PASSWORD: the PostgreSQL role backing the crawl database.

A fourth secret, the Jetty shutdown token, is rotated at the same time and is not something you need to use. Upstream ships that token as the literal string secret_token in every options file it publishes, which is exactly the kind of value an image must not inherit.

The URLs in the file are written against the VM's own address. To reach the console from your workstation, use the VM's public IP.

6

Sign in to the crawler console

Open http://<vm-ip>/mcf-crawler-ui/ in a browser. The root path redirects there for you.

Sign in with the MCF_ADMIN_USER and MCF_ADMIN_PASSWORD values from Step 5. You land on the console home with the main navigation down the left: Outputs, Authorities, Repositories, Jobs, Status Reports and History Reports.

7

Verify the security model

Run these four checks yourself. They matter because a crawler holds credentials for every repository it reads.

The console is published and answers on port 80, a 302 because an anonymous request is redirected to the login form rather than being shown any content:

curl -s -o /dev/null -w 'crawler console: %{http_code}\n' http://127.0.0.1/mcf-crawler-ui/
crawler console: 302

The authority service is explicitly refused rather than merely unpublished, so the intent is visible in the configuration:

curl -s -o /dev/null -w 'authority service: %{http_code}\n' http://127.0.0.1/mcf-authority-service/
authority service: 403

The JSON API refuses an unauthenticated caller:

curl -s -o /dev/null -w 'API without credentials: %{http_code}\n' http://127.0.0.1/mcf-api-service/json/outputconnections
API without credentials: 401

Finally, confirm that the three internal components are bound to loopback only, so nothing but nginx is exposed:

ss -ltn '( sport = :8345 or sport = :8983 or sport = :5432 )' | awk 'NR>1{print $4}'
127.0.0.1:5432 [::ffff:127.0.0.1]:8983 [::ffff:127.0.0.1]:8345

A note on how the API is protected. ManifoldCF's own API servlet authenticates with a hardcoded empty credential internally, so the upstream apilogin.* properties cannot be used to secure it. Setting them does not add a password. It makes every request fail permanently with no way to supply one. This image therefore leaves those properties at their upstream empty default and enforces authentication where it genuinely can be enforced: nginx HTTP Basic on /mcf-api-service, with Jetty unreachable except through nginx. Do not "harden" the image by populating apilogin.name and apilogin.password; it will lock you out of your own API.

8

Run the built-in crawl round trip

The image ships a six-document sample corpus, self-authored by SecureAnyCloud so that no third-party content with unclear licensing is included. List it:

ls -1 /var/lib/manifoldcf/sample-corpus
01-what-is-manifoldcf.txt 02-repository-connectors.txt 03-output-connectors.txt 04-jobs-and-scheduling.txt 05-document-security.txt 06-operating-the-appliance.txt

Each document carries a distinct planted sentinel phrase, which is what makes the round trip a real proof rather than a smoke test. Run the script:

sudo /usr/local/sbin/manifoldcf-crawl-roundtrip.sh

It ends with:

ROUNDTRIP_OK: 6 documents crawled from /var/lib/manifoldcf/sample-corpus into Solr with content verified

The script defines the Solr output connection, the Tika text extraction stage and a filesystem repository connection over the corpus, creates a job, runs it to completion, and then asserts against Solr, document count and document content, rather than against the job's status. That distinction is the point: a ManifoldCF job that ingests zero documents still finishes and still reports a terminal status, so asserting on job status would pass while the product was entirely broken.

9

Retrieve the indexed content

Confirm that the documents are in the index:

curl -s 'http://127.0.0.1:8983/solr/sac/select?q=*:*&rows=0' | grep numFound
"numFound":6,

Then retrieve one by its planted content, not by its filename. Note the quoted phrase: Solr's default analyzer splits SAC-SENTINEL-ALPHA on the hyphens and matches any of the parts, so an unquoted query would match every document in the corpus and prove nothing:

curl -s --get --data-urlencode 'q="SAC-SENTINEL-ALPHA"' 'http://127.0.0.1:8983/solr/sac/select?fl=id&rows=3'

Exactly one document comes back, and it is the right one:

"numFound":1, "docs":[{ "id":"file:/var/lib/manifoldcf/sample-corpus/01-what-is-manifoldcf.txt" }]

A control query for a sentinel that was never planted returns nothing, which confirms the query is discriminating rather than matching everything:

curl -s --get --data-urlencode 'q="SAC-SENTINEL-NEVERPLANTED"' 'http://127.0.0.1:8983/solr/sac/select?rows=0' | grep numFound
"numFound":0,
10

Inspect the crawl in the console

Everything the round trip built is visible in the console, and this is where you will work when you build your own crawls.

Under Jobs, Status and Job Management you can see the sample job, its terminal status and, the number that actually matters, how many documents it processed:

Under Repositories, List Repository Connections you can see the filesystem connection the job crawls, along with its connection type and authority group:

11

Build your own crawl

The workflow is always the same three objects, in this order:

  • An output connection: where documents go. sac-solr already exists and points at the local Solr core, so you can reuse it. Add your own under Outputs, List Output Connections if you are indexing elsewhere.
  • A repository connection: where documents come from. Under Repositories, List Repository Connections, choose a connection type (File system, Web, RSS, JDBC, and so on), give it the credentials for that system, and use the View action to confirm it reports working before going further.
  • A job: what to crawl and when. Under Jobs, List all Jobs, create a job that binds a repository connection to an output connection, set its document specification (paths, include and exclude rules), and optionally give it a schedule.

Two things that are easy to miss:

  • Add the Tika extraction stage to the job's pipeline if you want document text indexed rather than just metadata. This image's sample job does exactly that, so use it as a reference. Text extraction is performed in the crawler tier, which is both the Solr project's own recommendation and what makes extraction work reliably here.
  • Test the connection before you build the job. A repository connection that does not report working will produce a job that runs, completes and ingests nothing.
12

Using the JSON API

The API is at /mcf-api-service/json behind HTTP Basic. Anything the console can do, the API can do. It is the right way to script crawls.

sudo bash -c 'source /root/manifoldcf-credentials.txt; curl -s -u "$MCF_API_USER:$MCF_API_PASSWORD" http://127.0.0.1/mcf-api-service/json/outputconnections' | head -c 200

Job status is available the same way. Use it to poll a scheduled crawl:

sudo bash -c 'source /root/manifoldcf-credentials.txt; curl -s -u "$MCF_API_USER:$MCF_API_PASSWORD" http://127.0.0.1/mcf-api-service/json/jobstatuses' | head -c 400

One upstream quirk worth knowing before you script against it. ManifoldCF's JSON serialiser emits attribute keys as _attribute_name_ (with a trailing underscore) while its parser expects _attribute_name (without one). Reading an object with GET and feeding that response straight back into a PUT therefore stores a subtly corrupted configuration, and the connector will report a missing parameter for a value you can plainly see in the JSON. Build request bodies to the parser's spelling rather than by round-tripping a response.

13

Managing the services

systemctl status manifoldcf --no-pager --lines=0

To restart the crawler after a configuration change:

sudo systemctl restart manifoldcf.service

Logs go to the journal:

sudo journalctl -u manifoldcf -n 100 --no-pager

Solr and PostgreSQL are ordinary systemd services too (solr.service, postgresql.service), and they start before ManifoldCF at boot.

14

Changing the credentials

The console login lives in ManifoldCF's properties file, which is 0640 root:manifoldcf:

sudo sed -i 's|^\(.*org.apache.manifoldcf.login.password.*value="\)[^"]*|\1<new-password>|' /opt/manifoldcf/properties.xml
sudo systemctl restart manifoldcf.service

The API credential is an nginx htpasswd entry and does not require restarting ManifoldCF:

sudo htpasswd -B /etc/nginx/manifoldcf-api.htpasswd <api-user>
sudo systemctl reload nginx

Keep /root/manifoldcf-credentials.txt in step with any change you make, or treat the live configuration as authoritative from then on.

On password storage. ManifoldCF writes passwords into properties.xml in plaintext, and this image does not pretend otherwise. ManifoldCF does offer an "obfuscated" property variant, but its obfuscation uses a passcode and salt compiled into the shipped jar and identical in every installation worldwide. Anyone holding the jar can reverse it. Using it would be security theatre, so protection here is filesystem permissions: the file is readable only by root and the service account.

15

Use your own domain and HTTPS (production)

For anything beyond a trial, put a real hostname and a certificate in front of nginx. Install a certificate with Certbot against the nginx site this image already ships, then reload nginx. Everything reaches the appliance through nginx already, so no other component needs changing, and because the console login and the API credential both travel over that connection, HTTPS is the difference between authenticating privately and authenticating in clear text over the network.

16

Security recommendations

  • Restrict the source addresses on the network security group rule for port 80 to your own network. The console can configure crawls against your internal systems.
  • Terminate TLS before using this in production, per Step 15.
  • Treat the repository credentials as the sensitive part. A crawler necessarily holds working credentials for every system it reads; the console is as privileged as the sum of those systems.
  • Do not populate apilogin.*. See the note in Step 7. It does not harden the API. It disables it.
  • Use the authority services if your source content carries access control lists and you are building a user-facing search experience. Indexing permissions alongside documents is what stops search results leaking titles to people who cannot open them.
  • Keep the OS patched. Unattended security upgrades are enabled on this image by default.
17

Support and Licensing

Apache ManifoldCF and Apache Solr are free and open source software under the Apache-2.0 licence. The ManifoldCF binary distribution bundles third party components under BSD (the PostgreSQL JDBC driver, HSQLDB), CDDL 1.0 (several Sun/GlassFish components), EPL 1.0 (the Eclipse compiler) and MIT (json-simple). No GPL-only, SSPL or AGPL code ships in this image. The ASF excludes connectors requiring such jars from its release, and this image does not add them.

Apache Solr was chosen as the bundled search target deliberately: it is ManifoldCF's best supported output connector, and it is Apache-2.0. Elasticsearch was not viable for an image whose listing states Apache-2.0, because Elasticsearch is distributed under the SSPL. ManifoldCF's Elasticsearch and OpenSearch output connectors remain available to you if you are pointing at an existing cluster of your own.

This image is produced by SecureAnyCloud and is not affiliated with, endorsed by, or sponsored by The Apache Software Foundation. The Apache, ManifoldCF and Solr names are used only to identify the software included.

For help with the image, contact SecureAnyCloud support, which is available 24/7. For questions about ManifoldCF itself, see the upstream project at manifoldcf.apache.org.

Deploy on Azure

Launch Apache ManifoldCF on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on Azure