Apache Nutch

sudo systemctl is-active solr.service /opt/solr/bin/solr version NUTCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 /opt/nutch/bin/nutch version | grep -i nutch ss -tln | grep 8983

solr.service is active and listening on 8983. The bundled Solr reports 9.10.1 and Nutch reports 1.22. Nutch has no long-running daemon, because you run it as a batch job when you crawl.

curl -s -o /dev/null -w 'no-auth: HTTP %{http_code}\n' http://127.0.0.1:8983/solr/admin/info/system
PASS=$(sudo grep '^SOLR_ADMIN_PASSWORD=' /stage/scripts/nutch-credentials.log | cut -d= -f2-)
curl -sf -u "sac:${PASS}" 'http://127.0.0.1:8983/solr/admin/info/system?wt=json' | head -c 300

The first call returns 401 from the BasicAuthPlugin auth wall, so no unauthenticated Solr admin is reachable off-host. The second call authenticates as SecureAnyCloud and returns the system info JSON, including solr-spec-version.

sudo cat /stage/scripts/nutch-credentials.log

Note SOLR_ADMIN_URL, SOLR_ADMIN_USER (SecureAnyCloud), SOLR_ADMIN_PASSWORD and NUTCH_CORE_URL. The password is a fresh 32-character hex string generated on this VM's first boot, so no two VMs share credentials. The same secret is already injected into Nutch's Solr-indexer config, so the crawler authenticates automatically.

Run the bundled, self-contained demo to prove the whole pipeline. It serves a throwaway HTML page on 127.0.0.1:8899 and runs one Nutch crawl round that indexes into the Solr nutch core. It then queries the document back out with the per-VM credentials.

sudo nutch-demo-crawl
PASS=$(sudo grep '^SOLR_ADMIN_PASSWORD=' /stage/scripts/nutch-credentials.log | cut -d= -f2-)
curl -sf -u "sac:${PASS}" 'http://127.0.0.1:8983/solr/nutch/select?q=title:sac&fl=id,title,url&wt=json' | head -c 400

nutch-demo-crawl prints NUTCH_E2E_OK once the crawled document is searchable, and the follow-up query returns it under response.docs with numFound: 1.

Solr listens on 127.0.0.1 only, so first open an SSH tunnel from your workstation:

ssh -L 8983:127.0.0.1:8983 azureuser@<vm-ip>

Then browse to http://localhost:8983/solr/ and sign in as SecureAnyCloud with the per-VM password. The Dashboard shows the Solr instance, JVM memory (-Xmx512m on Standard_B2s), versions, and the active BasicAuthPlugin and RuleBasedAuthorizationPlugin.

Pick nutch in the Core Selector. This is the core that Nutch indexes every crawled page into. Its overview lists the schema, document statistics and the query interface.

Open the Query tab on the nutch core, set q to title:sac and click Execute Query. Solr returns the crawled demo document, where id is the page URL, with title, url, content and host fields populated by Nutch's indexing filters.

To crawl a real site, put its seed URLs in a seeds directory and run one or more crawl rounds. bin/crawl -i indexes each round straight into the Solr nutch core, using the per-VM credentials already wired into conf/index-writers.xml:

sudo install -d -m 0700 /var/lib/nutch /var/lib/nutch/urls echo 'https://<your-domain>/' | sudo tee /var/lib/nutch/urls/seed.txt export NUTCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 NUTCH_HEAPSIZE=1024 cd /opt/nutch
sudo -E bin/crawl -i -s /var/lib/nutch/urls /var/lib/nutch/crawl 2

Set http.agent.name (and ideally http.agent.email) in /opt/nutch/conf/nutch-site.xml to identify your crawler to the sites you fetch. Edit /opt/nutch/conf/regex-urlfilter.txt to restrict the crawl scope. Increase the round count (the trailing 2) to crawl deeper.

Nutch's Solr indexer target lives in /opt/nutch/conf/index-writers.xml. It points at the local nutch core with BasicAuth enabled:

sudo grep -E 'name="url"|name="auth"|name="username"' /opt/nutch/conf/index-writers.xml

The url is http://localhost:8983/solr/nutch, auth is true, and username is SecureAnyCloud. The password is injected at first boot and never baked into the image. The enabled plugins (protocol-http, parse-html/parse-tika, index-basic/index-anchor and indexer-solr) are set in /opt/nutch/conf/nutch-site.xml.

Solr heap is set in /etc/default/solr.in.sh (SOLR_HEAP="512m" on Standard_B2s); Nutch's batch JVM is capped by NUTCH_HEAPSIZE (MB). Both fit a crawl-and-index cycle inside 4 GB:

grep -E '^SOLR_HEAP|^SOLR_JAVA_HOME' /etc/default/solr.in.sh

For large crawls, raise SOLR_HEAP to 4096m+ and NUTCH_HEAPSIZE to 4096+ on D4s/D8s instances, then restart Solr with sudo systemctl restart solr.service. For very large crawls, migrate Nutch to a Hadoop cluster and Solr to SolrCloud (add ZooKeeper). The crawl commands are unchanged.

sudo journalctl -u solr.service --no-pager -n 40
sudo journalctl -u nutch-firstboot.service --no-pager -n 20
sudo tail -40 /var/solr/logs/solr.log 2>/dev/null

Solr writes its main log to /var/solr/logs/solr.log. Each Nutch crawl round prints its Hadoop-local job output to the terminal; redirect it to a file for long crawls. The systemd journal captures the first-boot credential-rotation script.