Scrapyd

Security model, and why it matters here

A default Scrapyd install is not safe on a public address. Scrapyd's /addversion.json endpoint accepts an arbitrary Python egg, and its runner then executes code from that egg.

Upstream ships with its username and password settings empty. Scrapyd treats either one being empty as "authentication off". It logs Basic authentication disabled and serves every route to anybody. An unauthenticated Scrapyd reachable from a network is therefore not an open job queue. It is remote code execution as the service account.

Upstream's own mitigation is to bind to 127.0.0.1 by default. A deployable appliance cannot keep that default and still be useful.

This image closes the gap with two independent credentials, both unique to your VM and both generated on its first boot:

Layer What it is Where its secret lives

1. The gate nginx on :80, HTTP Basic, bcrypt. This is the credential you use. /var/lib/scrapyd/.htpasswd, 0640 root:www-data

2. Scrapyd itself Scrapyd's own native HTTP Basic, a different password /etc/scrapyd/scrapyd.env, 0600 root:root

nginx does not pass your credential through to Scrapyd. It verifies the credential and then replaces the Authorization header with Scrapyd's own. Scrapyd never accepts your credential, and Scrapyd's credential never leaves the VM.

This separation does real work. Anything that can talk to Scrapyd can run code as the scrapyd account, so a project you deploy can read whatever that account can read, including Scrapyd's own credential. The gate credential survives, because its bcrypt file is not readable by scrapyd at all. A careless or hostile spider therefore cannot recover the password you use to reach the appliance.

The protection holds only because the two credentials are different. For that reason, the image generates two and refuses to start if they match.

Three further points follow from the same reasoning:

• No default credential exists anywhere in the image. A non-empty default would be a known password on every VM, and an empty one would silently switch authentication off. So /etc/scrapyd/scrapyd.conf ships with username and password genuinely empty, and the real values only ever exist in a file created on your first boot.
• The services refuse to start without a credential rather than starting open. scrapyd.service is gated on both the credential file and a bootstrap marker. It loads the file in a way that makes a missing file a startup failure, and it runs a preflight guard that rejects a missing, empty, short or wrongly owned credential. nginx.service has the same protections, and its configuration includes the credential file, so nginx cannot even load without it.
• Scrapyd and the sample catalogue are bound to loopback. Port 80 is the only ingress. Nothing on 6800 or 6810 is reachable from the network.

One limitation: HTTP Basic Auth sends your credential in every request, and this image serves plain HTTP on port 80. Do not expose port 80 to the internet without TLS in front of it. The Maintenance section below covers adding TLS.

Prerequisites

• An active Azure subscription, an SSH key pair, and a VNet and subnet in the target region
• A Standard_B2s VM (2 vCPU / 4 GiB RAM) or larger. This size is a comfortable starting point, and the concurrency defaults in this image are sized for it: at most two spiders run at once.
• NSG inbound rules that allow 22/tcp from your management network, 80/tcp for the API and console, and 443/tcp if you add TLS

Crawling is more network-bound and memory-bound than CPU-bound. Size up if you intend to run many spiders concurrently or crawl very large sites, and raise max_proc_per_cpu to match (see Maintenance).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Scrapyd by SecureAnyCloud and select Create.

On Basics, pick your subscription, resource group, region and size. Under Administrator account, choose SSH public key and paste your key. Under Inbound port rules, allow SSH (22) and HTTP (80). Then select Review + create and Create.

Step 2 - Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:scrapyd-ubuntu-24-04:default:latest
az vm create \
  --resource-group <your-rg> \
  --name scrapyd \
  --image sac1647283583153:scrapyd-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard
az vm open-port --resource-group <your-rg> --name scrapyd --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active scrapyd.service nginx.service scrapyd-firstboot.service /opt/scrapyd/venv/bin/python -c 'import scrapyd, scrapy; print("scrapyd", scrapyd.__version__, "| scrapy", scrapy.__version__)'
sudo ss -tlnp | grep -E ':(80|6800|6810) ' | sed 's/users:.*((/ /; s/))$//'

All three units report active, and the version line confirms the pinned releases. Scrapyd listens on the loopback address 127.0.0.1:6800 and the bundled sample catalogue on 127.0.0.1:6810. Only nginx is bound to 0.0.0.0:80, so neither Scrapyd nor the catalogue is reachable from the network.

scrapyd-firstboot.service is a one-shot unit. It runs once on the first boot of your VM and generates both credentials. It does nothing on later boots, because it is gated on a sentinel file that it creates. active here means that it ran and completed, which is the expected end state.

Step 5 - Retrieve your per-VM credential

sudo cat /root/scrapyd-credentials.txt
sudo stat -c '%n %a %U:%G' /root/scrapyd-credentials.txt /var/lib/scrapyd/.htpasswd /etc/scrapyd/scrapyd.env

SCRAPYD_USERNAME and SCRAPYD_PASSWORD are the credential for your instance. Store them in your password manager.

The credentials file is 0600 root:root, the gate's bcrypt file is 0640 root:www-data, and Scrapyd's own credential is 0600 root:root. These permissions keep a deployed project from reading either credential.

Nothing here is a SecureAnyCloud default. openssl created these values on your VM's first boot, and no other SecureAnyCloud instance has them.

Step 6 - Verify the security posture yourself

Every claim in the security model above is a single command away. Run this block to check the anonymous routes and the configuration:

for R in / /jobs /listprojects.json; do printf 'anonymous GET %-22s -> HTTP ' "$R" curl -s -o /dev/null -w '%{http_code}\n' -m 15 "http://127.0.0.1$R" done for R in /schedule.json /addversion.json; do printf 'anonymous POST %-21s -> HTTP ' "$R" curl -s -o /dev/null -X POST -d 'project=sac_demo&spider=catalogue' \
  -w '%{http_code}\n' -m 15 "http://127.0.0.1$R" done grep -E '^(bind_address|http_port|username|password|items_dir|max_proc_per_cpu) ' /etc/scrapyd/scrapyd.conf

Every route answers 401, including the two that would amount to remote code execution on an unauthenticated instance. The configuration dump shows username and password genuinely empty, which proves that no credential is baked into this image.

Next, prove that the two layers are separate. Read your gate credential from the file instead of typing it, so it never lands in your shell history:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-) printf 'gate credential, port 80 -> HTTP ' curl -s -o /dev/null -w '%{http_code}\n' -m 15 -u "$GU:$GP" http://127.0.0.1/ printf 'gate credential, Scrapyd port 6800 -> HTTP ' curl -s -o /dev/null -w '%{http_code}\n' -m 15 -u "$GU:$GP" http://127.0.0.1:6800/daemonstatus.json printf 'no credential, Scrapyd port 6800 -> HTTP ' curl -s -o /dev/null -w '%{http_code}\n' -m 15 http://127.0.0.1:6800/daemonstatus.json

The first request returns 200. The second and third both return 401. Your credential opens the front door, Scrapyd itself refuses it, and Scrapyd also refuses an anonymous caller on its own port.

This is the two-layer design at work. A spider you deploy cannot use any credential it finds to reach anything beyond what it could already reach.

Step 7 - Open the web console

Browse to http://<vm-public-ip>/ and sign in with SCRAPYD_USERNAME and SCRAPYD_PASSWORD from Step 5. The console lists the deployed projects and links to the Jobs, Items and Logs views.

Scrapyd's console is deliberately read-only, as its own home page says. You deploy projects and schedule crawls through the JSON API, which the next steps use.

Step 8 - Schedule your first crawl

The image ships a sac_demo project. Its catalogue spider crawls a static sample catalogue that this VM serves to itself on 127.0.0.1:6810. The demo is entirely self-contained. Scheduling it sends no traffic to any third party and depends on no external website.

Run this block to list the projects and spiders and to schedule a crawl:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-)
curl -s -u "$GU:$GP" http://127.0.0.1/listprojects.json; echo curl -s -u "$GU:$GP" 'http://127.0.0.1/listspiders.json?project=sac_demo'; echo
JOB=$(curl -s -u "$GU:$GP" http://127.0.0.1/schedule.json \
  -d project=sac_demo -d spider=catalogue \
  | /opt/scrapyd/venv/bin/python -c 'import json,sys; print(json.load(sys.stdin)["jobid"])')
echo "scheduled job: $JOB"

schedule.json returns immediately with a job ID, and the crawl runs in the background. Poll listjobs.json until the job leaves pending and running:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-) for i in $(seq 1 40); do
BUSY=$(curl -s -m 20 -u "$GU:$GP" 'http://127.0.0.1/listjobs.json?project=sac_demo' \
  | /opt/scrapyd/venv/bin/python -c 'import json,sys; d=json.load(sys.stdin); print(len(d["pending"])+len(d["running"]))') [ "$BUSY" = "0" ] && break sleep 3 done curl -s -u "$GU:$GP" 'http://127.0.0.1/listjobs.json?project=sac_demo' \
  | /opt/scrapyd/venv/bin/python -m json.tool | head -30

Step 9 - Read the crawl log and the scraped items

Every job writes two files: a Scrapy log and a JSON Lines feed of whatever the spider yielded. Both are named after the job ID, and both are served over the same authenticated API.

LOG=$(ls -t /var/lib/scrapyd/logs/sac_demo/catalogue/*.log | head -1)
IT=$(ls -t /var/lib/scrapyd/items/sac_demo/catalogue/*.jl | head -1) printf 'log: %s\n' "$(basename "$LOG")" grep -oE "'(item_scraped_count|response_received_count|finish_reason|robotstxt/request_count)': [^,}]+" "$LOG" printf 'items: %s (%s records)\n' "$(basename "$IT")" "$(grep -c . "$IT")" head -2 "$IT" | cut -c1-150

item_scraped_count is 12 and finish_reason is finished. Together, these show that the crawl worked. robotstxt/request_count is 1, which shows that the spider fetched the target's robots.txt before crawling it, because the demo project sets ROBOTSTXT_OBEY = True.

The console shows the same information. The Jobs view lists the finished crawl with its runtime and links straight to its log and its items.

The Items link shows the records that the spider extracted, one JSON object per line, ready to pipe into whatever consumes them.

The Log link shows Scrapy's own account of the run. Read the statistics block at the end. It lists request and response counts, the item count, the feed export destination and the finish reason.

Step 10 - Fetch the items with the API instead of the browser

Everything the console shows is also available as a plain HTTP fetch, which is how you collect results from a pipeline:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-)
IT=$(ls -t /var/lib/scrapyd/items/sac_demo/catalogue/*.jl | head -1)
JOB=$(basename "$IT" .jl)
curl -s -u "$GU:$GP" "http://127.0.0.1/items/sac_demo/catalogue/$JOB.jl" \
  | /opt/scrapyd/venv/bin/python -c 'import json,sys; [print(r["sku"], r["price"], r["title"]) for r in map(json.loads, sys.stdin)]' printf 'without a credential -> HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 15 "http://127.0.0.1/items/sac_demo/catalogue/$JOB.jl"

Your scraped data sits behind the same credential as everything else. An anonymous request for it returns 401.

Step 11 - Deploy your own Scrapy project

The demo project's source ships at /usr/share/scrapyd/demo-project. The fastest way to start is to copy it and edit the spider:

cp -r /usr/share/scrapyd/demo-project ~/my-project cd ~/my-project

scrapyd-client is already installed, so you can deploy from this VM. Add your credential to the [deploy] section of scrapy.cfg, where scrapyd-deploy supports username and password. Then deploy:

[deploy] url = http://127.0.0.1/ project = my_project username = <SCRAPYD_USERNAME> password = <SCRAPYD_PASSWORD>

/opt/scrapyd/venv/bin/scrapyd-deploy default -p my_project

To deploy from your workstation instead, install scrapyd-client there (pip install scrapyd-client), point url at http://<vm-public-ip>/ and use the same credential.

Deploying from anywhere other than the VM itself sends your credential and your project egg across the network. Put TLS in front of nginx first, as described in Maintenance.

If you prefer not to use scrapyd-client, send the egg directly to addversion.json:

curl http://<vm-public-ip>/addversion.json -u <SCRAPYD_USERNAME>:<SCRAPYD_PASSWORD> \
  -F project=my_project -F version=1 -F egg=@dist/my_project-1.0-py3.12.egg

Each upload is a new version of the project. listversions.json shows them all. schedule.json accepts an optional _version to pin a crawl to one version, and Scrapyd uses the highest version by default.

To roll back a bad spider, schedule against the previous version instead of redeploying.

Step 12 - Cancel a crawl, and remove a project

These are the destructive endpoints. cancel.json stops a running or pending job. delversion.json removes one version of a project, and delproject.json removes the project entirely, including every version.

curl http://127.0.0.1/cancel.json -u "$GU:$GP" -d project=my_project -d job=<job-id>
curl http://127.0.0.1/delversion.json -u "$GU:$GP" -d project=my_project -d version=1
curl http://127.0.0.1/delproject.json -u "$GU:$GP" -d project=my_project

Deleting a project removes its eggs. It does not remove the logs and items of crawls that already ran. Those stay under /var/lib/scrapyd/logs and /var/lib/scrapyd/items until you remove them or jobs_to_keep rotates them out.

Maintenance

• Rotate the gate credential: run sudo htpasswd -B /var/lib/scrapyd/.htpasswd scrapyd, then update /root/scrapyd-credentials.txt for your records. No restart is needed, because nginx rereads the file on every request. Do not edit /var/lib/scrapyd/.htpasswd by hand into something that is not a bcrypt hash. The preflight guard deliberately refuses to start nginx on its next restart if you do.
• Rotate Scrapyd's own credential: the password lives in /etc/scrapyd/scrapyd.env, and its base64 counterpart lives in /etc/nginx/sac-scrapyd-upstream-auth.conf. Change both together as a pair, then restart both services. You normally have no reason to change this credential, because it never leaves the VM.
• Raise the concurrency: max_proc_per_cpu = 1 in /etc/scrapyd/scrapyd.conf limits this image to one spider per CPU, which is two on the recommended size. On a larger VM, raise it (upstream's own default is 4) and run sudo systemctl restart scrapyd. Each spider is a separate Python process, so watch memory rather than CPU when you tune it.
• Keep more or fewer job records: finished_to_keep (200 here) caps the job history that the console shows. jobs_to_keep (20) caps how many log and item files are retained per spider. Both settings are in /etc/scrapyd/scrapyd.conf.
• Add TLS: HTTP Basic Auth sends your credential in every request, so anything beyond a private network needs TLS. Put your own domain in front of nginx, terminate TLS with a certificate from your CA or Let's Encrypt, and then close port 80 at the NSG. Until you do, scope the NSG rule for port 80 to your own address range instead of the internet.
• Where the data lives: eggs in /var/lib/scrapyd/eggs, crawl logs in /var/lib/scrapyd/logs, scraped items in /var/lib/scrapyd/items, and the spider queues plus job history in /var/lib/scrapyd/dbs. The scrapyd account owns the whole tree, and it is not world-readable, because a crawl log and an items feed record what you crawled and what you took. Back up eggs and items. The rest is reproducible.
• Crawl responsibly: this appliance runs crawlers you write, and nothing is crawled unless you schedule it. Only crawl sites you own or are permitted to crawl, respect robots.txt and rate limits, and identify your crawler with a USER_AGENT that gives a contact point. The full statement ships in the image at /usr/share/scrapyd/AUTHORISED-USE. The demo project's settings are a deliberately polite starting point.
• Updates: this image pins Scrapyd to 1.6.0 and Scrapy to 2.17.0 for reproducibility. It also pins setuptools below 81 on purpose. Scrapyd's egg loader imports pkg_resources, which setuptools 81 removed, so upgrading setuptools inside the virtualenv would break every deployed project. SecureAnyCloud publishes refreshed images. For security patches, apply OS updates with sudo apt update && sudo apt upgrade.

Support

SecureAnyCloud maintains this image and provides 24/7 support. For help deploying or operating Scrapyd on Azure, contact SecureAnyCloud support through the Azure Marketplace listing.

Deploy on Azure

Launch Scrapyd on Ubuntu 24.04 LTS on Azure with 24/7 support from SecureAnyCloud.

View this image

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com