Scrapyd 1.6.0

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Scrapyd 1.6.0 on Ubuntu 24.04 as the software version, choose your Region, then choose Launch through EC2.

Select m5.large, your key pair and a security group as described above.

To launch with the AWS CLI instead, replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=scrapyd}]'

First boot takes about a minute. It generates the instance's two credentials and its certificate, loads the egress fence and places the demo project. Then it starts Scrapyd and nginx.

Connect over SSH with these values:

Version: Scrapyd 1.6.0 on Ubuntu 24.04; SSH user: ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

Check the service itself, not just the unit state. Run these checks in order: the units, the health endpoint through nginx on HTTPS, and an anonymous API call that must be refused:

systemctl is-active scrapyd-egress-guard scrapyd-firstboot scrapyd nginx
H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt) sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" "$@"; }
C=$(sd -o /dev/null -w '%{http_code}' "https://$H/healthz")
A=$(sd -o /dev/null -w '%{http_code}' "https://$H/daemonstatus.json")
echo "GET /healthz -> HTTP $C"
echo "anonymous GET /daemonstatus.json -> HTTP $A" [ "$C" = 200 ] && [ "$A" = 401 ] && echo SCRAPYD_OK

active active active active GET /healthz -> HTTP 200 anonymous GET /daemonstatus.json -> HTTP 401 SCRAPYD_OK

The certificate check passes because --cacert names your instance's own certificate. curl verifies it exactly as a browser would once you trust it.

Port 80 redirects to HTTPS. Scrapyd and the demo catalogue answer on loopback only, and only for the accounts that need them. Any other user on the instance is refused, and nothing but 22, 80 and 443 listens on the network:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt)
R=$(curl -s -o /dev/null -w '%{http_code}' --resolve "$H:80:127.0.0.1" "http://$H/")
echo "http://$H/ -> HTTP $R (redirect to https)"
sudo ss -Htln | awk '{print $4}' | sort -u | tr '\n' ' '; echo
S=$(sudo -u nobody curl -s -o /dev/null -m 4 -w '%{http_code}' http://127.0.0.1:6800/daemonstatus.json || true)
D=$(sudo -u nobody curl -s -o /dev/null -m 4 -w '%{http_code}' http://127.0.0.1:6810/ || true)
echo "another user to Scrapyd :6800 -> HTTP $S, to the demo catalogue :6810 -> HTTP $D" [ "$R" = 301 ] && [ "$S" = 000 ] && [ "$D" = 000 ] && echo FENCED_OK

http://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ -> HTTP 301 (redirect to https) 0.0.0.0:22 0.0.0.0:443 0.0.0.0:80 127.0.0.1:6800 127.0.0.1:6810 127.0.0.53%lo:53 127.0.0.54:53 [::]:22 [::]:443 [::]:80 another user to Scrapyd :6800 -> HTTP 000, to the demo catalogue :6810 -> HTTP 000 FENCED_OK

HTTP 000 means the connection was refused: nothing reaches Scrapyd except through nginx.

sudo cat /root/scrapyd-credentials.txt

The file is readable by root only. Your values differ:

# sac Scrapyd - this instance's credentials (written at first boot; root 0600). Keep them private. # Web console and JSON API (sign in with the user name and password below): SCRAPYD_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ SCRAPYD_USERNAME=scrapyd SCRAPYD_PASSWORD=<32 characters generated on your instance> # The certificate is self-signed for this instance; compare this fingerprint with the one your browser shows: SCRAPYD_TLS_SHA256_FINGERPRINT=<unique to your instance> # Deploy a project from your workstation (pip install scrapyd-client): see the user guide for scrapy.cfg. # ONLY CRAWL SITES YOU OWN OR ARE PERMITTED TO CRAWL - /usr/share/scrapyd/AUTHORISED-USE

SCRAPYD_URL uses the instance's public DNS name from EC2 instance metadata, and the certificate also names the public and private IP addresses. To serve your own DNS name, see Step 10.

Store the password in your password manager. It was generated on your instance, and no other instance has it.

The commands in the rest of this guide read the password from that file into a curl configuration file that only you can read. The password therefore never appears on a command line or in your shell history:

( umask 077;
sudo sed -n 's/^SCRAPYD_PASSWORD=\(.*\)$/user = "scrapyd:\1"/p' /root/scrapyd-credentials.txt > ~/.scrapyd-curlrc ) stat -c '%n %a' ~/.scrapyd-curlrc
sudo stat -c '%n %a %U:%G' /root/scrapyd-credentials.txt /etc/nginx/scrapyd.htpasswd /etc/scrapyd/scrapyd.env [ "$(stat -c %a ~/.scrapyd-curlrc)" = 600 ] && echo CURLRC_OK

/home/ubuntu/.scrapyd-curlrc 600 /root/scrapyd-credentials.txt 600 root:root /etc/nginx/scrapyd.htpasswd 640 root:www-data /etc/scrapyd/scrapyd.env 600 root:root CURLRC_OK

Open SCRAPYD_URL in your browser. The certificate is self-signed for your instance, so the browser warns once. Compare the fingerprint it shows with SCRAPYD_TLS_SHA256_FINGERPRINT before you accept it. Sign in with SCRAPYD_USERNAME and SCRAPYD_PASSWORD.

The console is deliberately read-only, as its own home page says. It lists projects and jobs and links to every job's log and items. You deploy projects and schedule crawls through the JSON API, which the next steps use.

Every route except the health check is refused without the credential. Scrapyd itself refuses the credential you use, and Scrapyd's own configuration carries no credential at all:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt) sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" "$@"; } OK=1 for P in / /jobs /listprojects.json /daemonstatus.json; do
C=$(sd -o /dev/null -w '%{http_code}' "https://$H$P"); echo "anonymous GET $P -> HTTP $C"; [ "$C" = 401 ] || OK=0 done for P in /schedule.json /addversion.json; do
C=$(sd -o /dev/null -w '%{http_code}' -d project=sac_demo -d spider=catalogue "https://$H$P"); echo "anonymous POST $P -> HTTP $C"; [ "$C" = 401 ] || OK=0 done
G=$(sd -K ~/.scrapyd-curlrc -o /dev/null -w '%{http_code}' "https://$H/daemonstatus.json")
D=$(sudo curl -s -K ~/.scrapyd-curlrc -o /dev/null -w '%{http_code}' http://127.0.0.1:6800/daemonstatus.json)
N=$(sudo curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:6800/daemonstatus.json)
echo "your credential through nginx -> HTTP $G" echo "your credential straight to Scrapyd :6800 -> HTTP $D" echo "no credential straight to Scrapyd :6800 -> HTTP $N"
sudo grep -E '^(bind_address|http_port|username|password) ' /etc/scrapyd/scrapyd.conf [ "$OK" = 1 ] && [ "$G" = 200 ] && [ "$D" = 401 ] && [ "$N" = 401 ] && echo ACCESS_MODEL_OK

anonymous GET / -> HTTP 401 anonymous GET /jobs -> HTTP 401 anonymous GET /listprojects.json -> HTTP 401 anonymous GET /daemonstatus.json -> HTTP 401 anonymous POST /schedule.json -> HTTP 401 anonymous POST /addversion.json -> HTTP 401 your credential through nginx -> HTTP 200 your credential straight to Scrapyd :6800 -> HTTP 401 no credential straight to Scrapyd :6800 -> HTTP 401 bind_address = 127.0.0.1 http_port = 6800 username = password = ACCESS_MODEL_OK

• Two layers. nginx accepts your credential and replaces it with Scrapyd's own, which lives in /etc/scrapyd/scrapyd.env (root only) and in nginx's /etc/nginx/scrapyd-upstream-auth.conf. Anything that can talk to Scrapyd can run code as the scrapyd account, so a deployed project can learn Scrapyd's credential. It still cannot read /etc/nginx/scrapyd.htpasswd, the TLS key or /root, so the credential you sign in with stays safe.
• No stored default. username and password in /etc/scrapyd/scrapyd.conf are empty on purpose, because upstream treats empty as "authentication off". The real values exist only in the file that first boot writes, and scrapyd.service refuses to start if that file is missing, empty or malformed. nginx refuses to start in the same way if its credential file is missing, unreadable, empty or malformed.

The image ships a sac_demo project whose catalogue spider crawls a static sample catalogue that this instance serves on 127.0.0.1:6810. Scheduling it sends no traffic to any third party:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt) sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" -K ~/.scrapyd-curlrc "$@"; } sd "https://$H/listprojects.json"; echo sd "https://$H/listspiders.json?project=sac_demo"; echo
JOB=$(sd "https://$H/schedule.json" -d project=sac_demo -d spider=catalogue | python3 -c 'import json,sys; print(json.load(sys.stdin)["jobid"])')
echo "scheduled job: $JOB" for i in $(seq 1 40); do
ST=$(sd "https://$H/status.json?job=$JOB" | python3 -c 'import json,sys; print(json.load(sys.stdin)["currstate"])') [ "$ST" = finished ] && break sleep 3 done echo "job state: $ST" echo "$JOB" > ~/.scrapyd-last-job [ "$ST" = finished ] && echo CRAWL_OK

{"projects": ["sac_demo"], "status": "ok", "node_name": "ip-<private-ip-dashed>"} {"spiders": ["catalogue"], "status": "ok", "node_name": "ip-<private-ip-dashed>"} scheduled job: <job-id> job state: finished CRAWL_OK

schedule.json returns at once with a job id, and the crawl runs in the background. status.json reports pending, running and then finished.

Every job writes a Scrapy log and a JSON Lines file of whatever the spider yielded. Both are named after the job id and served behind the same credential:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt) sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" "$@"; }
JOB=$(cat ~/.scrapyd-last-job) sd -K ~/.scrapyd-curlrc "https://$H/logs/sac_demo/catalogue/$JOB.log" > /tmp/scrapyd-job.log sd -K ~/.scrapyd-curlrc "https://$H/items/sac_demo/catalogue/$JOB.jl" > /tmp/scrapyd-job.jl grep -oE "'(item_scraped_count|response_received_count|finish_reason|robotstxt/request_count)': [^,}]+" /tmp/scrapyd-job.log
N=$(python3 -c 'import json,sys; print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("sku")))' /tmp/scrapyd-job.jl)
echo "items: $N records" python3 -c 'import json,sys; [print(r["sku"], r["price"], r["currency"], r["title"]) for r in map(json.loads, list(open(sys.argv[1]))[:3])]' /tmp/scrapyd-job.jl
A=$(sd -o /dev/null -w '%{http_code}' "https://$H/items/sac_demo/catalogue/$JOB.jl")
echo "the same items without a credential -> HTTP $A" rm -f /tmp/scrapyd-job.log /tmp/scrapyd-job.jl [ "$N" = 12 ] && [ "$A" = 401 ] && echo ITEMS_OK

'finish_reason': 'finished' 'item_scraped_count': 12 'response_received_count': 17 'robotstxt/request_count': 1 items: 12 records CI-3004 84.00 GBP Windshield Kit CI-3003 197.50 GBP Boom Pole Carbon CI-3002 289.99 GBP Reference Monitor 5in the same items without a credential -> HTTP 401 ITEMS_OK

item_scraped_count of 12 and finish_reason of finished tell you the crawl worked.

robotstxt/request_count of 1 shows the spider fetched robots.txt first, because the demo project sets ROBOTSTXT_OBEY = True. Logs and items are served as plain text that a browser will not run as a page.

Spiders run as the scrapyd account under an nftables egress fence. Public addresses are allowed. The instance metadata service, loopback, private and reserved ranges, your VPC and IPv6 are refused at connect, whatever URL or redirect the spider follows:

probe() {
sudo -u scrapyd python3 -I -c 'import socket,sys s = socket.socket(); s.settimeout(5) try: s.connect((sys.argv[1], int(sys.argv[2]))); print("OPEN") except OSError as e: print("REFUSED", type(e).__name__)' "$1" "$2"; }
M=$(probe 169.254.169.254 80); echo "spider -> instance metadata 169.254.169.254:80 : $M"
L=$(probe 127.0.0.1 6800); echo "spider -> Scrapyd's own port 127.0.0.1:6800 : $L"
V=$(probe "$(hostname -I | awk '{print $1}')" 22); echo "spider -> this instance's private address :22 : $V"
P=$(probe 10.0.0.1 80); echo "spider -> a private address 10.0.0.1:80 : $P"
W=$(probe example.com 443); echo "spider -> a public site example.com:443 : $W" case "$M$L$V$P" in *OPEN*) false ;; esac && [ "$W" = OPEN ] && echo EGRESS_FENCE_OK

spider -> instance metadata 169.254.169.254:80 : REFUSED ConnectionRefusedError spider -> Scrapyd's own port 127.0.0.1:6800 : REFUSED ConnectionRefusedError spider -> this instance's private address :22 : REFUSED ConnectionRefusedError spider -> a private address 10.0.0.1:80 : REFUSED ConnectionRefusedError spider -> a public site example.com:443 : OPEN EGRESS_FENCE_OK

To let your spiders crawl a site inside your VPC or on your private network, list its address range in /etc/sac/scrapyd-egress-allow, one IPv4 or IPv6 CIDR per line, and reload the fence. A range that touches the metadata service, link-local or loopback addresses is rejected, and the running fence is kept:

echo '10.0.5.25/32' | sudo tee -a /etc/sac/scrapyd-egress-allow
sudo systemctl reload scrapyd-egress-guard

The instance's own addresses are reached over loopback, so services on the instance itself stay refused to spiders whatever the file lists.

The demo project's source is at /usr/share/scrapyd/demo-project; copy it as a starting point and edit the spider. Its settings.py is a deliberately polite template. It obeys robots.txt, identifies itself with a USER_AGENT, throttles itself and switches off Scrapy's telnet and remote-control consoles.

scrapyd-client is installed, so you can deploy from the instance. Add the URL and credential to the [deploy] section of scrapy.cfg:

[deploy] url = https://<SCRAPYD_URL host name>/ project = my_project username = scrapyd password = <SCRAPYD_PASSWORD>

The certificate is your instance's own, so point Python at it when you deploy:

cp -r /usr/share/scrapyd/demo-project ~/my-project && cd ~/my-project SSL_CERT_FILE=/etc/nginx/ssl/scrapyd.crt /opt/scrapyd/venv/bin/scrapyd-deploy default -p my_project

To deploy from your workstation, run pip install scrapyd-client there. Copy /etc/nginx/ssl/scrapyd.crt from the instance (it is public) and use the same SSL_CERT_FILE setting. Alternatively, give the instance your own DNS name and certificate (below).

Without scrapyd-client, addversion.json takes an egg directly:

curl --cacert scrapyd.crt -u scrapyd:<SCRAPYD_PASSWORD> https://<SCRAPYD_URL host name>/addversion.json \
  -F project=my_project -F version=1 -F egg=@dist/my_project-1.0-py3.12.egg

Each upload is a new version of the project. listversions.json lists them. schedule.json accepts _version to pin a crawl to one, and Scrapyd uses the newest by default. To roll back a bad spider, schedule the previous version.

cancel.json stops a job, delversion.json removes one version and delproject.json removes a project. Logs and items of crawls that already ran stay until jobs_to_keep rotates them out.

Your own DNS name and certificate. Point a DNS name at the instance, then let the image serve it. This mints a new certificate for the name, updates SCRAPYD_URL and reloads nginx:

sudo scrapyd-set-hostname <your-domain>

To use a certificate from your own CA, replace /etc/nginx/ssl/scrapyd.crt and /etc/nginx/ssl/scrapyd.key (keep the key 0600 root) and run sudo systemctl reload nginx.

/opt/scrapyd/venv/bin/python -c 'import scrapyd, scrapy; print("scrapyd", scrapyd.__version__, "| scrapy", scrapy.__version__)' /opt/scrapyd/venv/bin/pip list --format=freeze 2>/dev/null | grep -icE '==' | sed 's/^/python distributions: /' ls /usr/share/doc/scrapyd/ | head -20 head -5 /usr/share/scrapyd/AUTHORISED-USE /opt/scrapyd/venv/bin/python -c 'import scrapyd' && echo LICENCE_OK

scrapyd 1.6.0 | scrapy 2.20.0 python distributions: 54 LICENSE.scrapy LICENSE.scrapyd LICENSE.scrapyd_client requirements.lock wheels-sha256.txt sac Scrapyd image - authorised use ======================================= This image runs web crawlers that YOU write and deploy. sac supplies the runtime; the crawling is entirely under your control and is your responsibility. LICENCE_OK

Components and licences:

Scrapyd 1.6.0: BSD 3-Clause.

Scrapy 2.20.0: BSD 3-Clause.

scrapyd-client 2.0.3: BSD 3-Clause.

Twisted, lxml, parsel, w3lib and the other 51 Python dependencies pinned in /usr/share/doc/scrapyd/requirements.lock: permissive licences (MIT, BSD, Apache 2.0, PSF, ZPL 2.1) plus certifi under MPL 2.0, unmodified. Each is listed in /usr/share/scrapyd/dependency-licences.txt, and its text ships in its .dist-info directory.

nginx, nftables, Python 3.12 and Ubuntu 24.04 LTS: Ubuntu packages, licensed per package, in /usr/share/doc.

There is no Java runtime, no compiler and no optional cloud SDK on the image. The only change to any installed package is that the published private keys Twisted ships as test fixtures for its own test suite are removed, so no private key material is baked into the image. /usr/share/scrapyd/THIRD-PARTY-NOTICES lists every bundled component, and /usr/share/doc/scrapyd/wheels-sha256.txt records the hash of every installed wheel.