Crawl4AI

Search the Azure Marketplace for Crawl4AI on Ubuntu 24.04 and choose your VM size.

Attach a network security group (NSG) that allows TCP 22 (SSH) from your management network, and TCP 80 and 443 (the crawl API) from the networks and applications that will call it. In production, front port 80 with TLS (see Enabling HTTPS below).

# Accept the Marketplace terms once per subscription (required before the first CLI deploy)
az vm image terms accept --urn sac1647283583153:crawl4ai-ubuntu-24-04:default:latest
az vm create \
  --resource-group my-rg \
  --name crawl4ai-01 \
  --image sac1647283583153:crawl4ai-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
# Open the crawl API port to the callers that need it
az vm open-port --resource-group my-rg --name crawl4ai-01 --port 80

First boot takes a few minutes. The VM finishes its per-VM setup in the background, generating its own credentials and starting the services. If you connect as soon as the VM shows as running, wait for that setup to complete before continuing by running sudo cloud-init status --wait.

It prints status: done when the VM is ready. The credentials and services described below exist only after that.

SSH in as azureuser. First boot creates the container, generates your per-VM API token and brings the stack up. Confirm that the services are healthy:

systemctl is-active docker nginx
sudo docker ps --format '{{.Names}}\t{{.Status}}' curl -s -o /dev/null -w 'healthz: HTTP %{http_code}\n' http://127.0.0.1/healthz

Expected output:

active active crawl4ai Up 3 minutes (healthy) healthz: HTTP 200

Crawl4AI listens only on host loopback 127.0.0.1:11235, and nginx fronts it on port 80.

Read your unique API token from the root-only info note:

sudo cat /root/crawl4ai-info.txt

The file (mode 0600, owned by root) contains your CRAWL4AI_URL and CRAWL4AI_TOKEN. The token is unique to this VM and was generated on first boot. Keep it secret.

With a Standard SKU public IP (the Azure default), CRAWL4AI_URL can show the VM's private address, for example http://10.0.0.4/. From outside the VNet, call the API at your VM's public IP or DNS name instead.

Crawl4AI enforces your API token on every crawl request. A request with no token or a wrong token is rejected with HTTP 401, and only your per-VM token is accepted. Verify this from the VM:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
echo "no token -> HTTP $(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1/crawl -H 'Content-Type: application/json' -d '{"urls":["https://example.com"]}')"
echo "wrong token-> HTTP $(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1/crawl -H 'Authorization: Bearer wrong' -H 'Content-Type: application/json' -d '{"urls":["https://example.com"]}')"
echo "your token -> HTTP $(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1/md -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"url":"https://example.com"}')"

Expected output:

no token -> HTTP 401 wrong token-> HTTP 401 your token -> HTTP 200

The /md endpoint turns a page into clean Markdown in a single call. Pass your token as a bearer header:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
curl -s -X POST http://127.0.0.1/md \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | jq -r .markdown

Expected output is the Markdown of the page, starting with its text. example.com changes its wording from time to time, so yours may differ:

This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.

From a remote machine, replace 127.0.0.1 with your VM's address and use the same token. In production, put TLS in front (see Enabling HTTPS below) and call https://your-domain/md.

The /crawl endpoint returns the full result set, including the raw HTML, links, media and several Markdown variants. Use it when you need more than plain Markdown:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
curl -s -X POST http://127.0.0.1/crawl \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"cache_mode":"bypass"}}}' \
  | jq -r '.results[0] | "url: " + .url, "status: " + (.status_code|tostring), "links: " + (.links.internal|length|tostring), "markdown chars: " + (.markdown.raw_markdown|length|tostring)'

Expected output:

url: https://example.com status: 200 links: 0 markdown chars: 953

The character count follows the page's current content, so expect a different number for other pages or after example.com changes.

Crawl4AI also exposes /html (sanitised HTML), /screenshot (a PNG of the rendered page), /pdf (a PDF), /execute_js (run JavaScript on the page), and streaming and job endpoints. See the Crawl4AI documentation for the full API and its extraction options.

The container image is pinned at build time, and first boot pulls and recreates it cleanly, so every VM starts from a known-good image. The container runs with a restart policy, so it survives reboots.

Check the container's restart policy and whether any packages are held back:

sudo docker inspect -f 'image: {{index .Config.Image}} restart: {{.HostConfig.RestartPolicy.Name}}' crawl4ai systemctl is-enabled crawl4ai-firstboot.service docker nginx apt-mark showhold

Expected output:

image: unclecode/crawl4ai:0.9.2 restart: unless-stopped enabled enabled enabled

The empty last line from apt-mark showhold confirms that no security updates are held back. The Ubuntu base is patched at build time and ships with unattended security upgrades enabled.

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images, choose AWS Marketplace AMIs and search for Crawl4AI. Select the SecureAnyCloud listing, choose Select, then choose Continue on the subscription summary.

Choose an instance type. Crawl4AI drives a headless Chromium browser, which is memory-hungry. The recommended size is m5.large (2 vCPU, 8 GiB) for light to moderate crawling. Increase the size for higher concurrency.

Select your key pair and attach a security group that opens port 22 from your management network and ports 80 and 443 from the networks your crawl clients use. Then launch the instance.

To launch from the AWS CLI instead, subscribe to the product in the Marketplace console first. Then run the following command, substituting your own AMI id, key pair, subnet and security group:

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <your-key-pair> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --metadata-options "HttpTokens=required" \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=crawl4ai}]'

HttpTokens=required enforces IMDSv2, which is part of the SSRF protection described in Step 7.

Connect with the private key for the key pair you selected at launch. On Ubuntu 24.04 LTS, the SSH login user is ubuntu:

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Crawl4AI requires a bearer token on every crawl request. First boot generated a unique token on this instance and wrote it to a root-only file. Read the file with:

sudo cat /root/crawl4ai-info.txt

The file shows your CRAWL4AI_URL (the API base URL for this instance) and your CRAWL4AI_TOKEN. Treat the token as a secret.

If the file contains only a short holding note, first boot has not finished yet. Wait a minute and read it again.

Confirm that Docker, nginx and the Crawl4AI container are running and that the open health probe answers. Run this on the instance:

systemctl is-active docker.service nginx.service docker ps --format '{{.Names}} {{.Image}} {{.Status}}' curl -s -o /dev/null -w 'healthz: HTTP %{http_code}\n' http://127.0.0.1/healthz

You should see active for both services, the crawl4ai container reported as healthy on unclecode/crawl4ai:0.9.2, and healthz: HTTP 200.

The token gate refuses any crawl request that does not present a valid token. Confirm it locally:

curl -s -o /dev/null -w 'unauthenticated /crawl: HTTP %{http_code}\n' \
  -X POST http://127.0.0.1/crawl \
  -H 'Content-Type: application/json' \
  -d '{"urls":["raw:<h1>test</h1>"]}'

The 401 response shows that the endpoint is not open.

This command runs a real crawl on the instance. It reads the token straight from the info file, so nothing is echoed. It uses Crawl4AI's raw: scheme, which converts supplied HTML with no network fetch, so it works even without outbound internet access:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
curl -s -X POST http://127.0.0.1/crawl \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["raw:<h1>Hello Crawl4AI</h1><p>clean markdown</p>"],"crawler_config":{"type":"CrawlerRunConfig","params":{"cache_mode":"bypass"}}}'

The response is JSON with "success": true and a markdown field containing # Hello Crawl4AI.

To crawl a real public URL, pass its address in urls. From your own workstation, call the instance's public address with your token:

curl -X POST http://<public-ip>/crawl \
  -H "Authorization: Bearer <your-api-token>" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"]}'

Crawl4AI returns clean Markdown extracted from the rendered page, ready to feed into a retrieval-augmented generation pipeline, an AI agent or a dataset.

Crawl4AI exposes several endpoints through the same authenticated proxy. Send the same Authorization: Bearer <your-api-token> header to each:

• POST /crawl: crawl one or more URLs and return Markdown and extracted data
• POST /crawl/stream: stream results as each URL completes
• POST /html: return the cleaned HTML of a page
• POST /screenshot: capture a full-page screenshot
• POST /pdf: render a page to PDF
• GET /playground: a small built-in developer console for trying requests interactively, behind the token gate
• GET /health: the crawler's own health endpoint

Pass crawl configuration (cache mode, extraction strategy, JavaScript to execute on the page, and more) in the crawler_config object, following Crawl4AI's CrawlerRunConfig schema.

This appliance keeps the crawler safe by default, but a crawler is a powerful tool and you remain responsible for the network context you run it in.

• Authentication. Every crawl request needs the per-instance bearer token. The crawler binds to loopback only, and nginx on port 80 is the sole entry point. There are no shared or default credentials.
• SSRF protection. Crawl4AI 0.9.2 blocks crawls of loopback, link-local and internal (private) address ranges at the application layer, refusing them with URL blocked (SSRF protection). This image also enforces IMDSv2, so even if a crawl reached the instance metadata service at 169.254.169.254, it could not retrieve instance credentials. These are two independent defenses against metadata theft.
• Restrict the network. Crawl4AI fetches any public URL that a caller with the token asks for, so run it only in a network context you trust. Restrict inbound ports 80 and 443 in the security group to your own CIDR ranges instead of leaving them open to the whole internet.
• Add TLS. For production, add a domain name and a TLS certificate on port 443 so the token is never sent in clear text. Front the appliance with a certificate from your own CA, or install one with certbot for your domain. Then point clients at https://your-domain/crawl.

To rotate the token, recreate the container with a new value and record it. Connect with ssh ubuntu@<public-ip>, then run on the instance:

# rotate the per-instance API token (run on the instance at <public-ip>)
NEW_TOKEN=$(openssl rand -hex 32)
sudo docker rm -f crawl4ai
sudo docker run -d --name crawl4ai --restart unless-stopped --shm-size=1g --network host \
  -e CRAWL4AI_API_TOKEN="$NEW_TOKEN" -e GUNICORN_BIND=127.0.0.1:11235 unclecode/crawl4ai:0.9.2

Store $NEW_TOKEN securely and update /root/crawl4ai-info.txt. Switch all existing clients to the new token.

The Crawl4AI image is pinned to unclecode/crawl4ai:0.9.2 for reproducibility. To move to a newer release, pull the new tag and recreate the container with your token, using the same docker run form as in Step 8.

Both docker.service and nginx.service are managed by systemd and restart automatically. Read container logs with sudo docker logs crawl4ai and service logs with journalctl -u nginx.