Toxiproxy 2.13.0

In the AWS Marketplace listing, choose Continue to Subscribe and then Continue to Configuration. Pick Toxiproxy 2.13.0 on Ubuntu 24.04 as the software version, choose your Region, and choose Launch through EC2. Choose m5.large, your key pair, a security group as described in Prerequisites, and a subnet.

Alternatively, launch with the AWS CLI. Replace the values in angle brackets:

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=toxiproxy}]'

Version SSH user

Toxiproxy 2.13.0 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

First boot takes under a minute. It generates the certificate and the API token, loads the outbound fence, and then starts Toxiproxy and nginx. Check that everything is running:

systemctl is-active toxiproxy nginx toxiproxy-firstboot toxiproxy-egress-guard \
  && sudo test -f /var/lib/sac/data/toxiproxy-firstboot.done && echo FIRSTBOOT_DONE

active active active active FIRSTBOOT_DONE

Only SSH and nginx listen on the network. Toxiproxy's API (8474), the demo proxy (20000) and the demo page (8081) listen on loopback. Confirm the listeners:

sudo ss -lntp | awk 'NR>1 && $4 !~ /^127\.0\.0\.5[34]/ {print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u | tee /dev/stderr \
  | awk '$1 !~ /^127\.0\.0\.1:/ {sub(/.*:/, "", $1); print $1}' | sort -un | xargs | grep -qx '22 80 443' && echo "network listeners: 22 80 443"

0.0.0.0:22 users:(("sshd"),("systemd")) 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:20000 users:(("toxiproxy-serve")) 127.0.0.1:8081 users:(("nginx"),("nginx"),("nginx")) 127.0.0.1:8474 users:(("toxiproxy-serve")) [::]:22 users:(("sshd"),("systemd")) [::]:443 users:(("nginx"),("nginx"),("nginx")) [::]:80 users:(("nginx"),("nginx"),("nginx")) network listeners: 22 80 443

Run sudo cat /var/lib/sac/data/toxiproxy-credentials.txt to see your URL and token. Only root can read the file. It holds the following settings, all generated for this instance. This command lists their names without printing the token:

sudo sed -n 's/^\(TOXIPROXY_[A-Z0-9_]*\)=.*/\1/p' /var/lib/sac/data/toxiproxy-credentials.txt | tee /dev/stderr | grep -qx TOXIPROXY_AUTH_TOKEN && echo "token present"

TOXIPROXY_URL TOXIPROXY_HOST TOXIPROXY_AUTH_TOKEN TOXIPROXY_TLS_CERT TOXIPROXY_TLS_SHA256_FINGERPRINT TOXIPROXY_LOCAL_URL TOXIPROXY_DEMO_PROXY TOXIPROXY_PROXY_PORT_RANGE token present

TOXIPROXY_URL is https://<public-ip>. Send TOXIPROXY_AUTH_TOKEN as Authorization: Bearer <token> with every request. The instance's private IP address and its EC2 DNS names are served too, so clients inside your VPC can use them.

On the instance itself, the API is also at TOXIPROXY_LOCAL_URL (http://127.0.0.1:8474), and it needs the same token there.

The certificate is self-signed and generated on your instance. When a client reports the certificate, compare its SHA-256 fingerprint with TOXIPROXY_TLS_SHA256_FINGERPRINT before you trust the connection. To use a CA-issued certificate instead, see Step 11.

toxiproxy-cli reads the API address from TOXIPROXY_URL and the token from TOXIPROXY_AUTH_TOKEN. On the instance, point it at the HTTPS front door and trust the instance's own certificate (SSL_CERT_FILE). The following lines read the values from the credentials file without printing the token:

export TOXIPROXY_URL="https://$(sudo sed -n 's/^TOXIPROXY_HOST=//p' /var/lib/sac/data/toxiproxy-credentials.txt)"
export TOXIPROXY_AUTH_TOKEN="$(sudo sed -n 's/^TOXIPROXY_AUTH_TOKEN=//p' /var/lib/sac/data/toxiproxy-credentials.txt)"
export SSL_CERT_FILE=/etc/ssl/toxiproxy/toxiproxy.crt toxiproxy-cli list | tee /dev/stderr | grep -qP '^demo\t127\.0\.0\.1:20000\t127\.0\.0\.1:8081\tenabled' && echo "demo proxy listed"

demo 127.0.0.1:20000 127.0.0.1:8081 enabled 0 demo proxy listed

The columns are the proxy name, where it listens, its upstream, whether it is enabled and how many toxics it has. In an interactive terminal, toxiproxy-cli list prints the same information as a table with headings.

The demo proxy forwards 127.0.0.1:20000 to a small page served by nginx. Fetch the page through the proxy, add one second of latency, fetch the page again, and then remove the toxic:

export TOXIPROXY_URL="https://$(sudo sed -n 's/^TOXIPROXY_HOST=//p' /var/lib/sac/data/toxiproxy-credentials.txt)"
export TOXIPROXY_AUTH_TOKEN="$(sudo sed -n 's/^TOXIPROXY_AUTH_TOKEN=//p' /var/lib/sac/data/toxiproxy-credentials.txt)"
export SSL_CERT_FILE=/etc/ssl/toxiproxy/toxiproxy.crt curl -s http://127.0.0.1:20000/ | grep '^sac toxiproxy demo upstream' \
  && toxiproxy-cli toxic add -t latency -a latency=1000 demo \
  && T=$(curl -s -o /dev/null -w '%{time_total}' http://127.0.0.1:20000/) \
  && awk -v t="$T" 'BEGIN { printf "with the latency toxic: %.1f s\n", t; exit !(t >= 1) }' \
  && toxiproxy-cli toxic remove -n latency_downstream demo \
  && T=$(curl -s -o /dev/null -w '%{time_total}' http://127.0.0.1:20000/) \
  && awk -v t="$T" 'BEGIN { printf "toxic removed: %.1f s\n", t; exit !(t < 0.5) }'

sac toxiproxy demo upstream: this text was served by nginx on 127.0.0.1:8081 through the Toxiproxy proxy "demo". Added downstream latency toxic 'latency_downstream' on proxy 'demo' with the latency toxic: 1.0 s Removed toxic 'latency_downstream' on proxy 'demo' toxic removed: 0.0 s

The other toxics work the same way, for example -t bandwidth -a rate=64 (KB/s), -t timeout -a timeout=0 (the connection hangs and no data flows), -t reset_peer -a timeout=0 (the connection is reset), -t slicer -a average_size=64 and -t limit_data -a bytes=1024.

Add -u to apply a toxic to the upstream direction (what your application sends) instead of the downstream direction. Add --tox 0.5 to apply a toxic to half of the connections. toxiproxy-cli toxic lists every option.

Create a proxy for each dependency your application uses. A proxy listens on an address on this instance and forwards to its upstream. The upstream can be anywhere this instance can reach, such as a database or cache in your VPC, or a public API.

This example puts a proxy in front of a public HTTP service and shows that traffic flows. It then makes the proxy reset every connection and deletes the proxy:

export TOXIPROXY_URL="https://$(sudo sed -n 's/^TOXIPROXY_HOST=//p' /var/lib/sac/data/toxiproxy-credentials.txt)"
export TOXIPROXY_AUTH_TOKEN="$(sudo sed -n 's/^TOXIPROXY_AUTH_TOKEN=//p' /var/lib/sac/data/toxiproxy-credentials.txt)"
export SSL_CERT_FILE=/etc/ssl/toxiproxy/toxiproxy.crt toxiproxy-cli create -l 127.0.0.1:20001 -u checkip.amazonaws.com:80 checkip \
  && curl -s -H 'Host: checkip.amazonaws.com' http://127.0.0.1:20001/ | grep -qE '^[0-9.]+$' && echo "through the proxy: your public IP came back" \
  && toxiproxy-cli toxic add -t reset_peer -a timeout=0 checkip \
  && { curl -s -o /dev/null -H 'Host: checkip.amazonaws.com' http://127.0.0.1:20001/; echo "with reset_peer: curl exit code $?"; } | tee /dev/stderr | grep -qx 'with reset_peer: curl exit code 56' \
  && toxiproxy-cli delete checkip

Created new proxy checkip through the proxy: your public IP came back Added downstream reset_peer toxic 'reset_peer_downstream' on proxy 'checkip' with reset_peer: curl exit code 56 Deleted proxy checkip

For a database in your VPC, use the same command with its address, for example toxiproxy-cli create -l 0.0.0.0:20010 -u <your-db-host>:5432 postgres. Your application then connects to port 20010 on this instance instead of to the database.

Proxies you create through the API live until Toxiproxy restarts, which happens on an instance reboot or a token rotation (Step 15). Test suites normally create the proxies they need at the start of a run.

To have proxies that are always there, list them in /etc/toxiproxy/toxiproxy.json, which Toxiproxy loads at every start. Keep the demo entry or remove it as you like:

sudo tee /etc/toxiproxy/toxiproxy.json >/dev/null <<'JSON' [ {"name": "demo", "listen": "127.0.0.1:20000", "upstream": "127.0.0.1:8081", "enabled": true}, {"name": "postgres", "listen": "0.0.0.0:20010", "upstream": "db.<your-domain>:5432", "enabled": true} ] JSON
sudo systemctl restart toxiproxy

A proxy that listens on 127.0.0.1 is reachable from this instance only. When the application under test runs elsewhere, make the proxy listen on all addresses (0.0.0.0:<port>, which also covers IPv6). Then open that port in the instance's security group to your application's addresses only.

The image documents 20000-20099 as the proxy port range. Toxiproxy runs without privileges, so ports below 1024 are not available to it. For example, to let one subnet reach the whole range, run:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --ip-permissions 'IpProtocol=tcp,FromPort=20000,ToPort=20099,IpRanges=[{CidrIp=<your-mgmt-cidr>,Description=toxiproxy proxies}]'

A proxy port is a plain TCP relay to its upstream with no authentication of its own, so never open the range to 0.0.0.0/0.

If only your own workstation needs a proxy, keep it on 127.0.0.1 and reach it over SSH instead. The command ssh -i <your-key.pem> -L 20000:127.0.0.1:20000 ubuntu@<public-ip> makes the demo proxy available on localhost:20000 on your machine.

toxiproxy-cli (from the Toxiproxy releases, version 2.13.0 or later) works from your machine with the same three settings. Copy the certificate once and check its fingerprint against TOXIPROXY_TLS_SHA256_FINGERPRINT. Then run:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/ssl/toxiproxy/toxiproxy.crt ./toxiproxy.crt openssl x509 -in toxiproxy.crt -noout -fingerprint -sha256 export TOXIPROXY_URL=https://<public-ip> export TOXIPROXY_AUTH_TOKEN='<TOXIPROXY_AUTH_TOKEN>' SSL_CERT_FILE=./toxiproxy.crt toxiproxy-cli list

SSL_CERT_FILE is honored on Linux. On macOS and Windows, Go uses the system trust store instead. Either use a CA-issued certificate (Step 11) or open an SSH tunnel and use the loopback API, which needs the same token. Open the tunnel with ssh -i <your-key.pem> -N -L 8474:127.0.0.1:8474 ubuntu@<public-ip>, and then run TOXIPROXY_URL=http://127.0.0.1:8474 toxiproxy-cli list.

Any HTTP client can use the API directly. With curl, keep the token out of your shell history by putting it in a config file:

printf 'header = "Authorization: Bearer %s"\n' '<TOXIPROXY_AUTH_TOKEN>' > toxiproxy-token.cfg && chmod 600 toxiproxy-token.cfg curl --cacert toxiproxy.crt -K toxiproxy-token.cfg https://<public-ip>/proxies

Toxiproxy's client libraries for test suites must send the same Authorization: Bearer header. Check that your client library supports Toxiproxy's token or lets you add a header to its HTTP client.

The API is the one described in the Toxiproxy README. These commands run on the instance and pass the token to curl in a config file that is never written to disk:

H=$(sudo sed -n 's/^TOXIPROXY_HOST=//p' /var/lib/sac/data/toxiproxy-credentials.txt) tok() {
sudo sed -n 's/^TOXIPROXY_AUTH_TOKEN=\(.*\)/header = "Authorization: Bearer \1"/p' /var/lib/sac/data/toxiproxy-credentials.txt; } T=(curl -s --cacert /etc/ssl/toxiproxy/toxiproxy.crt --resolve "$H:443:127.0.0.1") "${T[@]}" -K <(tok) "https://$H/version" | tee /dev/stderr | grep -q '"version": *"2.13.0"' \
  && "${T[@]}" -K <(tok) "https://$H/proxies" | python3 -c 'import json, sys; p = json.load(sys.stdin)["demo"]; print(p["name"], p["listen"], p["upstream"], p["enabled"]); sys.exit(p["upstream"] != "127.0.0.1:8081")'

{"version": "2.13.0"} demo 127.0.0.1:20000 127.0.0.1:8081 True

State changes use POST, PATCH and DELETE. For example, POST /proxies/demo/toxics adds a toxic, and POST /reset removes every toxic and re-enables every proxy. A GET never changes anything.

Point a DNS record at the instance, and then set the name. The helper generates a new certificate for the name and the instance's addresses, pins nginx to those names (other host names get no answer) and prints the new fingerprint:

sudo toxiproxy-set-hostname <your-domain>

To use a CA-issued certificate, replace /etc/ssl/toxiproxy/toxiproxy.crt and /etc/ssl/toxiproxy/toxiproxy.key with your own files (keep the key root-only, mode 0600) and run sudo systemctl reload nginx.

If the instance's public IP changes after a stop and start, run sudo toxiproxy-set-hostname with the new address, or use an Elastic IP.

Without the token, every path answers 401. A wrong token gets the same plain 401, because there is no lockout to trip, and the token is 192 random bits. Toxiproxy's own loopback API also refuses a request without the token.

Even with the token, another web site cannot drive the API from a browser, the front door refuses compressed request bodies, plain HTTP only redirects, and unknown host names get no answer. Run these checks:

H=$(sudo sed -n 's/^TOXIPROXY_HOST=//p' /var/lib/sac/data/toxiproxy-credentials.txt) tok() {
sudo sed -n 's/^TOXIPROXY_AUTH_TOKEN=\(.*\)/header = "Authorization: Bearer \1"/p' /var/lib/sac/data/toxiproxy-credentials.txt; } T=(curl -s --cacert /etc/ssl/toxiproxy/toxiproxy.crt --resolve "$H:443:127.0.0.1") { "${T[@]}" -o /dev/null -w 'no token: %{http_code}\n' "https://$H/proxies" "${T[@]}" -o /dev/null -w 'wrong token: %{http_code}\n' -H 'Authorization: Bearer not-the-token' "https://$H/proxies" curl -s -o /dev/null -w 'loopback API, no token: %{http_code}\n' http://127.0.0.1:8474/proxies "${T[@]}" -K <(tok) -o /dev/null -w 'cross-site POST: %{http_code}\n' -H 'Origin: https://evil.example' -X POST "https://$H/reset" "${T[@]}" -K <(tok) -o /dev/null -w 'compressed body: %{http_code}\n' -H 'Content-Encoding: gzip' -H 'Content-Type: application/json' --data-binary 'x' "https://$H/proxies" curl -s -o /dev/null -w 'plain http: %{http_code}\n' --resolve "$H:80:127.0.0.1" "http://$H/proxies" curl -sk -o /dev/null -w 'unknown host name: %{http_code}\n' --resolve "other.example:443:127.0.0.1" https://other.example/proxies || true } | tee /dev/stderr | tr '\n' ' ' | grep -qx 'no token: 401 wrong token: 401 loopback API, no token: 401 cross-site POST: 403 compressed body: 415 plain http: 301 unknown host name: 000 ' && echo "all refused as expected"

no token: 401 wrong token: 401 loopback API, no token: 401 cross-site POST: 403 compressed body: 415 plain http: 301 unknown host name: 000 all refused as expected

The token travels in the Authorization header, and a browser never adds that header on its own. A page on another site therefore cannot make your browser drive the API, and the image sends no CORS headers.

A proxy connects to whatever upstream it is given, so the image stops Toxiproxy from connecting to the EC2 instance metadata service and to this instance's own services. The following commands show both fences. They also show a proxy aimed at the metadata service and a proxy aimed at this instance's SSH server failing, while the demo proxy keeps working:

systemctl show -p IPAddressDeny --value toxiproxy | tr ' ' '\n' | sort
sudo nft list table inet sac_toxiproxy_egress | grep -oE 'oifname "lo" .*(accept|reject)|ip6? daddr \{[^}]*\} counter packets [0-9]+ bytes [0-9]+ reject' | sed -E 's/counter packets [0-9]+ bytes [0-9]+ //' export TOXIPROXY_URL="https://$(sudo sed -n 's/^TOXIPROXY_HOST=//p' /var/lib/sac/data/toxiproxy-credentials.txt)" export TOXIPROXY_AUTH_TOKEN="$(sudo sed -n 's/^TOXIPROXY_AUTH_TOKEN=//p' /var/lib/sac/data/toxiproxy-credentials.txt)" export SSL_CERT_FILE=/etc/ssl/toxiproxy/toxiproxy.crt toxiproxy-cli create -l 127.0.0.1:20098 -u 169.254.169.254:80 metadata >/dev/null \
  && toxiproxy-cli create -l 127.0.0.1:20099 -u 127.0.0.1:22 local-ssh >/dev/null \
  && { curl -s -m 5 -o /dev/null -w 'proxy to the metadata service: %{http_code}\n' http://127.0.0.1:20098/latest/meta-data/ echo "proxy to this instance's SSH: $(timeout 5 bash -c 'exec 3<>/dev/tcp/127.0.0.1/20099; head -c 7 <&3' 2>/dev/null | grep -c SSH-) bytes of SSH banner" curl -s http://127.0.0.1:20000/ | grep -q '^sac toxiproxy demo upstream' && echo "demo proxy: working"; } \
  | tee /dev/stderr | tr '\n' ' ' | grep -qx "proxy to the metadata service: 000 proxy to this instance's SSH: 0 bytes of SSH banner demo proxy: working " \
  && toxiproxy-cli delete metadata >/dev/null && toxiproxy-cli delete local-ssh >/dev/null && echo "fenced"

168.63.129.16/32 169.254.0.0/16 fd00:ec2::254/128 fe80::/10 oifname "lo" ip daddr 127.0.0.53 udp dport 53 accept oifname "lo" ip daddr 127.0.0.53 tcp dport 53 accept oifname "lo" ip daddr 127.0.0.1 tcp dport 8081 accept oifname "lo" ct state new reject ip daddr { 168.63.129.16, 169.254.0.0/16 } reject ip6 daddr { fd00:ec2::254, fe80::/10 } reject proxy to the metadata service: 000 proxy to this instance's SSH: 0 bytes of SSH banner demo proxy: working fenced

The metadata address 169.254.169.254 is inside 169.254.0.0/16. The loopback rule applies to connections that Toxiproxy opens to this instance itself (on 127.0.0.1 or on its own private IP). Only DNS and the demo page are allowed.

Toxiproxy's log shows each refused upstream as Unable to open connection to upstream (journalctl -u toxiproxy).

The image ships a self-test that checks the following:

• The listening ports
• Every refusal of the gate, including 25 wrong tokens followed by the right one
• Toxiproxy's own token check on loopback
• The demo proxy round trip
• The request policy
• The certificate

Run the self-test:

sudo toxiproxy-selftest | tail -n 1 | tee /dev/stderr | grep -qx TOXIPROXY_SELFTEST_OK

TOXIPROXY_SELFTEST_OK

The helper generates a new token, puts it in force at once at nginx and in Toxiproxy (the old token stops working), and writes it to the credentials file. It never prints the token.

Toxiproxy restarts to pick up the new token, so proxies created through the API are gone afterwards. The proxies in /etc/toxiproxy/toxiproxy.json come back. Rotate the token:

sudo toxiproxy-rotate-token | tee /dev/stderr | grep -q 'a new API token is in force' \
  && sudo toxiproxy-selftest | tail -n 1 | grep -qx TOXIPROXY_SELFTEST_OK && echo "self-test passes with the new token"

toxiproxy-rotate-token: a new API token is in force; read it with:
sudo grep TOXIPROXY_AUTH_TOKEN= /var/lib/sac/data/toxiproxy-credentials.txt self-test passes with the new token

The certificate generated at first boot is a server certificate only. It cannot sign other certificates. Verify this:

openssl x509 -in /etc/ssl/toxiproxy/toxiproxy.crt -noout -ext basicConstraints,keyUsage,extendedKeyUsage | tee /dev/stderr | grep -q 'CA:FALSE'

X509v3 Basic Constraints: critical CA:FALSE X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication

The binaries are the ones that the build record describes, built from the Toxiproxy 2.13.0 module with Go 1.27.3, and no Go toolchain is installed. BUILD-INFO.txt also records the module and toolchain checksums and each binary's SHA-256. Check the binaries against the record:

toxiproxy-server -version 2>/dev/null || /opt/toxiproxy/bin/toxiproxy-server -version toxiproxy-cli --version (cd / && sha256sum -c /usr/share/doc/sac-toxiproxy/SHA256SUMS) \
  && sed -n '1p;4,5p' /usr/share/doc/sac-toxiproxy/BUILD-INFO.txt \
  && { command -v go || echo "no Go toolchain"; } | grep -qx 'no Go toolchain' && echo "no Go toolchain"

toxiproxy-server version 2.13.0 toxiproxy-cli version 2.13.0 opt/toxiproxy/bin/toxiproxy-server: OK opt/toxiproxy/bin/toxiproxy-cli: OK Toxiproxy 2.13.0 built from source by sac reason: the upstream release binaries are built with go1.27.2 (GO-2026-6809, crypto/x509, fixed in go1.27.3) CVE gate: govulncheck v1.9.0 -mode=binary on unstripped builds: no vulnerabilities no Go toolchain

Toxiproxy is MIT licensed. Its license and the licenses of every library compiled into it are in /usr/share/doc/sac-toxiproxy/.

Toxiproxy logs proxy events (proxies starting, clients connecting, upstream failures) to the journal. At the default info level, it does not log API requests, and it never logs request headers. The level is LOG_LEVEL in /etc/toxiproxy/toxiproxy.env.

Ubuntu security updates install automatically. To apply everything now:

sudo apt-get update && sudo apt-get upgrade -y

New Toxiproxy releases arrive as new versions of the AWS Marketplace listing. Launch the new version, and copy /etc/toxiproxy/toxiproxy.json across if you changed it.