Deployment guide

Concourse CI 8.3.3

Overview

Concourse is an open-source continuous integration and delivery system. Everything is a pipeline written in YAML: resources such as a Git repository, an image, an S3 bucket or a timer feed jobs, and every task in a job runs in a fresh container. You drive it from the web interface and from fly, its command-line client.

This guide covers launching and using the Concourse CI 8.3.3 on Ubuntu 24.04 AMI from SecureAnyCloud on Amazon EC2. It is one version of the SecureAnyCloud Hardened Apache Subversion (SVN) & CI/CD Tools listing on AWS Marketplace. You pick it as the software version when you subscribe and launch.

What you get:

A complete single-node install. The Concourse web node (API, web interface and the TSA worker gateway), one containerd worker and PostgreSQL 16 run on the instance as systemd services. The database is reached over its local socket only, with no network listener and no password.
Nothing shared between instances. The image ships no password, key or certificate. First boot creates the administrator password, the session signing key, the TSA host key, the worker key, the database encryption key and the HTTPS certificate on your instance. Concourse, the worker and nginx do not start until first boot has finished.
HTTPS from first boot, and sign-in first. nginx serves Concourse on port 443 with a certificate generated on your instance, and port 80 redirects to HTTPS. Anyone who is not signed in gets the sign-in page. The API answers them with 401, except for /api/v1/info and the fly download.
Task containers fenced in. Pipeline tasks run unprivileged, in a user namespace. They cannot reach the instance metadata service, so they cannot pick up the instance's IAM role. They also cannot reach the instance itself over IPv4 or IPv6, or Concourse's own internal ports. They can reach the internet, which most pipelines need.
A web node that talks to loopback only. The kernel drops every packet between concourse-web and anything but 127.0.0.1/::1. Its internal ports cannot be reached from your VPC or from a task, and a pipeline cannot use it to reach other machines in your network.
Worker APIs closed to local accounts. The worker's garden and baggageclaim APIs, and the TSA's tunnels to them, have no authentication of their own. A kernel rule lets only root and the Concourse web node open them, so no other account on the instance can use them to start a container.
A rate-limited sign-in. The sign-in endpoint accepts 30 requests a minute from one address, in every spelling of the endpoint. Cross-origin state-changing requests are refused.
Metadata service blocked for every Concourse service, PostgreSQL, nginx and the helper commands. The AMI requires IMDSv2.
fly preinstalled, and logged in without typing a password. sudo concourse-fly-login saves a fly target for you.
Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on. Concourse and fly are the official 8.3.3 release, checked against upstream's published SHA-256 sums.

Trademark notice. Concourse is an open-source project released under Apache-2.0. "Concourse" is used here only to describe the software this image runs. SecureAnyCloud is not affiliated with, endorsed by, or sponsored by the Concourse project.

Prerequisites

An AWS account subscribed to the listing, an EC2 key pair, and a VPC subnet with a route to the internet. Tasks pull their container images, and most pipelines fetch code.
A security group that allows TCP 443 (the web interface and fly) and TCP 80 (redirects to HTTPS) from the networks that should reach them, and TCP 22 (SSH) from your own networks only. Concourse needs no other inbound port. The web node, the TSA, the worker and PostgreSQL listen on loopback only.

Launch the instance

In the AWS Marketplace listing, choose Continue to Subscribe, then Continue to Configuration. Pick Concourse CI 8.3.3 on Ubuntu 24.04 as the software version, choose your Region, then choose Launch through EC2.

Choose t3.large or larger, your key pair, and a security group as above. Builds run on the instance itself, and m5.xlarge suits a busy team.

The root volume is 20 GiB. Give it more if your pipelines build large images or caches.

Connect to your instance

Connect over SSH as the user for your version:

Version SSH user

Concourse CI 8.3.3 on Ubuntu 24.04 ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

Check the services

First boot takes about a minute. After that, the web node, the worker, PostgreSQL, nginx and the first-boot unit are all active:

systemctl is-active concourse-web concourse-worker postgresql@16-main nginx concourse-ci-firstboot

active active active active active

nginx is the only Concourse-facing service listening on the network, on ports 80 and 443. SSH listens on 22.

sudo ss -tlnp | awk 'NR==1 || /nginx|sshd|concourse/' | awk '{print $1, $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g'

State Local Peer LISTEN 127.0.0.1:8888 users:(("concourse")) LISTEN 0.0.0.0:22 users:(("sshd"),("systemd")) LISTEN 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx")) LISTEN 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx")) LISTEN 127.0.0.1:2222 users:(("concourse")) LISTEN 127.0.0.1:8079 users:(("concourse")) LISTEN 127.0.0.1:8080 users:(("concourse")) LISTEN 127.0.0.1:7777 users:(("concourse")) LISTEN 127.0.0.1:7776 users:(("concourse")) LISTEN 127.0.0.1:7787 users:(("concourse")) LISTEN 127.0.0.1:7788 users:(("concourse")) LISTEN [::]:22 users:(("sshd"),("systemd")) LISTEN [::]:80 users:(("nginx"),("nginx"),("nginx")) LISTEN *:33073 users:(("concourse")) LISTEN [::]:443 users:(("nginx"),("nginx"),("nginx")) LISTEN *:37725 users:(("concourse"))

The web node listens on 127.0.0.1:8080 (API and web interface) and 127.0.0.1:2222 (the TSA, where the worker registers). The worker's garden and baggageclaim APIs are on 7777 and 7788.

The two *:<random> ports are the TSA's tunnels to the worker. Concourse always opens them on all addresses and has no setting to change that, so this image fences them in the kernel instead. The web node exchanges packets with loopback only, and Step 11 shows the rule. The port numbers change every time the worker reconnects.

Garden, baggageclaim and the tunnels have no authentication of their own. A second kernel rule lets only root and the web node open them, also shown in Step 11.

The API's info endpoint is the one call that answers without a session. The pipeline list does not, and port 80 redirects to HTTPS:

curl -s --cacert /etc/nginx/ssl/concourse.crt https://127.0.0.1/api/v1/info; echo
curl -s -o /dev/null -w 'anonymous pipeline list: %{http_code}\n' --cacert /etc/nginx/ssl/concourse.crt https://127.0.0.1/api/v1/teams/main/pipelines
curl -s -o /dev/null -w 'port 80: %{http_code}\n' http://127.0.0.1/

{"version":"8.3.3","worker_version":"3.0","feature_flags":{"build_rerun":false,"cache_streamed_volumes":false,"global_resources":false,"resource_causality":false},"external_url":"https://<public-ip>","cluster_name":"sac"} anonymous pipeline list: 401 port 80: 301

Check the release

Print the Concourse and fly versions, then list the resource types built into the worker:

/opt/concourse/bin/concourse --version fly --version cat /usr/share/doc/concourse-sac/resource-types.txt

8.3.3 8.3.3 bosh-io-release 1.3.7 bosh-io-stemcell 1.7.2 git 1.24.1 github-release 1.14.3 hg 1.5.7 mock 0.14.8 pool 1.8.4 registry-image 1.19.1 s3 2.5.7 semver 2.0.5 time 1.11.6

These are the resource types built into the worker. Upstream's deprecated docker-image type is not shipped, because its image carries packages with known, unfixed vulnerabilities. Use registry-image instead, which fetches and pushes images without a Docker daemon.

Get your sign-in details

The credentials file holds the address Concourse is served on and the administrator account, both generated on this instance. Read it with sudo:

sudo cat /root/concourse-ci-credentials.txt

CONCOURSE_URL=https://<public-ip>/
# CONCOURSE_ADMIN_USER=admin CONCOURSE_ADMIN_PASSWORD=<28 letters and digits>
#
# admin is the main team's owner (the Concourse administrator). # fly -t ci login -c https://<public-ip>/ -u admin --ca-cert concourse.crt

The password comes from the kernel's random number generator on first boot and is known to nobody else. Concourse stores only a bcrypt hash of it, in /etc/concourse/web.env, which only root can read.

The same file holds CONCOURSE_ENCRYPTION_KEY, which encrypts pipeline configurations and credentials in the database. Keep a copy of it with your database backups (Step 13).

The external URL is the instance's public IPv4 address, read from the instance metadata service at first boot. Step 10 shows how to serve Concourse under your own DNS name.

Check the certificate

The HTTPS certificate is self-signed and was generated on your instance. Read its fingerprint here, over SSH, and compare it with what your browser shows before you accept it:

openssl x509 -in /etc/nginx/ssl/concourse.crt -noout -subject -enddate -fingerprint -sha256 -ext subjectAltName | sed -E 's/[0-9]+(\.[0-9]+){3}/<ip>/g; s/ec2-[0-9-]+/ec2-<ip>/g; s/ip-[0-9-]+/ip-<ip>/g; s/=([0-9A-F]{2}:){31}[0-9A-F]{2}/=<64 hex digits>/'

subject=CN = <ip>, O = sac notAfter=Feb 9 19:01:38 2029 GMT sha256 Fingerprint=<64 hex digits> X509v3 Subject Alternative Name: IP Address:<ip>, IP Address:<ip>, DNS:ec2-<ip>.compute-1.amazonaws.com, DNS:ip-<ip>.ec2.internal, IP Address:<ip>, DNS:localhost

The sed filter only hides this example's addresses. Run the openssl command on its own to see yours.

The certificate names the instance's public and private addresses and DNS names, and 127.0.0.1.

Sign in to the web interface

Open https://<public-ip>/ in your browser and accept the certificate once you have checked its fingerprint. The sign-in page appears:

Sign in as admin with the password from Step 5. The dashboard lists the pipelines of every team you belong to. It stays empty until you set a pipeline (Step 8 sets hello):

On this image, signing in comes first. Upstream Concourse lets anonymous visitors browse pipelines that a team has exposed, and shows them an empty dashboard otherwise. Here, nginx sends every visitor without a session to the sign-in page, so exposed pipelines and status badges are visible to signed-in users only.

Use fly on the instance and run a pipeline

fly is installed at /usr/local/bin/fly. concourse-fly-login gets a token for admin from Concourse's token endpoint. The password never appears on a command line, in your shell history or on the screen.

The command saves the token as the target ci in your ~/.flyrc (mode 0600, with this instance's certificate). The token is valid for 24 hours. Run the command again when it expires.

sudo concourse-fly-login fly -t ci workers

fly target 'ci' -> https://127.0.0.1 (team main, admin) saved in /home/ubuntu/.flyrc; try: fly -t ci pipelines ip-172-31-81-63 3 linux none none running 3.0 2m49s

Set a pipeline with one job, unpause it, and run the job while following its output:

WORKDIR="$(mktemp -d)" cat > "${WORKDIR}/hello.yml" <<'YML' jobs: - name: hello plan: - task: say-hello config: platform: linux image_resource: type: registry-image source: {repository: public.ecr.aws/docker/library/alpine, tag: "3.20"} run: path: sh args: [-c, 'echo "hello from Concourse: $(cat /etc/alpine-release) on $(uname -m)"'] YML fly -t ci set-pipeline -p hello -c "${WORKDIR}/hello.yml" -n fly -t ci unpause-pipeline -p hello fly -t ci trigger-job -j hello/hello --watch rm -rf "${WORKDIR}"

pipeline created! you can view your pipeline here: https://127.0.0.1/teams/main/pipelines/hello unpaused 'hello' started hello/hello #1 initializing initializing check: image selected worker: ip-172-31-81-63 fetching public.ecr.aws/docker/library/alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc selected worker: ip-172-31-81-63 running sh -c echo "hello from Concourse: $(cat /etc/alpine-release) on $(uname -m)" hello from Concourse: 3.20.10 on x86_64 succeeded

The pipeline and its build also appear in the web interface:

fly execute runs a one-off task without a pipeline. The task below shows what a task can and cannot reach on this image:

WORKDIR="$(mktemp -d)" cat > "${WORKDIR}/probe.yml" <<'YML' platform: linux image_resource: type: registry-image source: {repository: public.ecr.aws/docker/library/alpine, tag: "3.20"} run: path: sh args: - -c - | echo "task user: $(id -u) (root inside the task's user namespace)" wget -q -T 5 -O /dev/null https://aws.amazon.com/ && echo "internet: reachable" wget -q -T 5 -O /dev/null http://169.254.169.254/latest/meta-data/ 2>/dev/null && echo "instance metadata: REACHABLE" || echo "instance metadata: blocked" wget -q -T 5 -O /dev/null "http://$(ip route | awk '/default/ {print $3}'):8080/" 2>/dev/null && echo "host port 8080: REACHABLE" || echo "host port 8080: blocked" YML fly -t ci execute -c "${WORKDIR}/probe.yml" rm -rf "${WORKDIR}"

executing build 12 at https://127.0.0.1/builds/12 initializing initializing check: image selected worker: ip-172-31-81-63 INFO: found existing resource cache selected worker: ip-172-31-81-63 task user: 0 (root inside the task's user namespace) internet: reachable instance metadata: blocked host port 8080: blocked succeeded

When you finish with the example, remove it:

fly -t ci destroy-pipeline -p hello -n

!!! this will remove all data for pipeline `hello` `hello` deleted

Use fly from your workstation

Download fly from your instance, along with the instance's certificate, then log in. The download is the one part of the server that needs no session.

Without -p, fly opens the sign-in page in your browser and receives the token when you sign in:

curl -k -o fly 'https://<public-ip>/api/v1/cli?arch=amd64&platform=linux' && chmod +x fly scp -i <your-key.pem> ubuntu@<public-ip>:/etc/nginx/ssl/concourse.crt concourse.crt ./fly -t ci login -c https://<public-ip>/ -u admin --ca-cert concourse.crt

Use platform=darwin or platform=windows for macOS or Windows.

Check the downloaded certificate's fingerprint against Step 6 before you rely on it. --ca-cert makes fly trust exactly that certificate, so you never need -k.

fly -p <password> also works, but it puts the password on the command line, where other users of the machine and your shell history can see it.

Your own DNS name and certificate

Point a DNS name at the instance. An Elastic IP keeps the address across stops. Then run:

sudo concourse-set-hostname ci.your-domain.example

The command sets Concourse's external URL (used in sign-in redirects, fly login and links), nginx's server name and the cross-origin check to the new name. It generates a new certificate that names it and restarts the web node.

To use a certificate from your own CA instead, run the command first. Then replace /etc/nginx/ssl/concourse.crt (the full chain) and /etc/nginx/ssl/concourse.key (mode 0600, owned by root), and reload nginx with sudo nginx -t && sudo systemctl reload nginx. Browsers and fly then need no certificate exception or --ca-cert.

Security checks

Run the commands below to verify four things. The web node exchanges packets with loopback only. Task containers' IPv6 traffic to and through the host is dropped. The worker refuses task traffic to the metadata addresses. Root cannot log in over SSH.

systemctl show -p IPAddressAllow -p IPAddressDeny concourse-web
systemctl show -p IPAddressDeny concourse-worker nginx postgresql@16-main
sudo ip6tables -S INPUT | grep concourse0 grep -o -- '--containerd-restricted-network [^ ]*' /etc/systemd/system/concourse-worker.service grep -E '^CONCOURSE_(BIND_IP|CONTAINERD_ALLOW_HOST_ACCESS|CONTAINERD_DNS_PROXY_ENABLE)=' /etc/concourse/worker.env
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication) '

IPAddressAllow=::1/128 127.0.0.0/8 IPAddressDeny=0.0.0.0/0 ::/0 IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128 -A INPUT -i concourse0 -j DROP --containerd-restricted-network 169.254.169.254/32 --containerd-restricted-network 168.63.129.16/32 CONCOURSE_BIND_IP=127.0.0.1 CONCOURSE_CONTAINERD_DNS_PROXY_ENABLE=false CONCOURSE_CONTAINERD_ALLOW_HOST_ACCESS=false permitrootlogin no passwordauthentication no

Only root (the worker) can open garden and baggageclaim. Only root and the concourse account (the web node) can open the TSA's tunnel ports. Any other account on the instance gets a refused connection.

concourse-ci-local-guard.service applies the rule (IPv4 and IPv6) at every boot, before the web node and the worker start:

sudo iptables -S SAC-CI-LOCAL
sudo -u nobody bash -c 'echo > /dev/tcp/127.0.0.1/7777' && echo reachable || echo refused

-N SAC-CI-LOCAL -A SAC-CI-LOCAL -m owner --uid-owner 0 -j RETURN -A SAC-CI-LOCAL -p tcp -m multiport --dports 7777,7788 -j REJECT --reject-with tcp-reset -A SAC-CI-LOCAL -m owner --uid-owner 997 -j RETURN -A SAC-CI-LOCAL -p tcp -m tcp --dport 32768:60999 -j REJECT --reject-with tcp-reset bash: connect: Connection refused bash: line 1: /dev/tcp/127.0.0.1/7777: Connection refused refused

The tunnel ports are picked from the kernel's ephemeral range (32768 to 60999). The rule therefore covers that whole range on loopback for every account except root and concourse. If you run your own service on this instance for other local users, give it a loopback port below 32768.

What a task can do. Tasks run as root inside a user namespace, which maps to an unprivileged user on the instance. They have no access to the instance's own services and no IAM role.

A task or resource type with privileged: true is different. It runs as real root with the host's capabilities and devices, which amounts to root on the instance. Use it only for tasks you trust, such as building images with a Docker daemon, and prefer registry-image and rootless image builders.

Who can run code here. Anyone who can set a pipeline or run fly execute in a team can run arbitrary code on the worker. Only members of a team can do either, and only the main team's owners can create teams. Add people as members of their own team rather than sharing admin, and review pipelines that ask for privileged: true.

Opening the web node to the network. concourse-web talks to loopback only. Features that make the web node itself call out, such as GitHub, GitLab or OIDC sign-in, or a Vault or other credential manager on another machine, therefore need a deliberate exception. Allow just the addresses it must reach, for example a Vault server in your VPC:

sudo systemctl edit concourse-web.service
# add, then save: [Service] IPAddressAllow=10.0.12.34/32

Then run sudo systemctl restart concourse-web. The metadata service stays blocked unless you allow it.

Anything you allow can also reach the TSA's tunnel ports. Keep those addresses trusted, and keep your security group limited to 22, 80 and 443.

Where things are

The table below lists where each part of the installation lives.

What Where

Pipelines, builds, teams and encrypted credentials (PostgreSQL 16) /var/lib/postgresql/16/main (database concourse)

Worker volumes and containers (caches; rebuilt as needed) /var/lib/concourse/worker

Web node settings, password hash and encryption key /etc/concourse/web.env (root only)

Worker settings /etc/concourse/worker.env

Keys (session signing, TSA host, worker) /etc/concourse/keys/

The administrator's password /root/concourse-ci-credentials.txt

HTTPS certificate and key /etc/nginx/ssl/

Concourse and fly /opt/concourse, /usr/local/bin/fly

Logs are in the journal: sudo journalctl -u concourse-web -u concourse-worker.

Backups and upgrades

Back up the database and the encryption key together; one is useless without the other:

sudo -u postgres pg_dump -Fc concourse > concourse-$(date +%F).dump
sudo grep '^CONCOURSE_ENCRYPTION_KEY=' /etc/concourse/web.env > concourse-encryption-key.txt

Copy both off the instance, to S3 for example, and keep them private. The worker's volumes need no backup, because Concourse fetches resources again when it needs them.

Ubuntu, PostgreSQL and nginx receive security updates through unattended upgrades. Concourse is not upgraded automatically.

To move to a newer release, take a backup and stop concourse-worker and concourse-web. Replace /opt/concourse and /usr/local/bin/fly with the new release, then start the services again. The web node migrates the database on start.

Security recommendations

Keep TCP 443 and 80 open only to your team's networks. Build logs and pipeline configurations describe your systems in detail.
Replace the self-signed certificate with your own (Step 10) before you share the address widely.
Give each team its own Concourse team and keep admin for administration. Refuse privileged: true tasks you have not reviewed.
Keep secrets in Concourse's credential management, not in pipeline files. They are encrypted at rest with your encryption key.
Keep the instance patched, and upgrade Concourse deliberately after a backup.

Support and licensing

SecureAnyCloud provides 24/7 technical support for this image by email and live chat: sales@secureanycloud.com or https://secureanycloud.com/support. Include the instance type, the Region and the output of Steps 3 and 11.

Concourse is released under Apache-2.0. Its license and notice are at /usr/share/doc/concourse-sac/LICENSE.md and NOTICE.md. PostgreSQL and nginx are Ubuntu archive packages under their own licenses (see /usr/share/doc/<package>/copyright). SecureAnyCloud is not affiliated with, endorsed by, or sponsored by the Concourse project.

Deploy on AWS

Launch Concourse CI 8.3.3 on Ubuntu 24.04 on AWS with 24/7 support from SecureAnyCloud.

View on Marketplace

Need Help?

Our support team is available 24/7.

sales@secureanycloud.com

Deploy on AWS