Docker

RWX supports running and building Docker containers inside of tasks using the docker CLI. To enable support for Docker in a task, specify docker: true:

tasks:
  - key: container
    docker: true
    run: docker run hello-world

RWX's Docker functionality is most commonly used to run background services as part of a task:

tasks:
  - key: packages
    run: |
      sudo apt-get update
      sudo apt-get install netcat redis-tools
      sudo apt-get clean

  - key: ping-redis
    use: [packages]
    docker: true
    background-processes:
      - key: redis
        run: docker run -p 6379:6379 redis
        ready-check: redis-cli ping
    run: |
      redis-cli SET mykey "Hello, Redis!"
      redis-cli GET mykey

Preserving Docker data

By default, RWX deletes all data related to Docker at the end of a task. If you want to preserve the images you have pulled or the volumes used by your containers, you can specify docker: preserve-data. This is very useful for pulling and caching images ahead of the task that actually uses them:

tasks:
  - key: docker-images
    docker: preserve-data
    run: docker pull hello-world

  - key: some-task
    use: docker-images
    docker: true
    run: docker run hello-world

RWX's internal networking and cache layer management is faster than Docker's, so typically pre-pulling large images in a cached task will be faster than pulling them on-demand in the task that needs them. You can also pull images with docker compose pull in an RWX task if you're using Docker compose.

Ready checks

As described in the documentation for background processes, it's recommended to use service-specific tools for ready checks. However, you may not want to have to install those tools outside of a docker container. In this case, you can use docker health commands.

tasks:
  - key: docker-images
    docker: preserve-data
    run: docker pull redis

  - key: demo-redis
    use: docker-images
    docker: true
    background-processes:
      - key: redis
        run: |
          docker run -p 6379:6379 \
            --name redis \
            --health-cmd 'redis-cli ping' \
            --health-interval 5s \
            redis
        ready-check: |
          docker ps -a # for observability / debugging
          docker inspect redis | jq -e '.[0].State.Health.Status == "healthy"'
    run: |
      echo -e '*1\r\n$4\r\nPING\r\n' | nc localhost 6379

RWX also provides an rwx-docker-ready-check command to make sure all docker services are healthy.

tasks:
  - key: docker-images
    docker: preserve-data
    run: docker pull redis

  - key: demo-redis
    use: docker-images
    docker: true
    background-processes:
      - key: redis
        run: |
          docker run -p 6379:6379 \
            --name redis \
            --health-cmd 'redis-cli ping' \
            --health-interval 5s \
            redis
        ready-check: rwx-docker-ready-check
    run: |
      echo -e '*1\r\n$4\r\nPING\r\n' | nc localhost 6379

Caching Docker volumes

You can also take advantage of preserved Docker data to cache Docker volumes needed by your task. For example, if your test suite relies on a Postgres database being set up a certain way, you can perform your database setup in one task and then have your testing task depend on that setup task:

tasks:
  - key: packages
    run: sudo apt-get update && sudo apt-get install netcat postgresql-client && sudo apt-get clean

  - key: postgres-image
    docker: preserve-data
    run: docker pull postgres

  - key: postgres-data
    use: [packages, postgres-image]
    docker: preserve-data
    background-processes:
      - key: postgres
        run: |
          docker volume create pgdata
          docker run -p '5432:5432' -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=password -v pgdata:/var/lib/postgresql/data postgres
        ready-check: pg_isready -U postgres -h 127.0.0.1 -p 5432
    run: |
      psql postgres://postgres:[email protected]:5432 -c "CREATE DATABASE db;"
      psql postgres://postgres:[email protected]:5432/db -c "CREATE TABLE users (id SERIAL PRIMARY KEY, email VARCHAR(255));"
      psql postgres://postgres:[email protected]:5432/db -c "INSERT INTO users (email) VALUES ('[email protected]');"

  - key: test
    use: postgres-data
    docker: true
    background-processes:
      - key: postgres
        run: docker run -p '5432:5432' -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=password -v pgdata:/var/lib/postgresql/data postgres
        ready-check: pg_isready -U postgres -h 127.0.0.1 -p 5432
    run: psql postgres://postgres:[email protected]:5432/db -c "SELECT * FROM users;"

Enabling incremental Docker builds

Preserving Docker data also allows you to enable incremental Docker builds inside your task. Using a tool cache, you can preserve the Docker build cache from a Docker build and restore it the next time the task attempts the same Docker build:

tool-cache:
  vault: default

tasks:
  - key: dockerfile
    run: |
      cat << EOF > Dockerfile
      FROM ubuntu:22.04
      RUN apt-get update &&  apt-get install -y curl && apt-get clean && rm -rf /var/lib/apt/lists/*
      # Uncomment the next line to try incremental building
      # RUN touch example.txt
      EOF

  - key: docker-build
    use: dockerfile
    docker: preserve-data
    tool-cache: docker-build
    run: docker build .

Running this sample, then uncommenting the line in the dockerfile task and running it again will show that the apt-get build step is a cache hit in the second run.