Parallelism

With RWX, you can manually define multiple tasks to perform work in parallel, or you can leverage the power of dynamic tasks to generate parallel tasks based on templates.

RWX also supports static configuration of parallelism via the parallel key, allowing you to parallelize your commands, packages, and embedded runs quickly.

There are several of ways to specify parallelism. Let's start with a simple example to discuss how parallelism works in RWX and then move on to more complex examples.

Consider the following tasks:

- key: code
  call: git/clone 1.8.0
  with:
    repository: https://github.com/YOUR_ORG/YOUR_REPO.git
    ref: main

- key: tests
  use: code
  run: bin/run-my-tests.sh --index 0 --total 1

Eventually, your tests task might take a long time to run and you may want to split it into multiple tasks, where each is devoted to a subset of the test suite. The parallel key makes this easy:

- key: tests
  parallel: 2
  use: code
  run: bin/run-my-tests.sh --index ${{ parallel.index }} --total ${{ parallel.total }}

With the addition on parallel: 2, RWX will generate two tasks (tests-0 and tests-1) as subtasks of tests and provide them with the proper metadata in the parallel context to run the tests in parallel.

Kinds of Parallelism

RWX offers three ways to configure parallelism.

Total Parallelism

"Total" refers to the total number of parallel tasks to generate and expects a positive integer. When using total parallelism, you can reference parallel.index and parallel.total in your commands to affect the task behavior. Additionally, unlike the other kinds of parallelism, you can access the $RWX_PARALLEL_INDEX and $RWX_PARALLEL_TOTAL environment variables by default.

You've seen one example of this so far. They're all enumerated below:

A Static Number

- key: tests
  parallel: 2
  # ...

An expression which evaluates to a number

- key: tests
  parallel: ${{ tasks.other-task.values.parallelism }}
  # ...

An object which specifies the total key

- key: tests
  parallel:
    total: 2
  # ...

# or
- key: tests
  parallel:
    total: ${{ tasks.other-task.values.parallelism }}
  # ...

Matrix Parallelism

Matrix parallelism generates all combinations of the values you specify for your matrix keys. When using matrix parallelism, you can reference the matrix keys within the parallel context in your commands to affect the task behavior. Here's one example of how to use matrix parallelism:

- key: build-binary
  parallel:
    matrix:
      os: [linux, macos, windows]
      arch: [x86, arm64]
  # ...

When RWX generates the parallel tasks, you'll end up with 6: build-binary-x86-linux, build-binary-arm64-linux, build-binary-x86-macos, build-binary-arm64-macos, build-binary-x86-windows, and build-binary-arm64-windows.

You can use any number of keys in your matrix so long as they result in at least one parallel task.

Additionally, you can use expressions in your matrix:

- key: print-greeting
  parallel:
    matrix:
      greeting: ['${{ tasks.other-task.values.greeting }}']
      names: ${{ tasks.other-task.values.names }}
  # ...

When using an expression to produce the values for a matrix key, ensure you provide a JSON array of numbers, strings, or booleans.

Values Parallelism

Values parallelism uses the values you specify to generate the individual tasks. The values must all have the same keys. When using values parallelism, you can reference the keys of your values within the parallel context in your commands to affect the task behavior.

There are two ways to use values parallelism:

An array of values

- key: build-binary
  parallel:
    values:
      - os: linux
        arch: x86
      - os: macos
        arch: arm64
      - os: windows
        arch: ${{ tasks.other-task.values.windows-arch }}
  # ...

In this case, if tasks.other-task.values.windows-arch is x86, RWX will generate three tasks: build-binary-x86-linux, build-binary-arm64-macos, and build-binary-x86-windows.

An expression

- key: build-binary
  parallel:
    values: ${{ tasks.other-task.values.os-arch-combinations }}
  # ...

When using an expression to produce the values, ensure you provide a JSON array of objects which have numbers, strings, or booleans as values. Additionally, the objects must all have the same keys.

Configuring the generated key

You can configure the key of the generated parallel tasks by specifying key within the parallel configuration:

- key: build-binary
  parallel:
    key: generated-build-binary-${{ parallel.os }}-${{ parallel.arch }}
    matrix:
      os: [linux, macos, windows]
      arch: [x86, arm64]
  # ...

Configuring the tasks limit

When using a dynamic value for parallelism (e.g. an expression for total parallelism, an expression for values parallelism, or matrix parallelism), RWX will limit the number of generated tasks to at most 16. This serves as a mechanism to avoid accidentally generating a large number of tasks and wasting resources. If you reach this limit and need to generate more than 16 tasks, you can specify tasks-limit in your parallel configuration:

- key: tests
  parallel:
    total: ${{ init.parallelism }}
    tasks-limit: 20
  # ...

Currently, RWX supports between 1 and 256 tasks generated in parallel. If you find that you need more than 256, please reach out to us so we can discuss your use case.