Parallelism

In addition to manually defining multiple tasks to perform work in parallel or leveraging the power of dynamic tasks to generate parallel tasks based on a template, Mint supports static configuration of parallelism via the parallel key to quickly parallelize your commands, leaves, and embedded runs.

There are a number of ways to specify parallelism. We'll start with a simple example to discuss how parallelism works in Mint and then we'll move onto more complex examples.

Consider the following tasks:

- key: code
  call: mint/git-clone 1.2.3
  with:
    repository: https://github.com/YOUR_ORG/YOUR_REPO.git
    ref: main

- key: tests
  use: code
  run: bin/run-my-tests.sh --index 0 --total 1

Eventually, your tests task might take a long time to run and you may want to split it into multiple tasks where each is devoted to a subset of the test suite. This is made easy by the parallel key:

- key: tests
  parallel: 2
  use: code
  run: bin/run-my-tests.sh --index ${{ parallel.index }} --total ${{ parallel.total }}

With the addition on parallel: 2, Mint will generate two tasks (tests-0 and tests-1) as subtasks of tests and provide them with the proper metadata in the parallel context to run the tests in parallel.

Kinds of Parallelism

Mint offers three ways to configure parallelism.

Total Parallelism

"Total" refers to the total number of parallel tasks to generate and expects a positive integer. When using total parallelism, you can reference parallel.index and parallel.total in your commands to affect the task behavior. Additionally, unlike the other kinds of parallelism, you can access the $MINT_PARALLEL_INDEX and $MINT_PARALLEL_TOTAL environment variables by default.

You've seen one example of this so far but they're all enumerated below:

A Static Number

- key: tests
  parallel: 2
  # ...

An expression which evaluates to a number

- key: tests
  parallel: ${{ tasks.other-task.values.parallelism }}
  # ...

An object which specifies the total key

- key: tests
  parallel:
    total: 2
  # ...

# or
- key: tests
  parallel:
    total: ${{ tasks.other-task.values.parallelism }}
  # ...

Matrix Parallelism

Matrix parallelism generates all combinations of the values you specify for your matrix keys. When using matrix parallelism, you can reference the matrix keys within the parallel context in your commands to affect the task behavior. Here's one example of how to use matrix parallelism:

- key: build-binary
  parallel:
    matrix:
      os: [linux, macos, windows]
      arch: [x86, arm64]
  # ...

When Mint generates the parallel tasks, you'll end up with 6: build-binary-x86-linux, build-binary-arm64-linux, build-binary-x86-macos, build-binary-arm64-macos, build-binary-x86-windows, and build-binary-arm64-windows.

You can use any number of keys in your matrix so long as they result in at least one parallel task.

Additionally, you can use expressions in your matrix:

- key: print-greeting
  parallel:
    matrix:
      greeting: ['${{ tasks.other-task.values.greeting }}']
      names: ${{ tasks.other-task.values.names }}
  # ...

When using an expression to produce the values for a matrix key, ensure you provide a JSON array of numbers, strings, or booleans.

Values Parallelism

Values parallelism uses the values you specify to generate the individual tasks. The values must all have the same keys. When using values parallelism, you can reference the keys of your values within the parallel context in your commands to affect the task behavior.

There are two ways to use values parallelism:

An array of values

- key: build-binary
  parallel:
    values:
      - os: linux
        arch: x86
      - os: macos
        arch: arm64
      - os: windows
        arch: ${{ tasks.other-task.values.windows-arch }}
  # ...

In this case, if tasks.other-task.values.windows-arch is x86, Mint will generate three tasks: build-binary-x86-linux, build-binary-arm64-macos, and build-binary-x86-windows.

An expression

- key: build-binary
  parallel:
    values: ${{ tasks.other-task.values.os-arch-combinations }}
  # ...

When using an expression to produce the values, ensure you provide a JSON array of objects which have numbers, strings, or booleans as values. Additionally, the objects must all have the same keys.

Configuring the generated key

You can configure the key of the generated parallel tasks by specifying key within the parallel configuration:

- key: build-binary
  parallel:
    key: generated-build-binary-${{ parallel.os }}-${{ parallel.arch }}
    matrix:
      os: [linux, macos, windows]
      arch: [x86, arm64]
  # ...

Configuring the tasks limit

By default, Mint will limit your parallel tasks to 16 generated tasks. This serves as a mechanism to avoid incidentally generating a large number of tasks and wasting resources. If you need more than 16 (or would like to limit the number of tasks further), you can specify tasks-limit in your parallel configuration:

- key: tests
  parallel:
    tasks-limit: 20
    total: 20
  # ...

Currently, Mint supports between 1 and 256 tasks generated in parallel. If you find that you need more than 256, please reach out to us so we can discuss your use case.