Adaptive CI Pipelines with Dynamic Tasks

CI/CD
May 31, 2024
Dan Manges
Adaptive CI Pipelines with Dynamic Tasks

In most CI platforms, tasks needs to be defined statically when the run starts. This limits the flexibility in workflows, especially as projects become more advanced and yaml configuration gets more complicated. It's especially unfortunate to have this limitation in a dev tool built for engineers.

Engineers are adept at writing code to solve problems. That's why when we built Mint, we knew we wanted to support dynamic tasks. We wanted to give engineers the ability to define tasks at runtime, utilizing any programming language and logic that they wanted.

Dynamic tasks enable building CI pipelines which are adaptive based on things like the commit contents or data from external data sources. They unlock a lot of possibilities for building highly performant CI pipelines.

Task Data Structures

YAML gets a lot of criticism in its use for configuring infrastructure. It's usually not YAML itself that's the issue though. YAML is just a format for serializing data. The pain points around YAML configuration are usually more centered around

having to git push to run workflows, not being able to debug, and implementing complex logic in embedded expressions rather than just writing code.

To generate dynamic tasks in Mint, you can use any programming language to build an array of task definitions, and then serialize it as YAML.

Here's a simple example using bash:

1
tasks:
2
- key: generate-dynamic-task
3
run: |
4
cat << EOF | tee $MINT_DYNAMIC_TASKS/tasks.yml
5
- key: dynamic-task-1
6
run: echo this is the first dynamic task
7
- key: dynamic-task-2
8
run: echo this is the second dynamic task
9
EOF

You probably don't want to use bash to generate dynamic tasks, although we've used it in some of our own workflows by leveraging envsubst with template files.

Here's a more practical example using Ruby;

1
tasks:
2
- key: ruby
3
call: mint/install-ruby 1.0.9
4
with:
5
ruby-version: 3.3.1
6
7
- key: generate-dynamic-tasks
8
use: [ruby]
9
run: |
10
ruby -e '
11
require "yaml"
12
tasks = [
13
{
14
"key" => "dynamic-task",
15
"run" => "echo this task was generated from ruby"
16
}
17
]
18
puts YAML.dump(tasks)
19
' | tee $MINT_DYNAMIC_TASKS/tasks.yaml

We showed implementing this Ruby script inline with the task definition, but it's more common to check the script in as a separate file, such as .mint/generate-tasks.rb and then call ruby .mint/generate-tasks.rb > $MINT_DYNAMIC_TASKS/tasks.yaml in your Mint task definition.

For more information, see the documentation on generating dynamic tasks in Mint.

Demo

Here's a 3 minute video on how easy it is to generate dynamic tasks in Mint.

Never miss an update

Get the latest releases and news about RWX and our ecosystem with our newsletter.

Share this post

Enjoyed this post? Pleas share it on your favorite social network!

Related posts

Read more on updates and advice from the RWX engineering team

See all posts
A Node.js Maintainer Lost Their Keys
CI/CD

A Node.js Maintainer Lost Their Keys

A Node.js maintainer lost the keys used to sign a publicly available release. For engineering teams running their CI/CD pipelines on RWX, everything was fine.

Jul 3, 2025
Read now
Preventing Cache Races
CI/CD

Preventing Cache Races

When two tasks have the same cache key, RWX will prevent cache races.

Jul 2, 2025
Read now
Mint is Now RWX
CI/CD

Mint is Now RWX

Given the impact of our CI/CD platform, we've decided to deprecate the Mint name, and rebrand the CI/CD functionality directly as RWX.

Jul 1, 2025
Read now