Adaptive CI Pipelines with Dynamic Tasks

In most CI platforms, tasks needs to be defined statically when the run starts. This limits the flexibility in workflows, especially as projects become more advanced and yaml configuration gets more complicated. It's especially unfortunate to have this limitation in a dev tool built for engineers.

Engineers are adept at writing code to solve problems. That's why when we built RWX, we knew we wanted to support dynamic tasks. We wanted to give engineers the ability to define tasks at runtime, utilizing any programming language and logic that they wanted.

Dynamic tasks enable building CI pipelines which are adaptive based on things like the commit contents or data from external data sources. They unlock a lot of possibilities for building highly performant CI pipelines.

Task Data Structures

YAML gets a lot of criticism in its use for configuring infrastructure. It's usually not YAML itself that's the issue though. YAML is just a format for serializing data. The pain points around YAML configuration are usually more centered around

having to git push to run workflows, not being able to debug, and implementing complex logic in embedded expressions rather than just writing code.

To generate dynamic tasks in RWX, you can use any programming language to build an array of task definitions, and then serialize it as YAML.

Here's a simple example using bash:

1
tasks:
2
- key: generate-dynamic-task
3
  run: |
4
    cat << EOF | tee $RWX_DYNAMIC_TASKS/tasks.yml
5
    - key: dynamic-task-1
6
      run: echo this is the first dynamic task
7
    - key: dynamic-task-2
8
      run: echo this is the second dynamic task
9
    EOF

You probably don't want to use bash to generate dynamic tasks, although we've used it in some of our own workflows by leveraging envsubst with template files.

Here's a more practical example using Ruby;

1
tasks:
2
  - key: ruby
3
    call: ruby/install 1.0.9
4
    with:
5
    ruby-version: 3.3.1
6

7
  - key: generate-dynamic-tasks
8
    use: [ruby]
9
    run: |
10
      ruby -e '
11
        require "yaml"
12
        tasks = [
13
          {
14
            "key" => "dynamic-task",
15
            "run" => "echo this task was generated from ruby"
16
          }
17
        ]
18
        puts YAML.dump(tasks)
19
      ' | tee $RWX_DYNAMIC_TASKS/tasks.yaml

We showed implementing this Ruby script inline with the task definition, but it's more common to check the script in as a separate file, such as .rwx/generate-tasks.rb and then call ruby .rwx/generate-tasks.rb > $RWX_DYNAMIC_TASKS/tasks.yaml in your RWX task definition.

For more information, see the documentation on generating dynamic tasks in RWX.

Demo

Here's a 3 minute video on how easy it is to generate dynamic tasks in RWX.