Ensuring Version Consistency Across All Environments

As a best practice, engineering teams should ensure that the same version of tools and runtimes are used across all environments.

As tools and languages mature, they generally do a good job of maintaining backwards compatibility. However, sometimes subtle bugs can arise from version differences.

We recently ran into a bug in Mint because of a version difference in Node.

We were developing on version 20.12.2 and calling filehandle.read which takes four arguments: (buffer, offset, length, position).

In 20.12.2, offset, length, and position have default values and only buffer is required.

However, we had a piece of our infrastructure that was running on 20.11.1 instead of 20.12.2. In that version of Node, those parameters do not have default values, which resulted in an unexpected runtime error.

Thankfully we have a thorough acceptance test suite which caught the bug in staging before it made its way to production. But we would have rather caught this in development. It’s not enough to use the same major version – everything should be running the same minor and patch version as well.

The best way to enforce version consistency is checking for it explicitly in a CI pipeline.

Using .tool-versions

Ideally, you should have a single source of truth for the versions that you’re using. However, you may need to specify the version in multiple places. In that scenario, you should check for consistency among the disparate files in a CI task.

At RWX, our engineering team uses asdf for development, so we use a .tool-versions file as the primary source of truth. We then run a Mint task that extracts values from .tool-versions and also ensures consistency in other files that need to reference the respective versions.

If you're not familiar with .tool-versions, our file looks like this.

1
2
3
golang 1.22.2
nodejs 20.12.2
pnpm 8.15.8

And then here's our Mint task.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
- key: tool-versions
  use: code
  run: |
    set -x
    grep '^nodejs ' .tool-versions | awk '{print $2}' | tee $MINT_VALUES/nodejs
    nodejs=$(cat $MINT_VALUES/nodejs)
    grep "nodejs-$nodejs-1" ami/task-server/provisioning/provision.sh
    grep "nodejs=$nodejs-1" bin/agent-devel/provisioning/provision.sh
    grep ""@types/node": "$nodejs"" package.json

    grep '^golang ' .tool-versions | awk '{print $2}' | tee $MINT_VALUES/golang
    golang=$(cat $MINT_VALUES/golang)
    grep "/go${golang}.linux-" bin/agent-devel/install-deps.sh

    grep '^pnpm ' .tool-versions | awk '{print $2}' | tee $MINT_VALUES/pnpm
    pnpm=$(cat $MINT_VALUES/pnpm)
    grep "pnpm@${pnpm}" bin/agent-devel/install-deps.sh
  filter:
    - .tool-versions
    - package.json
    - ami/task-server/provisioning/provision.sh
    - bin/agent-devel/provisioning/provision.sh
    - bin/agent-devel/install-deps.sh

The following lines extract the desired versions from .tool-versions and set the results as Mint values

1
2
3
grep '^nodejs ' .tool-versions | awk '{print $2}' | tee $MINT_VALUES/nodejs
grep '^golang ' .tool-versions | awk '{print $2}' | tee $MINT_VALUES/golang
grep '^pnpm ' .tool-versions | awk '{print $2}' | tee $MINT_VALUES/pnpm

We then use these values in the Mint tasks which install the respective languages or tools.

1
2
3
4
5
6
7
8
9
10
11
12
13
- key: node
  call: mint/install-node 1.0.6
  with:
    node-version: ${{ tasks.tool-versions.values.nodejs }}

- key: pnpm
  use: node
  run: npm install -g pnpm@${{ tasks.tool-versions.values.pnpm }}

- key: go
  call: mint/install-go 1.0.6
  with:
    go-version: ${{ tasks.tool-versions.values.golang }}

The grep lines make sure other files which reference the Node version are using the same version.

1
2
3
4
nodejs=$(cat $MINT_VALUES/nodejs)
grep "nodejs-$nodejs-1" ami/task-server/provisioning/provision.sh
grep "nodejs=$nodejs-1" bin/agent-devel/provisioning/provision.sh
grep ""@types/node": "$nodejs"" package.json

The lines which we're checking in the referenced files are:

1
sudo yum install nodejs-20.12.2-1nodesource --setopt=nodesource-nodejs.module_hotfixes=1 -y

1
sudo apt-get install -y nodejs=20.12.2-1nodesource1

1
2
3
"devDependencies": {
  "@types/node": "20.12.2"
}

Similarly the grep command for Go:

1
2
golang=$(cat $MINT_VALUES/golang)
grep "/go${golang}.linux-" bin/agent-devel/install-deps.sh

is checking this line:

1
curl -L https://go.dev/dl/go1.22.2.linux-amd64.tar.gz -o /tmp/go1.22.2.linux-amd64.tar.gz

And the grep command for pnpm:

1
2
pnpm=$(cat $MINT_VALUES/pnpm)
grep "pnpm@${pnpm}" bin/agent-devel/install-deps.sh

is checking this line:

1
sudo npm install -g [email protected]

The Mint task is fast to run, but it's also using a filter so that the majority of the time it will be a cache hit.

1
2
3
4
5
6
filter:
  - .tool-versions
  - package.json
  - ami/task-server/provisioning/provision.sh
  - bin/agent-devel/provisioning/provision.sh
  - bin/agent-devel/install-deps.sh

Read more on filtering files to produce cache hits in Mint.

Follow Along

We write a lot about software engineering best practices and CI/CD pipelines in Mint. Follow along on X at @rwx_research, LinkedIn, or our email newsletter