Designing a Software Supply Chain for Security and Reliability

Software supply chains can impose severe risks related to security and reliability. While many companies require scrutinizing all code changes in peer review, they may be pulling in dependencies from open source packages built and released by a single person.

When we started building the package manager for Mint, our CI/CD platform, we wanted to make sure we got this right. In CI/CD, reliability issues can grind an engineering team to a halt. Not managing the supply chain thoughtfully can result in builds breaking all of a sudden, requiring intervention to get an engineering team moving again.

Security is even more important as CI pipelines have access to sensitive code, and CD pipelines have access to sensitive deployment keys to production environments.

Locking

For both security and reliability, package managers should be designed to always lock (or "pin") packages to a specific version.

Without this, an attack on the supply chain via a malicious change to a third-party package will rapidly propagate with upstream projects having no time or opportunity for recourse. It’s akin to having a remote code execution (RCE) vulnerability, giving an attacker direct access to your systems.

Security concerns aside, updates to packages can also cause issues with reliability. Locking ensures that things only break when the lock file is updated, giving an opportunity to detect the issue in a CI pipeline before the update is merged into a main branch.

Updating

While locking solves the security and reliability problems, it also creates a problem of its own caused by using stale dependencies. If an update to a package contains a security fix, then you want that fix to propagate quickly. Additionally, with the majority of projects running the latest version, it’s possible that older versions aren’t scrutinized as heavily for potential vulnerabilities. Running outdated software can often lead to security vulnerabilities.

Additionally, running old versions of packages can create maintenance problems if updates aren’t backwards compatible. Staying on the latest version helps detect migration issues quickly, giving a better opportunity to address them.

To address the risks of running outdated packages, package managers should also provide an easy way to upgrade packages to the latest version.

Semantic Versioning

To make the update path as smooth as possible, packages should ideally be released using semantic versioning. Although this is generally done by convention and best efforts, new package managers have the opportunity to enforce it by analyzing the interface of packages and ensuring the the interface is backwards compatible. Of course, there still could be non backwards compatible implementations, but having guarantees in compatibility around the interface is helpful to making the upgrade path as smooth as possible.

Risk Mitigation, not Elimination

These techniques do not eliminate the risk of supply chain vulnerabilities. It’s still possible that a malicious update can propagate into a project undetected. However, locking and controlled updates create an opportunity for detection and mitigation.

Mint’s Design

We designed the interface for Mint’s package manager to incorporate these principles. We call third-party packages leaves, and they’re always pinned to a specific version. In this example, we’re using version 1.0.7 of the mint/install-node leaf.

1
2
3
4
5
tasks:
  - key: node
    call: mint/install-node 1.0.7
    with:
      node-version: 20.13.1

Updating leaves is as easy as running mint leaves update. It’ll output a list of updates:

1
2
3
Updated the following leaves:
	mint/install-go 1.0.5 → 1.0.6
	mint/install-node 1.0.6 → 1.0.7

Additionally, Mint validates updates to leaves, ensuring that the updates are either backwards compatible or that they bump the major version number.

CI/CD with Security and Reliability in Mind

Mint powers the fastest builds and has the best developer experience in CI/CD. Learn more about what makes Mint different or schedule time to chat with our engineers.