Filtering Files

RWX caches tasks based on the contents of the file system (docs on caching). However, some tasks may only need to use a subset of the files. When defining an RWX task, you can add a filter so that only the specified files will be present on disk. This will result in more frequent cache hits.

Example

tasks:
  - key: write-files
    run: |
      echo ${{ run.id }} | tee changes-every-time.txt
      echo static file contents | tee stays-the-same.txt

  - key: hash-files
    use: [write-files]
    run: sha256sum *.txt

  - key: hash-static-files
    use: [write-files]
    filter:
      - stays-the-same.txt
    run: sha256sum *.txt

The output of hash-files will look like this:

09ebb220c8a4997acd2b9f9f99aaf200d9861b689ffcc26a3cd10c15efead761  changes-every-time.txt
672939a0624c8c668ffa85ea007e600e3c4ba503dfcf00ca1b603b7cb5c6c56b  stays-the-same.txt

The output of hash-static-files will look like this:

672939a0624c8c668ffa85ea007e600e3c4ba503dfcf00ca1b603b7cb5c6c56b  stays-the-same.txt

And if you run these tasks again, write-files and hash-files will always produce a cache miss, whereas hash-static-files will always produce a cache hit.

Filter negation

You can negate any filter entry to exclude files from the filesystem. If every filter entry is negated, then all files that do not match a negative filter will be included. Otherwise, only the files that match positive filters will be included. For example, "!abc.txt" will include all files on the filesystem except abc.txt, but the filter [foo, "!foo/abc.txt"] will include only the files in the foo directory, except for foo/abc.txt.

If multiple filter entries match a file, the last filter entry to match the file wins. Thus [foo.txt, "!foo.txt"] will exclude foo.txt, but ["!foo.txt", foo.txt] will include foo.txt.

Filter patterns

RWX supports filter patterns for more advanced filtering use cases. Filter patterns are similar to Bash glob patterns. The following pattern features are supported:

  • *: match zero or more characters within a path segment. For example, foo/*.txt will match foo/a.txt and foo/b.txt, but not foo/c/d.txt.
  • **: match all path segments recursively. For example, foo/**/*.txt will match foo/a.txt, foo/b/c.txt, foo/d/e/f.txt and so on.
  • {}: match any of the comma-delimited patterns specified inside the braces. For example, foo/file.{js,ts} will match foo/file.js and foo/file.ts.

If you do not use filter patterns, then filter entries will only match exactly the file they specify. For example, foo.txt will match foo.txt but not bar/foo.txt. If you want to match all files in the workspace named foo.txt, use **/foo.txt.

Workspace

RWX uses the contents of the entire file system for determining the cache key. However, the filter only applies to the workspace directory, which is /var/mint-workspace. If any files change outside of the workspace, such as with system dependencies or configuration, it'll always result in a different cache key.