Test Results

RWX supports test results as a semantic output. When a task outputs test results, the RWX UI will display richly formatted test failures. Test failures in the UI can be much easier to understand compared to reading log output.

Example

Here's an example task. Rather than actually running tests, we're downloading a test framework fixture file from captain. In real usage, you'd configure your test framework to write the test results to a file.

tasks:
  - key: run-tests
    run: curl -sO 'https://raw.githubusercontent.com/rwx-research/captain/6c83b74c5ebbf115ec14cca65083d7f4ddea285b/test/fixtures/jest.json'
    outputs:
      test-results:
        - path: jest.json

Note that test-results accepts an array, so if your task is producing more than one test results file, you can specify all of them in the test results outputs.

Language and Framework

RWX uses captain to parse test results. Therefore, it supports all of the languages and frameworks that Captain supports. See the Captain docs for the full list. Captain will normally infer which parser to use, but you may need to specify the language and framework. The Captain docs for your respective framework will indicate if specifying is necessary, and if it is, you'll want to set those attributes under an options key along with your test results file.

tasks:
  - key: run-tests
    run: curl -sO 'https://raw.githubusercontent.com/rwx-research/captain/6c83b74c5ebbf115ec14cca65083d7f4ddea285b/test/fixtures/jest.json'
    outputs:
      test-results:
        - path: jest.json
          options:
            language: JavaScript
            framework: Jest

De-duplicating test results

The underlying test ID RWX uses for semantic test results is based on a combination of factors that vary by test framework, but usually includes the test name/description and file location.

In some cases, there may be multiple tests that share the same computed test ID, for example those that are defined inside a loop (and therefore have the same name/description and file/line number).

When this happens, duplicates will be filtered out from the RWX UI, but targeted retries and quarantining may not work as expected. It is strongly encouraged to ensure all tests in your test suite have unique names/descriptions.

If you are running your tests through Captain, you can configure Captain to fail when it detects duplicate computed test IDs.