Test results

By running your test suites through Captain inside of RWX, you can view and interact with richly formatted test failures, partition your tests into automatically balanced parallel tasks, track and resolve flaky tests, and retry test failures without also retrying tests that succeeded.

Running your tests with Captain

To use Captain inside RWX, install the captain CLI and then run your test framework through it. Make sure to output Captain-compatible results. The following example uses RSpec; find instructions for your framework here.

tasks:
  - key: captain
    call: rwx/install-captain 1.2.0

  - key: rspec
    use: captain # you'll likely also depend on your code, your Ruby dependencies, etc.
    run: |
      captain run captain-examples-rspec \
        --command "bundle exec rspec --format json --out rspec.json --format progress" \
        --test-results "rspec.json" \
        --print-summary

See Captain on RWX for more info on the other ways Captain can improve your test suites on RWX.

Viewing test results without Captain

If you don't want to run your test suite through Captain, you can still view richly formatted test results in the RWX UI by configuring a test results output. RWX uses Captain behind the scenes to parse these test results, so it supports all of the languages and frameworks that Captain supports. See the Captain docs for the full list.

There are two ways to provide your test results directly to RWX: by declaring outputs.test-results, or by writing test results directly to $RWX_TEST_RESULTS.

outputs.test-results

Rather than actually running tests, the following example downloads a test framework fixture file, but in real usage you'd configure your test framework to write the test results to a file.

tasks:
  - key: run-tests
    run: curl -sO 'https://raw.githubusercontent.com/rwx-research/captain/6c83b74c5ebbf115ec14cca65083d7f4ddea285b/test/fixtures/jest.json'
    outputs:
      test-results:
        - path: jest.json

Note that test-results accepts an array, so if your task is producing more than one test results file, you can specify all of them in the test results outputs.

RWX infers which parser to use, so you don't have to tell RWX what produced the file. You can still set the language and framework explicitly under an options key, which is worth doing when your framework writes a generic format like JUnit XML that Captain can parse but can't attribute to a specific framework.

tasks:
  - key: run-tests
    run: curl -sO 'https://raw.githubusercontent.com/rwx-research/captain/6c83b74c5ebbf115ec14cca65083d7f4ddea285b/test/fixtures/jest.json'
    outputs:
      test-results:
        - path: jest.json
          options:
            language: JavaScript
            framework: Jest

$RWX_TEST_RESULTS

As an alternative to declaring outputs.test-results, your task can write result files directly into the $RWX_TEST_RESULTS directory. RWX parses every file it finds there, so none of them need to be listed in the task definition.

tasks:
  - key: run-tests
    run: curl -s 'https://raw.githubusercontent.com/rwx-research/captain/6c83b74c5ebbf115ec14cca65083d7f4ddea285b/test/fixtures/jest.json' -o "$RWX_TEST_RESULTS/jest.json"

There's no way to explicitly specify the language and framework when writing directly to $RWX_TEST_RESULTS, so opt for declaring outputs.test-results whenever you need to specify those.