RSpec

To use Captain with RSpec, you need to configure your test suite to output test results to a file and then tell Captain where to find those test results.

Getting started

RSpec can output test results to a file using the --format and --out flags. Configure Captain by creating a .captain/config.yaml file in the root directory of your repository:

test-suites:
  your-project-rspec:
    command: bundle exec rspec --format json --out tmp/rspec.json --format progress
    results:
      path: tmp/rspec.json

You can change your-project-rspec to any name you like, but we typically recommend using the name of your project followed by a dash followed by rspec. The command is the command you already use to run your test suite. Captain will invoke this command to run your tests. The example above shows what you might use if you use bundle exec rspec and want to store test results in tmp/rspec.json.

Once Captain is configured, you can run captain run your-project-rspec --print-summary. If you see your typical test output following by a captain block like this:

--------------------------------------------------------------------------------
----------------------------------- Captain ------------------------------------
--------------------------------------------------------------------------------

then you've configured everything correctly! You can now supercharge your test framework's capabilities. See below for configuring each of Captain's features.

Quarantining Tests

Traditionally, you might mark a test as pending or skipped to triage flaky or failing tests. With Captain, you can quarantine them instead. When only quarantined tests fail, Captain will still report your build as successful and exit with a 0 exit code. Unlike skipped tests, quarantined tests will continue to run, so you can still view their failure messages and see how frequently they are failing.

You can quarantine tests in OSS mode with captain add quarantine like so:

captain add quarantine your-project-rspec \
  --file ./spec/example_spec.rb \
  --description "Example is true"

See the identifying tests section of this page for more information on finding the file and description, and see the OSS quarantining guide for more information on managing quarantined tests in OSS mode.

Retrying Tests

You can configure Captain to automatically retry failed tests to help you determine if failing tests are flaky or are genuinely failing. To configure retries, update your .captain/config.yaml file like so:

test-suites:
  your-project-rspec:
    command: bundle exec rspec --format json --out tmp/rspec.json --format progress
    results:
      path: tmp/rspec.json
    output:
      print-summary: true
    retries:
      attempts: 2
      command: bundle exec rspec --format json --out tmp/rspec.json --format progress {{ tests }}

Once configured, Captain will invoke your original test command, check for any failures, and retry your tests however many times you've specified (in this example, two additional times) by templating the failures into the command specified by retries.command. The output.print-summary option is not required, but we've added it for convenience in understanding the overall results after the retries have been factored in.

Partitioning

Captain can optimally partition your test suite's files into multiple groups for execution on multiple CI nodes. Captain tracks your test file runtime so that it can balance each partition.

You can configure Captain to partition your tests by updating your .captain/config.yaml file like so:

test-suites:
  your-project-rspec:
    command: bundle exec rspec --format json --out tmp/rspec.json --format progress
    results:
      path: tmp/rspec.json
    output:
      print-summary: true
    partition:
      command: bundle exec rspec --format json --out tmp/rspec.json --format progress {{ testFiles }}
      globs:
        - spec/**/*_spec.rb

Captain will fill in the testFiles placeholder of your partition.command with the files resulting from expanding your configured partition.globs.

Running with Partitioning

Partitioning the files requires specification of the total number of partitions (--partition-total) and the specific, 0-based partition (--partition-index) being run. These values are used alongside the configured partition command and glob patterns.

captain run your-project-rspec --partition-index 0 --partition-total 8

This command partitions your suite into 8 groups and fills in the testFiles at the specified index -- in this case, partition 0.

When initially configuring partitioning, we recommend comparing the test count with and without partitioning to ensure all of your tests are being run (i.e. all of your test files are covered by the configured globs).

GitHub Actions Example

Partitioning works on any CI provider with parallelized or matrix jobs. Partition your tests by passing the appropriate values through to captain run.

Here is a full example using a GitHub Actions workflow:

run_tests:
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix:
      partition_index: [0, 1, 2, 3, 4, 5, 6, 7]
      partition_total: [8]
  steps:
    - uses: actions/checkout@v3
    - uses: ruby/setup-ruby@v1
      with:
        bundler-cache: true
    - uses: rwx-research/setup-captain@v1
    - name: run tests
      run: |
        captain run your-project-rspec \
        --partition-index ${{ matrix.partition_index }} \
        --partition-total ${{ matrix.partition_total }}

Usage with ABQ

Captain works with ABQ to support parallel test distribution while respecting the quarantine status of any tests. To use Captain with ABQ, the Captain CLI will wrap ABQ and ABQ will wrap your test suite. For example:

test-suites:
  your-project-rspec:
    command: bash -c "abq test --worker $ABQ_WORKER --reporter rwx-v1-json=tmp/abq.json -- bundle exec rspec"
    results:
      path: tmp/abq.json

ABQ_WORKER=0 captain run your-project-rspec

Usage with one-liner RSpec expectations

If you use the RSpec one-liner syntax like:

  it { is_expected.to eq(something.id) }

then your tests may not work well with Captain. Captain identifies tests based on their test descriptions. Since tests with the one-liner syntax do not have explicit test description, RSpec automatically infers a test description for the test. Unfortunately, the inferred test description includes the values of variables, which can lead to the inferred test description being non-deterministic. For example, in the case above, the inferred description may be is expected to eq "2172925c-cb2e-4338-bbd5-7aca9127fdb4" where the UUID changes every time the test is run (because something.id changes every time the test is run). As a result, features like retries and quarantining will not work with this test.

To resolve this issue, you can monkey-patch the generated description that RSpec uses to instead use the source code for the test. Install the method_source gem and then add the following at the end of your spec_helper.rb file after any RSpec.configure blocks.

module RSpecGenerateDescriptionMonkeyPatch
  def generate_description
    metadata[:block].source.strip
  end
end

RSpec::Core::Example.prepend(RSpecGenerateDescriptionMonkeyPatch)

With this in place, the previous example will generate the deterministic description it { is_expected.to eq(something.id) }, which will work correctly with all of Captain's features.

Identifying Tests

Captain uses framework specific "identity recipes" to identify the tests in your suite. These recipes are order dependent components extracted from native test framework output.

We use this identity to track the executions of a test over the course of their lifetime in your suite. This enables us to do things like flake detection, quarantining, and retries.

For RSpec, Captain constructs the identity by parsing out the file and description attributes.