TestCafe

To use Captain with TestCafe, you need to configure your test suite to output test results to a file and then tell Captain where to find those test results.

Getting Started

TestCafe ships with a built-in json reporter that produces output Captain can parse directly. Pass it via the -r / --reporter flag using the name:outputFile syntax to write the results to a file. You can list additional reporters (separated by commas) to keep your existing console output alongside the JSON report. Configure Captain by creating a .captain/config.yml file in the root directory of your repository:

test-suites:
  your-project-testcafe:
    command: npx testcafe chrome:headless tests/ --reporter spec,json:tmp/testcafe.json
    results:
      path: tmp/testcafe.json

You can change your-project-testcafe to any name you like, but we typically recommend using the name of your project followed by a dash followed by testcafe. The command is the command you already use to run your test suite. Captain will invoke this command to run your tests. The example above shows what you might use if you use npx testcafe chrome:headless tests/ and want to store test results in tmp/testcafe.json.

Make sure to include the --reporter flag with a :outputFile segment — without it, TestCafe writes the JSON to stdout and Captain won't find a file at the configured results.path.

Once Captain is configured, you can run captain run your-project-testcafe --print-summary. If you see your typical test output followed by a captain block like this:

--------------------------------------------------------------------------------
----------------------------------- Captain ------------------------------------
--------------------------------------------------------------------------------

then you've configured everything correctly! You can now supercharge your test framework's capabilities. See below for configuring each of Captain's features.

Identifying Tests

Captain uses framework specific "identity recipes" to identify the tests in your suite. These recipes are order dependent components extracted from native test framework output.

We use this identity to track the executions of a test over the course of their lifetime in your suite. This enables us to do things like flake detection, quarantining, and retries.

For TestCafe, Captain constructs the identity by parsing out the file and description attributes. The file is the path to the fixture file (as reported by TestCafe's JSON reporter), and the description is the fixture name and test name joined with a space.

Quarantining Tests

Captain makes managing flaky tests easier than ever. When a test is identified as flaky, you can quarantine the test without modifying it, so that if only those tests fail, Captain reports a success with a 0 exit code. Unlike skipped tests, quarantined tests will continue to run, so you can still view their failure messages and see how frequently they are failing.

You can quarantine tests in OSS mode with captain add quarantine like so:

captain add quarantine your-project-testcafe \
  --file "./tests/checkout.testcafe.ts" \
  --description "Checkout flow applies coupon code"

See the OSS quarantining guide for more information on managing quarantined tests in OSS mode.

Retrying Tests

You can configure Captain to automatically retry failed tests to help you determine if failing tests are flaky or are genuinely failing. To configure retries, update your .captain/config.yml file like so:

test-suites:
  your-project-testcafe:
    command: npx testcafe chrome:headless tests/ --reporter spec,json:tmp/testcafe.json
    results:
      path: tmp/testcafe.json
    output:
      print-summary: true
    retries:
      attempts: 2
      command: npx testcafe chrome:headless '{{ file }}' -T '{{ grep }}' --reporter spec,json:tmp/testcafe.json

Once configured, Captain will invoke your original test command, check for any failures, and retry your tests however many times you've specified (in this example, two additional times) by templating the failures into the command specified by retries.command. The {{ file }} placeholder is filled with the fixture file containing the failure, and {{ grep }} is filled with a regular expression matching the failed test names; Captain runs one retry command per fixture file with failures. The output.print-summary option is not required, but we've added it for convenience in understanding the overall results after the retries have been factored in.

Partitioning

Captain can optimally partition your test suite's files into multiple groups for execution on multiple CI nodes. Captain tracks your test file runtime so that it can balance each partition.

Configure partitioning in .captain/config.yml:

test-suites:
  your-project-testcafe:
    command: npx testcafe chrome:headless tests/ --reporter spec,json:tmp/testcafe.json
    results:
      path: tmp/testcafe.json
    output:
      print-summary: true
    partition:
      command: npx testcafe chrome:headless {{ testFiles }} --reporter spec,json:tmp/testcafe.json
      globs:
        - tests/**/*.testcafe.ts

Captain will fill in the testFiles placeholder of your partition.command with the files resulting from expanding your configured partition.globs.

Then partition across your CI provider's parallel jobs:

# .rwx/ci.yml

tasks:
  - key: code
    call: git/clone 2.0.7

  - key: node
    call: nodejs/install 1.1.14
    with:
      node-version: 20

  - key: deps
    use: [code, node]
    run: npm ci

  - key: captain
    call: rwx/install-captain 1.1.6

  - key: testcafe
    use: [deps, captain]
    parallel: 8
    run: captain run your-project-testcafe