Captain Cloud

Captain is an open source CLI that can detect and quarantine flaky tests, automatically retry failed tests, partition files for parallel execution, and more. It’s compatible with 15 testing frameworks.

Captain Cloud enhances the CLI experience by analyzing and managing the artifacts produced by Captain CLI. To enable Captain Cloud, set cloud.disabled to false in your Captain config.yaml file, and then generate and use an RWX access token.

Flake Detection

Running your tests through Captain CLI produces standardized test result artifacts. Captain CLI uses this context to enable enhancements to how your tests run.

💪 In OSS mode, you can manually manage flaky tests with Captain CLI using the add and remove commands.

🦾 When using Captain Cloud, test results are uploaded and analyzed as your test suites run, automatically detecting tests that produce inconsistent results. In addition, you can use the Captain Cloud web interface to triage flaky tests and help limit the impact on your team's builds.

Test Quarantine

Test Quarantining is one of the most effective strategies for handling flaky tests. Rather than deleting or pending a troublesome test, you allow it to continue running without failing your build.

💪 In OSS mode, you can manually manage your suite's quarantine using the add and remove commands. While this does alleviate some of the problems of flaky tests it doesn't address the often painful mean-time-to-resolution of flakiness.

🦾 When using Captain Cloud, before running your tests, Captain CLI first retrieves the managed list of quarantined tests. In addition, you can mark any test as quarantined directly from the Captain Cloud web interface, allowing your team to move forward immediately without any code change.

Targeted Retries

Another common approach is configuring automatic retries of tests. Often this is done as a uniform configuration across a test suite. Unfortunately, using a consistent number of retries for the entire test suite can be challenging to calibrate. If the number of retries is too low, it may not provide sufficient mitigation for flakiness. If the number of retries is too high, it can lengthen build times and result in wasted compute when a build has legitimate failures.

Captain CLI alleviates this by allowing you to configure both global policies and policies that target known flaky tests. In addition, you can configure circuit breakers that fail fast and consider maximum retry thresholds.

💪 In OSS mode you get the full power of Captain retry functionality with the caveat that you're on the hook for managing the list of flaky tests.

🦾 With Captain Cloud, Captain CLI is aware of the flaky tests automatically detected when analyzing your test result artifacts. In addition, the Captain Cloud web interface provides visibility into how many times your tests are being retried.

File Partitioning

Partitioning your test suite using test file timings is one of the most simple and effective techniques on your path to faster tests.

Captain CLI makes this incredibly easy. In OSS mode, you can use Captain CLI to write your test results to your local file system using the update command. Captain CLI reads these results to partition your suite into approximately even distributions.

💪 This technique is oftentimes an immediate win. However, over time, as your codebase changes, these file timings require regular maintenance to keep reasonably evenly distributed partitions.

🦾 Captain Cloud automates this by passively keeping track of how long your files take to run as you use Captain CLI. In addition, you can use Captain Cloud web interface's partition simulator to arrive at the optimal number of partitions for your test suite.

Web Interface

In Cloud mode, you can manage your organization's test suites using the Captain Cloud web interface. View test suite performance, test suite run details, and maintain your team's flaky and quarantine lists all in one convenient location.