Flaky Tests API

See the documentation on authenticating to configure access to the Captain API.

You can use the flaky tests endpoint to identify your most flaky test cases.

https://cloud.rwx.com/captain/api/test_suites/flaky_tests

Currently, test_suite_identifier is required as a query string parameter. Remember to URL encode any values when passing them into the query string.

https://cloud.rwx.com/captain/api/test_suites/flaky_tests?test_suite_identifier=your-identifier-here

Example

A fully authenticated request with curl, assuming an access token is set in an RWX_ACCESS_TOKEN environment variable, will look like this:

curl \
  --header "Authorization: Bearer $RWX_ACCESS_TOKEN" \
  "https://cloud.rwx.com/captain/api/test_suites/flaky_tests?test_suite_identifier=your-test-suite-identifier"

Response

The API will respond with a JSON document containing the following fields:

  • results – an array of test cases
  • total_count – the total count of flaky tests
  • message – only included on error responses, which will also have a 4xx http response

results is limited to returning 10,000 test cases. If your test suite has more than 10,000 test cases with flaky occurrences, the API will return the 10,000 test cases with the most flaky test occurrences.

total_count will return the total number of test cases with flaky occurrences so that you can determine whether results were limited or not.

Result Structure

Each object under results contains the following properties:

  • file
  • flake_occurrence_count – the number of occurrences within the past 7 days
  • last_seen_at – may be null if the test case was previously flaky but not within the past 7 days
  • name
  • quarantined_test_occurrence_count – the number of times this test case has been quarantined
  • state – this will always be unresolved
  • test_case_id
  • under_quarantine – whether the test is currently quarantined

Example Response

{
  "results": [
    {
      "file": "feedback.spec.ts",
      "flake_occurrence_count": 0,
      "issue_id": "5a27b3f8-6604-4b49-a712-4e10afe40ddf",
      "last_seen_at": "2024-02-16T04:24:55.897Z",
      "name": "Allows the submission of feedback to the support email address",
      "quarantined_test_occurrence_count": 0,
      "state": "unresolved",
      "test_case_id": "24764cb8-971b-46a2-8938-2adc3f2bf265",
      "under_quarantine": false
    }
  ],
  "total_count": 1
}