Retrying tests with ABQ
ABQ understands that sometimes you need to re-run certain tests, or workers, in order to resolve flakes and unblock your team. ABQ supports automated and manual retries when you need them.
Automated test retries
Automated retries of tests can be a great way to unblock your team in the presence of flaky tests.
ABQ supports configuring automated retries for any supported test framework. You can specify the maximum number of times a test
should be retried with the
--retries flag in the ABQ CLI. A test will be
retried until its first success, or all its retries have been exhausted.
--retries flag is not specified, ABQ will not retry tests.
ABQ does not currently support configuring retry counts on a per-test basis.
An example with Ruby and RSpec
Consider the following flaky Ruby RSpec test
RSpec.describe do it "is flaky" do expect(rand(2)).to eq(1) end end
To run this test through ABQ with two retries, make sure you've installed ABQ's RSpec integration
--retries 2 when invoking ABQ.
abq test --retries 2 -- bundle exec rspec flaky.rb
This test run might look like
FF. --- is flaky: FAILED --- 1) is flaky Failure/Error: expect(rand(2)).to eq(1) expected: 1 got: 0 (compared using ==) # ./flaky.rb:6:in `block (2 levels) in
' (completed in 5 ms [worker 0]) --- is flaky: FAILED (attempt 2) --- 1) is flaky Failure/Error: expect(rand(2)).to eq(1) expected: 1 got: 0 (compared using ==) # ./flaky.rb:6:in `block (2 levels) in ' (completed in 5 ms [worker 0]) Finished in 0.69 seconds (0.00 seconds spent in test code) 1 tests, 0 failures, 1 retried
Indicating that after 2 failures, this flaky test ultimately passed.
Manual test retries
Retrying your CI test jobs may be necessary in the presence of new flakes, or
sporadic startup failures. ABQ supports retrying individual workers for a
run-id without re-running your whole test suite.
Retrying an individual worker will only retry the tests allocated to that worker in the first attempt. The test results produced in the worker retry will be recorded and reported as retries during test result aggregation.
For example, consider an RSpec/Ruby test suite run through ABQ with two workers:
# on worker machine 0 ABQ_RUN_ID=my-rspec-test abq test --worker 0 -- bundle exec rspec # on worker machine 1 ABQ_RUN_ID=my-rspec-test abq test --worker 1 -- bundle exec rspec
If only worker 1 fails, re-running the same command
# retry on worker machine 1 ABQ_RUN_ID=my-rspec-test abq test --worker 1 -- bundle exec rspec
will run only the tests originally assigned to worker 1, and subsequent calls to
abq report will aggregate test results executed by the only execution of
worker 0, and both executions of worker 1.