What's a flaky test?
It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.
When a test frequently fails in
a ~"master:broken" issue
should be created.
If the test cannot be fixed in a timely fashion, there is an impact on the
productivity of all the developers, so it should be placed in quarantine by
This means it will be skipped unless run with
bin/rspec --tag quarantine
Before putting a test in quarantine, you should make sure that a ~"master:broken" issue exists for it so it won't stay in quarantine forever.
Once a test is in quarantine, there are 3 choices:
- Should the test be fixed (i.e. get rid of its flakiness)?
- Should the test be moved to a lower level of testing?
- Should the test be removed entirely (e.g. because there's already a lower-level test, or it's duplicating another same-level test, or it's testing too much etc.)?
Quarantine tests on the CI
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
rspec-pg-quarantine(CE & EE)
Automatic retries and flaky tests detection
We also use a home-made
RspecFlaky::Listener listener which records flaky
examples in a JSON report file on
update-tests-metadata jobs), and warns when a new flaky example
is detected in any other branch (
flaky-examples-check job). In the future, the
flaky-examples-check job will not be allowed to fail.
This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/13021.
Problems we had in the past at GitLab
rspec-retryis bitting us when some API specs fail: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9825
Sporadic RSpec failures due to
- FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!):
spec/mailers/notify_spec.rbmore robust: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10015
Transient failure in
- Replace FFaker factory data with sequences: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10184
- Transient failure in spec/finders/issues_finder_spec.rb: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10404
Time-sensitive flaky tests
Array order expectation
- Be sure to create all the data the test need before starting exercize: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12059
- Bis: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12604
- Bis: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12664
- Assert against the underlying database state instead of against a page's content: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10934
- In JS tests, shifting elements can cause Capybara to misclick when the element moves at the exact time Capybara sends the click
- Triggering JS events before the event handlers are set up
Capybara viewport size related issues
- Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10411
Capybara JS driver related issues
- Don't wait for AJAX when no AJAX request is fired: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10454
- Bis: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12626
PhantomJS / WebKit related issues
- Memory is through the roof! (TL;DR: Load images but block images requests!): https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12003
- Flaky Tests: Are You Sure You Want to Rerun Them?
- How to Deal With and Eliminate Flaky Tests
- Tips on Treating Flakiness in your Rails Test Suite
- 'Flaky' tests: a short story
- Using Insights to Discover Flaky, Slow, and Failed Tests