Are you tired of seeing sporadic test failures in your test suite? Do you wish there was an easy way to detect and fix these issues? Look no further than the –repeat flag in Pest! Currently, I am working on a big project with a very large team. One of the things that we often encounter is that flaky tests make it through the CI. This is bad because it can block the CI later on. Because of this, it is desirable to detect flaky tests with as little effort as possible.

What is a flaky test?

First, let’s define what a flaky test is. A flaky test is a test that sometimes passes and sometimes fails, without any changes to the code being tested. This can be caused by factors such as timing issues, concurrency issues, or external dependencies.

Flaky tests are bad because they can lead to false positives or false negatives in your test suite. False positives occur when a test fails even though the code is working correctly, while false negatives occur when a test passes even though the code is broken. Both of these scenarios can be dangerous because they can lead to developers ignoring failing tests, which can cause bugs to slip through the cracks.

Detecting flaky tests

This is where the –repeat flag comes in. When you run your tests with this flag, Pest will repeat each test multiple times, and only report a failure if the test fails on every repetition. This helps to weed out flaky tests by increasing the chances of detecting inconsistencies in the test results.

For example, let’s say you have a test that occasionally fails due to a timing issue. If you run the test with –repeat=10, and it fails on every repetition, then you know that it’s not just a fluke and there is an actual issue with the code. On the other hand, if the test only fails once or twice out of 10 repetitions, then you can be more confident that it’s just a flaky test and not a real issue that lives in your code. It will help you point out which tests are flaky, so you can fix them later on.

Using the –repeat flag can save you a lot of time and headaches in the long run by helping you catch flaky tests before they cause serious problems. So next time you’re suspecting a flaky test, consider using the –repeat flag to ensure the reliability of your tests.