Retry an erratic test.

In Enqueue project we do functional testing to make sure MQ transports work as expected. For that we need real brokers, such as Amazon SQS or Kafka. Sometimes tests fail not because of the bug at our side but because of the way a broker works.

For example, You cannot reliably consume a message as soon as you publish it to Amazon SQS. There is a chance it returns it but Amazon does not guarantee an immediate delivery, same goes to Kafka.

You might face similar problems while testing an interaction with 3rd party API which acts up from time to time. Paypal and Klarna Sandbox APIs were spotted by doing so.

Such erratic tests result in random failures on CI server, although, there is no error in your code. Obviously, If you rerun the build it might pass but you still have to spend time taking care of it, waiting for new results, digging into the code to find out why it has happened.

The solution is to instruct PHPUnit to rerun such tests in case of failure several times before completely giving up.

To make things easier we created RetryTrait. Once it mixed-in a test case, it adds class-level and method-level retry annotations. You could get it by installing enqueue/test package or copy-pasting the trait to your source code.

Here’s a test example:

Conclusion: You should do your best to write tests that do not require retrying. Though, there are specific cases where it might come in handy.