Avoiding intermittent oranges
Writing tests which are resilient against intermittent failures is hard. In the process of trying to fix a large number of intermittent orange bugs, I’ve found out that a large portion of them are just caused by mistakes in writing tests, and almost all of those mistakes fall into commonly reoccurring patterns. It’s hard to avoid those mistakes, unless you know how they lead to intermittent oranges, and how to avoid them.
In order to share my experience about what type of patterns could cause a test to fail intermittently, I’ve gathered a list of common intermittent failure patterns on MDN, and I urge everybody who writes tests for the Mozilla project to go ahead and take a look at that list. I think if the test writers and reviewers have those items on mind when creating or reviewing a test, we can reduce the amount of new tests which are susceptible to intermittent failures dramatically.
And please make sure to add to the list if you know of other such patterns that I’ve missed.