Blog Archives

Avoiding intermittent oranges

Writing tests which are resilient against intermittent failures is hard.  In the process of trying to fix a large number of intermittent orange bugs, I’ve found out that a large portion of them are just caused by mistakes in writing tests, and almost all of those mistakes fall into commonly reoccurring patterns.  It’s hard to avoid those mistakes, unless you know how they lead to intermittent oranges, and how to avoid them.

In order to share my experience about what type of patterns could cause a test to fail intermittently, I’ve gathered a list of common intermittent failure patterns on MDN, and I urge everybody who writes tests for the Mozilla project to go ahead and take a look at that list.  I think if the test writers and reviewers have those items on mind when creating or reviewing a test, we can reduce the amount of new tests which are susceptible to intermittent failures dramatically.

And please make sure to add to the list if you know of other such patterns that I’ve missed.

Tagged with: , ,
Posted in Blog

Assisted starring of oranges

As a good citizen in the Mozilla developer community, you need to watch the tree, and star any random oranges with bug numbers, and put a comment inside the relevant bugs with a link to the log of the orange in order to help debugging the problem.  That’s too much work, and worse, it’s repetitive and boring.  Last weekend, I got sick of it, and decided to hack Tinderboxpushlog to make this a bit easier.  That work is now deployed on Markus‘ instance of Tinderboxpushlog for you all to enjoy.

How to use it?

It’s easy.  Click on a machine name in a build.  A summary of the log appears.  If there are any suggestions available, they will be listed inside the summary, like below.

Bug suggestions inside the log summary

Here, Tinderboxpushlog has found three suggestions for the test failure.  It’s your job to go and read the bugs to figure out which one of the three this failure is related to.  Also, note that the third suggestion has been marked as strike-through, that’s because it’s a resolved bug (I chose to show the suggestions for bugs which have already been resolved because sometimes they need to be reopened, and sometimes there are not resolved on the branch that you’re watching.)  The status of each bug appears as a tooltip of its link.

After you choose which bug is relevant, and you want to star it, you press the Add a comment link as usual.  The dialog which appears should contain suggestions in addition to the usual fields, like below:

Suggestions in the Add Comemnt dialog

You can click on any of the suggestions, and Tinderboxpushlog will add that bug to the comment box.  From that point, you can modify the comment text if needed, and click Add Comment when you’re done.  The orange will be starred as usual, but something else happens as well:

TinderboxPushlog Robot making a comment on the bug

We have hired a robot whose only job is to watch the tinderbox, and go ahead and add a comment to the bug with the log link and a bunch of helpful info.  Please meet TinderboxPushlog Robot, or as his friends call him, tbplbot.  He’s not that smart, but he knows not to comment on the same bug about the same log twice.  I’m posting a picture of him due to popular demand:

TinderboxPushlog Robot

Are there any insider tips?

Just one.  If for some reason you don’t want to add a comment to the bug, you should avoid clicking on the bug link in the Add a Comment dialog (and type in the bug number if you want).  tbplbot is smart enough to honor your request and not comment in the bug.

Wait, I don’t get suggestions in my summary logs!

In order to maximize the performance, and minimize the chance of double-starring, we don’t show suggestions for already-starred logs.  Also, the suggestions are made by looking at the log summary for file names, and looking them up in Bugzilla, so they won’t work for things which typically don’t have those in the log summary, such as leaks and crashes.  If there is a file name, the log is not starred, and still no suggestions appear, there might be a good chance that there’s no bug on file for that orange.  But remember, you should go ahead and search Bugzilla yourself to make sure that the algorithm has not missed something.

I want to help!

Great!  Clone Markus’ repository, file a bug for what you want to do, and hack away!

That’s it!  Go and star some oranges.  Or better yet, try to fix a few of them!

Tagged with: , ,
Posted in Blog