Many of the patches that we write are fixes to things that have broken as a result of a change, often known as regressions. An important aspect of a high quality release is for us to be able to identify and fix as many of these regressions as we can for each release, and this requires collaboration between people who file bugs, those who triage them, people who fix them, and of course the release managers.
As engineers, one of the things we can do when fixing a bug is to double check whether the bug was introduced as a result of a recent code change, and based on that decide whether the fix needs to be backported to older branches. In this post, I’m going to talk about how I usually do this.
Identifying the source of a bug
Sometimes it is clear from the description of a bug that the bug didn’t exist previously, as is often the case for severely broken scenarios. It’s a good practice to always ask yourself “did this used to work?” If the answer is yes and you have a way to reproduce the bug, then we need to figure out what code change broke things in the first place. We have a really helpful tool called mozregression which allows you to bisect the history of the codebase to find the offending code change. There is documentation available for using the tool, but to summarize: it walks you back in the history using a binary search algorithm to allow you to relatively quickly find out the specific code change that introduced a regression. This tool handles downloading the old Firefox versions and running them for you – all you need to do is to try to reproduce the bug in each Firefox instance it opens up and then tell the tool whether that version was good or bad. Mozregression also handles the creation of new profiles for each build so that you don’t have to worry about downgrades when you go from one build to another, or the impact of leaving the profile in a dirty state while testing during bisection. All you need to have at hand to start using it is knowing which version of Firefox shows a bug, and which version doesn’t. (When I don’t know when a bug first appeared, I usually use the –launch command to run a super old version to test whether the problem exists there to find a good base version; as it doesn’t matter much how wide the regression range you start with is.)
But sometimes there are no specific steps to reproduce available, or it may not be obvious that a bug is a regression. In those cases, when you have a fix for the bug at hand, it would be nice to take a few extra minutes to look at the code that you have modified in the fix to see where the code changes are coming from. This can be done using a blame/annotate source code tool. These tools show which changeset each line of code is coming from. Depending on your favorite revision control system (hg or git) and your favorite source code editor/IDE (Vim, Emacs, Eclipse, VS Code, Sublime, etc.) you can find any number of plugins and many online tutorials on how to view annotated source code. An easy way, and the that I use most of the time these days is looking at annotations through Searchfox. But you may also want to familiarize yourself with a plugin that works with your workflow to assist you in navigating the history of the code.
For example, when you hover the left-hand sidebar (I’ve sometimes heard this referred to as “the gutter”) when viewing a file in Searchfox, you will see an info box like this:
There are a few helpful links here for following the history of the code. “Show earliest version with this line” is useful to look at the revision of this file which introduced the line you hovered. “Show latest version without this line” is helpful to look at the parent revision of this file with respect to the aforementioned changeset. Using these two links you can navigate backwards in history to get to the point in time when a part of the code was first introduced. Often times you need to walk several steps back in the history before you get to the correct revision since code can be moved around, re-indented, modified in slight ways that don’t matter to you at the time, etc.
Once you get to the final version of the file that introduced the code in question, you will see something like this at the top of the page:
This shows you the changeset SHA1 for the problematic code in question (I picked a completely arbitrary changeset here which as far as I'm aware has not caused any regressions for demonstration purposes!) By clicking on the changeset identifier (bc62a859 in this example), and following the “hg” link from there, you can get to a table showing you which release milestone it got landed in:
This information is also available on the bug (which you can get to by clicking the link with the bug number next to “bugs” - 1358447 in this example):
Marking the bug as a regression
Once you identify the version of Firefox that first had the broken code checked into it, it is helpful to communicate this information to the release management team. Doing this involves a simple three-step process:
- Add the regression keyword to the bug if it doesn’t have it already.
- Based on the version of Firefox that was first impacted by the bug found in the previous step, under Firefox Tracking Flags, set the Tracking flag for the respective versions to “?”.
- Similarly, set the Status flag for the affected versions to “affected”, and for the older version(s) to “unaffected”. Bugzilla will help you fill out a comment section describing why this tracking is needed to the release management team.
