Quantum Flow Engineering Newsletter #17

Next week, Nightly will switch to the 57 branch, beginning the development cycle of what will be the last train leaving the station towards Firefox 57.  Around 5 months ago, I started writing the first one of these newsletters, which of course was well past when the Quantum Flow project got started.  It’s probably a good time for a retrospective on the way that we have come so far.

During this time, many small and medium size performance projects were started.  Some are finished (and shipping even!) and some still ongoing, but the rate of progress has been quite astonishing.  I have tried to talk about as many of these as possible in some detail in the newsletter, and these are only one part of the overall performance work happening.  Here is a list of some of these projects with some overall status report:

  • We had a lot of reports of performance issues such as during page loads that were either caused or exacerbated by long running sync IPCs from the content to the chrome process.  We started a focused effort to deal with this problem, which started from restricting the addition of new synchronous messages and gradually working towards removing the existing ones that were slow according to telemetry data.  I’ve reported on the status of this work periodically, and it will probably be an ongoing effort for some more time, but we are well on the path of solving the majority of the severe issue that affects our users by Firefox 57 at this point.
  • The most important performance issues are the ones that affect the real users.  In the past we had built an infrastructure for reporting backtraces from hangs that users experience through telemetry so that we can diagnose and fix them called Background Hang Reports, but this setup hadn’t survived the passage of time.  We started to stand up some python scripts to process the data coming through the telemetry servers to start getting actionable data while starting to create an awesome new UI for it.  Many thanks to Michael Layzell and Doug Thayer for their great work on this so far.  Through this data, we have found and fixed a number of bugs.  The rate of discovery and fixing bugs from this data has been slower than I would have liked.  The reason is we didn’t have enough engineers to look through this data and extract actionable bugs out of it.  This process is still manual and very slow and time consuming.
  • We’ve kept up a rigorous process of continually measuring and tracking the performance of the browser in various workloads with the goal of identifying the most severe performance issues and eliminating them where possible.  In order to get help from the broad group of engineers and contributors, it’s important to communicate what issues we consider as the most critical, so we have created an active bug triage process to identify the most important bugs and you have all probably heard all about this by now.  🙂  This may sound like a lot of process, but after several months when I look back now at the rate of performance fixes that have landed as a result of this, I think this has been fairly effective.  I think there has also been a lot about this triage process that we could have done better, like maintaining more consistency in prioritizing bugs, communicating more clearly about the criteria being used, etc.  But time pressure and the sheer number of bugs to deal with forced the situation in a lot of cases anyway.
  • A few projects grew into their own independent parallel efforts.  One example is some of the initial bugs that we had filed in various UI components in the front-end code started to emerge patterns that seemed to warrant some mini-projects formed around them.  One example was synchronous layout and style flushes triggered by various code in the browser front-end (and sometimes by Gecko invoked by the front-end code), or various code such as timers going off at random time in the front-end code.  Chasing issues like this is now part of the Photon Performance project and is being actively worked on, and the difference this is making is quite noticeable in the performance of various parts of the UI.  Another example is reflow performance.  We had seen expensive reflows in many profiles and even though we didn’t have much concrete information about the sources of the problems, we reached out to the Layout team and asked for help.  That resulted into an effort to improve reflow performance, which is actively continuing.  Thanks to both the front-end and layout teams for leading these efforts!
  • We have continued to improve the Gecko Profiler.  It is really impressive how far this effort has come.  On the platform side, the profiler has become from unusable to 100% reliable on Win64.  The UI keeps getting improved, and is very effective to use both for developers and users who want to help us by submitting profiles.  We have also tried to work on some documentation for profiling using it.  Thanks to Markus Stange, David Major and Nick Nethercote who work on the back-end and Greg Tatum and Julien Wajsberg who work on the tool’s front-end.
  • Since about 2 months ago, we started actively focusing on improving on the Speedometer 2 benchmark.  Ben Kelly made some tweets this week about our progress on optimizing for this benchmark so far which are worth mentioning: “Firefox Nightly has gotten ~38% faster at speedometer v2 in the last 2 months.” and “Firefox Nightly 56 is ~80% faster at speedometer v2 compared to currently released Firefox 54.”  This is great!  But we aren’t done yet.

Firefox Nightly 56 is ~80% faster at speedometer v2 compared to currently released Firefox 54

In other performance related news this week, Stylo, our new parallel CSS engine written in Rust, is ready for testing on the Nightly channel.  I’ve been playing with it on my main browser profile for a while and so far I’ve been really liking what I’m seeing.  I have gone to some pages where restyling cost typically shows up in profiles, and have added “StyleThread” to the list of threads in the Gecko Profiler settings, and have watched the parallel restyles happening in profiles, it’s quite interesting.  Especially when you see magic such as the Rust rayon threads calling into C++ code without any kind of synchronization on a background thread and the safety of this is ensured through static analysis.  We are indeed living in the future!  It is so exciting that we’re probably going to be the first browser engine deploying a parallel CSS engine through the superpowers of Rust, deploying this technology into the hands of millions of people.

And now again it’s time for the most important part of these newsletters, taking a moment to say thanks to those who helped make Firefox faster last week.  Apologies to those of you who I’m forgetting to name here.

Posted in Blog Tagged with: , ,
19 comments on “Quantum Flow Engineering Newsletter #17
  1. idiotFour says:

    Is scrolling in the active tab supposed to be 100% smooth all the time after high priority queue for input events was landed? Or it still can be affected by background tabs?

    • ehsan says:

      That does help somewhat but smooth scrolling is a complex topic. It mostly depends on how you scroll (what input method you use to scroll, the mouse wheel, the scroll bar thumb, the keyboard, the touchpad), each one of these has different characteristics and we are trying to enable smooth scrolling for as many of these input methods as possible, although in some cases (for example scrolling with keyboard) we will also need some cooperation from the page you are visiting, so scrolling may end up being smoother on some sites than on others.

      • Joonas says:

        About scrolling smoothly: do I understand bug 1384708 correctly? Does it mean that Firefox now deliberately waits 1 extra frame before starting scrolling (16 ms, which is a VERY noticeable delay). Is this really the right compromise? I would (perhaps naively) think that always reacting to the user input ASAP is *very* desirable.


        “As this does delay the responsiveness of scrolling to user input by 1 frame, we may want to start by landing this change under a pref.”

        “See bug 1367770 comment 2 for an explanation of why we may want to do this.”

        • ehsan says:

          Hi Joonas,

          This isn’t my area of expertise, so what I say may be highly inaccurate, but the way I read bug 1367770 comment 2 (the rationale behind this change) is that the delay of 1 frame is forced upon us in this case due to the nature of the architecture of the way asynchronous scrolling works, where we have some information on the main thread and some information on the compositor thread and there is no fundamental way for both sides to always remain in sync, so one side has to lag behind the other. So the trade-off is between delaying by one frame versus scrolling synchronously which doesn’t result in a smooth scrolling experience (it’s how browsers used to scroll a few years ago which looked generally jittery.)

          Also, the delay of 1 frame is entirely not perceptible to the human eye. See http://citeseerx.ist.psu.edu/viewdoc/download?doi= for example. 🙂

        • Laurentiu says:

          I wonder if we’re not seeing the same thing. But in my case the delay is certainly longer than a frame, I’d guess 500ms – 1s.

          I sometimes get to press page down twice before Firefox scrolls. When it wakes it seems to process both presses at once. I’ve seen this on Linux x64, on a slower PC, possibly when under high disk or CPU load.

          @Ehsan Does APZ require more inter-thread coordination? I wonder if the OS scheduler is choosing other threads instead.

          • ehsan says:

            Hmm, I think we are definitely not seeing the same thing, what you are describing indeed sounds like a performance problem to me. We have a system that allows us to get direct information about what’s going on inside Firefox (what we call a performance profile) which any user can easily use, and I’m very interested to dig into the issue you are seeing if you can help record a profile or two for me so that I can have a look. That way we don’t have to guess about the source of the lag you’re observing (and hopefully we can fix it!) Is it OK if I contact you by email directly? (I see your address in the blog contact system, no need to post it here!)

          • ehsan says:

            To answer your question, yes our multi-process architecture (which is where we use APZ) does rely on more inter-thread communication, and I actually see the kind of issue you’re referring to in profiles from time to time, but one sign of those issues is they are rarely consistent. Those types of issues are also sometimes visible in the profiles I was mentioning before, so capturing and looking at such a profile is helpful with those cases too.

            There are a number of further improvements we are planning to make in this area to assist the OS to prioritize our threads better.

          • Laurentiu says:

            Yes, you can contact me, but note that I don’t have an easy way to reproduce that issue (it happens maybe once a day or even less often).

            (I’m replying to my post since there’s no Reply link on yours).

  2. Mayank says:

    Realistically, would FF57 be able to match Chrome on the SpeedometerV2 benchmark?
    ( V8 did some changes which has increased their score quite a lot: https://arewefastyet.com/#machine=17&view=single&suite=speedometer-misc&subtest=score )

    • ehsan says:

      It is hard to predict exactly where Firefox 57 would stand compared to Chrome on the Speedometer benchmark, since as you can see we are chasing a moving target. 🙂 All I can say is that we are in no way done yet, and have still quite a few more optimizations down the line being worked on. Also, let’s remember that there are releases after Firefox 57, so we need to keep that in mind as well!

      BTW that jump in Chrome’s scores I believe is due to them recently switching to clang on Windows.

      • Do you plan to switch to Clang as well then? When or why not?
        Do these improvements translate to better battery life? Currently had to switch to Edge for that.
        Thank you for making Firefox better.

        • ehsan says:

          Actually around 5 years ago a colleague and I started to try to port Firefox to compile with clang on Windows. This has been a massive amount of work, and we have made a lot of progress. In January 2016, for the first time I managed to make a full Firefox build with clang on Windows. Since then, with a lot of extra work, we have used this for static analysis of our Windows code, and we have ongoing work for AddressSanitizer support on Windows.

          When and how we will switch the builds that we distribute to our users to use clang isn’t clear yet, but we have already been leveraging clang on Windows for a while now. 🙂

          • Kshitij Chawla says:

            Thank you for the reply!
            Can you clarify on why changing Chrome’s compile tool chain would improve its benchmark scores?
            Also, any progress on reducing Firefox’s battery usage? Will using The Great Suspender or Tab Unloader help?

          • ehsan says:

            To answer your first question, I’m not quite sure. My rough guess is that LLVM’s x86 code generator generates code that runs that benchmark better than the code that MSVC manages to produce for some reason, whether or not this difference carries to general browsing usage requires investigation, which I haven’t done. Note that 64-bit builds of Chrome don’t show the same improvement as a result of the switch, so it is CPU architecture dependent.

            About battery usage, we did start Project Candle to reduce our power consumption, but the project is currently on hiatus due to the current focus on performance. I hope that we would get back to focus on it again in the future, but I don’t know of any specific plans (that being said, Mozilla is a big project, with lots of people so that may not mean much!) About those extensions, I have only heard of them, I don’t have any experience using them unfortunately.

  3. Laurentiu says:

    Thank you for working on this. I have one question, what happens after the Firefox 57 release? Will the Quantum Flow effort wind down or go into maintenance mode, or will it continue?

    • ehsan says:

      That is a good question! I don’t know, I’m not sure if anyone has really thought about it to be honest! We’ll probably take a look at where we are a bit closer to the release and evaluate what the right next step is.

  4. Great Work Ehsan ! Eyval 🙂

  5. wgianopoulos says:

    The problem here is Just Like Mr. Trump you have lost your base. Sorry to say that but I have switched to working on SeaMonkey because they at least appreciate my contributions rather than saying they do not support some direction from people who I feel don NOT understand the Mozilla mission.

    • ehsan says:

      I’m sorry, but I don’t really know what you are referring to. At any rate, I don’t appreciate being compared to the American head of state, and I respectfully request you refrain from such tone in the future. Thank you.