Quantum Flow Engineering Newsletter #21

We’re now about mid-way through the Firefox 57 development cycle.  The progress of Quantum Flow bugs has been steady, we now have 65 open [qf:p1] bugs at the time of this writing and 283 fixed bugs.  There are still more bugs being flagged for triage constantly.  I haven’t really spoken much about the triage process lately and the reason is that it has been working as per usual and the output should be fairly visible to everyone through our dashboard.

On the Speedometer front, if you are watching the main tracking bugs, the addition of new dependencies every once in a while should be an indication that we are still profiling the benchmark looking for more areas where we can think of speedup opportunities.  Finding these new opportunities has become more and more difficult as we have been fixing more and more of the existing performance issues, which is exactly what you would expect working on improving the performance of Firefox on such a benchmark workload.  Of course, we still have ongoing work in the existing dependency tree (which is quite massive at this point) so more improvements should hopefully arrive as we keep landing more fixes on this front.

I realize that I have been quite inconsistent in having a performance story section in these newsletters, and I hope the readers will forgive me for that!  🙂  But these past couple of weeks, Jan de Mooij’s continued effort on removing getProperty/setProperty JSClass hooks from SpiderMonkey made me want to write a few sentences about some useful lessons we have learned from performance measurements which can hopefully be used in the future when designing new components/subsystems.  Often times when we are thinking of how to design software, one can think of many extension points at various levels which consumers of the code can plug into in order to customize behavior.  But many such extension points come at a runtime cost.  The cost is usually quite small, we may need to consume some additional memory to store more state, we may need to branch on some conditions, we may need to perform some more indirect/virtual calls, etc.  The problem is that usually this cost is extremely small, and it can easily go unnoticed.  But this can often happen in many places, and over time performance issues like this tend to creep in and hide in corners.  Of course, usually when these extension points are added there are good reasons for creating them, but it may be a good idea to ask questions like “Is this mechanism too high level of a solution for this specific problem?”, “Is the runtime cost paid for this over the years to come justified to solve the issue at hand?”, “Could this issue be solved by adding an extension point in a more specialized place where the added cost would only affect a subset of the consumers?”, etc.  The reality of software engineering is that in a lot of cases we need to trade off having a generic, extensible architecture in our code versus having efficient code, so if you end up choosing extensibility, it’s a good idea to ensure you have had the performance aspects in mind.  It’s even better if you document the performance concerns!

And since we touched on this, now may be a good time to also take a quick moment to call out another issue which I have seen come up on some of the performance issues we have been looking into in the past few months.  That is the death by a thousand cuts performance problems.  In my experience, many of the performance issues that we need to deal with, when profiled turn out to be caused by only a few really badly performing parts of the code, or at least are due to a few underlying causes.  But we also have no shortage of the other kind of performance issues which are honestly much more difficult to deal with.  The way things work out in the opposite scenario is you look at a profile from the badly performing case, you narrow down on the section of the profile which demonstrates the issue, and no matter how hard you squint, there are no major issues to be fixed.  Rather, the profile shows many individual issues each contributing to a tiny portion of the time spent during the workload.  These performance issues are much harder to analyze (since there are typically many ways you can start approaching it and it’s unclear where is a good place to start) and they take a much longer time to result in measurable improvements, as you’d need to fix quite a few issues in order to be able to measure the resulting improvement.  For a good example of this, please look at the saga of optimizing setting the value property of input elements.  This project has been going on for a few months now, and during this time the workload has been made faster by more than an order of magnitude, but still if you look at each of the individual patches that have landed, they look like micro-optimizations, and that’s for a good reason, because they are.  But overall they add up to significant improvements.

Before closing, it is worth mentioning that the ongoing performance work isn’t suddenly going to stop with the release of Firefox 57!  In fact we have large performance projects which are going to get ready after Firefox 57, and that is a good thing, since I view Firefox 57 not as an ultimate performance goal, but as a solid performance foundation for us to start building upon.  A great example is the Quantum Render project which has been going on for a long time now.  This project aims to integrate the WebRender component of Servo into Firefox.  This project now has an exciting newsletter, and the first two issues are out!  Please take a moment to check it out.

And now it is time to take a moment to thank the contributions of those who helped make Firefox faster last week.  As usual I hope I’m not forgetting any names!

Posted in Blog Tagged with: , ,
19 comments on “Quantum Flow Engineering Newsletter #21
  1. Idiot says:

    Hello!
    Thank you for the summary! The bug that allows to decrease the process priority of content processes that are not running a foreground tab landed two months ago:
    https://bugzilla.mozilla.org/show_bug.cgi?id=1366356

    However, according to comment 19, “We need the platform-specific part. I wrote the linux one. We need the windows and mac ones.” When will Windows-specific part land in Nightly?

    I am asking because I hope this functionality will improve performance in my use case because I often load many bookmarks at once.

    • ehsan says:

      As you have noted, the Windows part of it hasn’t landed yet. I don’t know anything about the plans around doing this on Windows, but I asked a question about it to start a conversation going on it.

      However if you are noticing a performance problem in your use case, I encourage you to file a bug about it. Chances are that the performance issue is actually caused by other factors that may be at play, and filing a bug will allow the right engineers to look into the specifics of the issue you’re observing. It is through these bug reports that we can make Firefox faster, so please help us by providing the invaluable information about what’s not working well!

      • IdiotFour says:

        Thank you very much!

        What about performance problem, I have browser.sessionstore.restore_on_demand set to false. After I restore previously saved session, Firefox starts to load 3 tabs at once until all open tabs are loaded. The problem is that while those 3 tabs are loading in the background, scrolling fps in the current tab drops quite often. In other words, scrolling animation stutters. Once all tabs are loaded and there is nothing loading in the background , scrolling fps is rock solid at 60 frames per second.

        I have it i5 2300 and HDD instead of SSD so maybe my hardware is the bottleneck. Saved sessions are also pretty large – 150-200 tabs.

        • ehsan says:

          Hello there, sorry for the late reply!

          Firstly, setting browser.sessionstore.restore_on_demand to false isn’t recommended. Doing that will give you poor performance, as that pref exists as a performance optimization. But more importantly if you only have three tabs that are loading you shouldn’t be seeing scrolling lag, but I’m a bit confused since further down you mention that your session is fairly large (150-200 tabs). If you have that large of a session, with that pref change, then you’re basically in an unsupported configuration of Firefox and your change in the default configuration is the cause of the performance issue you’re observing, please consider toggling browser.sessionstore.restore_on_demand back to true. 🙂

          Otherwise, if you are really seeing this issue when restoring only 3 tabs, I can help you record some information about what’s happening inside Firefox to help uncover the reason behind this performance problem. Please let me know!

          • IdiotFour says:

            Thank you very much!
            Surprisingly, I’ve just reproduced these fps drops in a new profile with just 51 tab. Here is what I did: create new profile, set the following config prefs:

            browser.sessionstore.restore_on_demand false
            //scrolling settings
            general.smoothScroll.mouseWheel.durationMaxMS 120
            general.smoothScroll.mouseWheel.durationMinMS 60
            mousewheel.min_line_scroll_amount 23

            //disable disk cache, my might be too slow
            browser.cache.disk.enable false
            browser.cache.disk.capacity 0
            browser.cache.disk.smart_size.enabled false
            browser.cache.disk_cache_ssl false

            Install latest uMatrix WE beta from AMO, block everything except for .css and images (that means scripts were blocked)

            Open 50 tabs in the 1st window, open 1 tab with a long web page in the second window (that’s the active tab)

            Close Nightly, launch Nightly, restore session, start scrolling the web page in the active tab (~1 mouse wheel click per second)

            I’ve recorded a performance profile using Shift-F5, screenshots here: http://imgur.com/a/uTBEJ
            Stylo was enabled by default, Kaspersky was running in the background all this time, I’ll try to record a profile without it. CPU/Memory/HDD usage never reached 100% when recording the profile, as far as I am aware.

            I would like to record more information about what’s happening inside Firefox to help uncover the reason behind this performance problem. I guess the next step is profiling with the Gecko Profiler? Is this article up to date?
            https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler

          • ehsan says:

            Yes, the last link that you have found is up to date, and it describes how to capture a profile with the Gecko Profiler. However, I wasn’t guessing before when I recommended turning browser.sessionstore.restore_on_demand back to true. 🙂 That is the reason behind the performance issue you are experiencing. Part of the work that we do to load web pages happens on the UI thread, because the content processes offload some of their work back to the parent process.

            Also, the other changes that you have made to the default settings are probably harmful to the performance of Firefox. For one thing, disabling the local HTTP cache is certainly harmful. The networking team is actually right now experimenting with a feature to enable loading resources that you have in your local cache from the network if your local disk is slower than the network without the user having to manually toggle things like this. I’m personally not familiar with the impact of the changes you have made to smooth scrolling settings but given the fact that you’re observing scrolling issues, it is possible that those changes may be contributing to those issues. Sometimes these values have very intricate interactions with a lot of different settings in other parts of Firefox and changing one part makes Firefox act in a mode that developers never really test, and will cause performance issues that will probably never will be diagnosed and fixed. So our recommendation to users always is, never modify about:config settings unless you know exactly what the impact of the modifications are. Honestly I have even shot myself in the foot by toggling something in about:config for testing and never turning it back off because I forgot and broke something in my own Firefox instance as a result. 🙂

          • IdiotFour says:

            I’ve recorded a profile with uMatrix and Kaspersky disabled using Gecko profiler: https://perfht.ml/2eyabxd

            That means no addons were enabled except for the profiler, only the following config prefs/settings were modified:

            browser.sessionstore.restore_on_demand false
            //scrolling settings
            general.smoothScroll.mouseWheel.durationMaxMS 120
            general.smoothScroll.mouseWheel.durationMinMS 60
            mousewheel.min_line_scroll_amount 23

            //disable disk cache, my might be too slow
            browser.cache.disk.enable false
            browser.cache.disk.capacity 0
            browser.cache.disk.smart_size.enabled false
            browser.cache.disk_cache_ssl false

            There is one scrolling fps drop, I tried to stop recording right after that bug pressed Ctrl-Tab accidently. Please don’t spend too much time analyzing the profile because I could do something wrong. And please confirm you’ve received the profile.

          • ehsan says:

            I took a look at the profile, I think my recommendation is still reverting your changes to these prefs and trying again. One thing that your profile is showing is that there’s some thread starvation happening, which means your system is under too much load with too many threads competing to get CPU time. Reducing the number of concurrent page loads should probably help with that also.

          • IdiotFour says:

            I can’t reply to your last comment so I’ll leave a comment here. Thank you very much for taking your time and looking at my profiles! Now I am gonna upgrade my computer and wait until the second wave of major architectural changes lands in Nightly (Quantum DOM scheduler, WebRender, etc). If it doesn’t help, I’ll just pay someone to figure out what’s going on.

            I’ve also tried to create a new profile with the only modified pref browser.sessionstore.restore_on_demand false, uMatrix installed, KIS disabled. Here is the profile just in case: https://perfht.ml/2eSih0l
            Shift-F2 still shows frequent fps drops to 10-20fps but they are less noticeable for some reason. Maybe it is just scrolling animation that is buggy?

            And here is one more profile with disabled cache and modified scroll settings, fps drops were much better visible in this one so maybe this profile will better show what’s going on: https://perfht.ml/2eyktgz

            Thank you very much!

          • ehsan says:

            Hello again!

            As I have said a few times already, my advice to you in order to fix the performance issue you are seeing is to revert the changes to the prefs you are making. That being said, I noticed a performance issue in the second profile you submitted which I filed. Hope this helps.

  2. jeyoung says:

    > Ming-Chou Shih enabled coalescing mousemove events to once per refresh cycle. This feature helps performance by dispatching fewer mousemove events on pages which have expensive mousemove handlers, and is shipped in Chrome recently. It is currently disabled behind a preference for testing.

    This is just begging for us to try it out, isn’t it? 🙂 Anyway, it’s now enabled on my Nightly. Will see if there is any perceptible difference.

    • ehsan says:

      Yes, thank you! Also having these features enabled helps us by sending crash reports and telemetry even if you don’t see perceptible performance differences. 🙂

  3. user says:

    Looks nice, but I see a pattern here – almost all of these improvements could be implemented without killing support for “legacy” extensions.

    • ehsan says:

      These improvements are only a part of the work that is going into Firefox 57. Another huge aspect of it is switching all of our user base to use the multi-process browser architecture that we call “e10s-multi“. Also, the old style extensions cause a lot of performance issues for our users which we have no good way to fix. I have talked about the synchronous IPC messages generated by add-ons in my previous newsletters several times. A lot of careful thought has been put into how to solve these issues while maintaining backwards compatibility with our extensions ecosystem but it has been clear for a long time that isn’t a viable option for technical reasons because the old APIs that these extensions relied on were never designed to be used in a multi-process architecture. This is why we have invested quite heavily in developing a cross-browser extension API and have been working with our add-on developer community for a long time now to help them to transition to this new API with the goal of keeping as many of our existing extensions as possible. While change is hard, and nobody likes breaking backwards compatibility, we need to remember that the existing ecosystem isn’t without its problems, and sometimes the technical challenges in front of us dictate a solution that’s pragmatic and focused on maintaining a healthy ecosystem in the future, even if it means a rough time during the transition. I hope that these newsletters at least show that the technical decisions we make aren’t made arbitrarily and hopefully you trust that if there was a way to maintain backwards compatibility with old style extensions, we would have considered doing so.

  4. klop*cz says:

    Thank you for the report. It’s very nice and fun to read it every week. Please keep continuing of performance improvements in Fx, because it’s worth it. Sometime I feel, that one certain action in Fx is faster than before after daily update of Nightly. Actually I can’t remember how or if Fx was slow before, because Nightly runs like “Devilfox” 😀

  5. voracity says:

    Just getting back to you on the scrolling choppiness issue I reported to you in newsletter #18. Here is the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1394193

  6. Omega says:

    How much of these changes actually help the Mobile side of things. It doesn’t seem like Mobile Firefox is getting any faster.

    • ehsan says:

      A lot of these improvements are made to the shared code that we also use on Firefox for Android. The exact answer to your question is very complex and can be given on a case by case basis. But for now most of our focus is catching up on desktop performance for the Firefox 57 release.