Monthly Archives: October 2008

First bits of the Private Browsing patch landed

I’m pleased to announce that the first pieces of the Private Browsing feature have just landed on Firefox trunk!  This might not be something to get too excited about, since all of the landed code remains disabled for now, but it’s a big breakthrough for me, considering the fact that I’ve been playing with this code since January!  Some of you may even remember that this feature was cut off Firefox 3 because of the fact that it was too big to take at that stage of Firefox 3 development, it will be included in the final release of Firefox 3.1.

Based on my latest chat with Marcia on IRC, the current plan is to finish the work on the code for private browsing, and land all of the remaining pieces by the end of October 26.  After that, during the week of October 27, we are going to have a test week for the community to start testing the Private Browsing mode to make sure that it will be rock solid in the final release.  Of couse, every piece of this patch has automated unit tests (on which Aaron has been helping me) to make sure that the feature at least works according to the functional specification, but a feature of this size still needs lots of human testing as well.  Stay tuned for more updates on the schedule.

A little bit more of technical details follows.  The pieces landed at this stage include the nsIPrivateBrowsingService, and the private browsing implementation and unit tests for the Places, Cookies, Content Preferences, and Form History modules.  These include all of the code necessary to implement Private Browsing handling in those modules, but because the implementation has been designed to ignore the absence of the Private Browsing service if it’s not available (like the case of these pieces of shared code using in other applications than Firefox), nothing will change in the functionality of these modules.  Those who want to test the Private Browsing mode should still run try server builds that I post to the Private Browsing bug.

For the details of what was checked in, check out the links below (pun intended):

Posted in Blog Tagged with: , , ,

Testing the cache service

I was trying to write a unit test for my modifications to the cache service as part of the implementation of private browsing, and the experience wasn’t as smooth as I would have liked.  So, I decided to document my experience here.

Firstly, there doesn’t seem to be any automated unit tests for the cache service.  The only test which uses the cache service directly inside netwerk/test/unit is the test for bug 356133.  There are a bunch of stand-alone js and C++ tests residing in netwerk/test, but looking through the makefile, none of them seem to be built or run by default.  Oh, and I know that the cache service code is fairly old, tested, and stable, but still, no unit tests at all?  We’ve got to do better than that!

Moving on to writing the unit test anyway, the first step was to check if all three cache devices (memory, disk, and offline) are available.  So, I wrote the following code in order to test this:

var cs = get_CacheService();

var visitor = {
  QueryInterface: function (iid) {
    const interfaces = [Ci.nsICacheVisitor,
                        Ci.nsISupports];
    if (!interfaces.some(function(v) iid.equals(v)))
      throw Cr.NS_ERROR_NO_INTERFACE;
    return this;
  },
  visitDevice: function (deviceID, deviceInfo) {
    print(deviceID);
    return false;
  },
  visitEntry: function (deviceID, entryInfo) {
    do_throw("nsICacheVisitor.visitEntry should not be called " +
      "when checking the availability of devices");
  }
};

cs.visitEntries(visitor);

What I expected this to output was something like the following:

memory
disk
offline

But I was quite surprised to find out that the visit operation does not find any devices whatsoever!  After some time spent reading the source, I figured out that the cache service initializes the devices lazily.  In other words, when the cache service is initialized, none of the cache devices get created, and they would each get created on a per-use basis.  So, what I did was write some code to add a cache entry for each type of device, and then try to check the availability of the devices.  The code I wrote was based on the storeCache function, and looked like this:

function store_in_cache(aKey, aContent, aWhere) {
  var storageFlag, streaming = true;
  if (aWhere == "disk")
    storageFlag = Ci.nsICache.STORE_ON_DISK;
  else if (aWhere == "offline")
    storageFlag = Ci.nsICache.STORE_OFFLINE;
  else if (aWhere == "memory")
    storageFlag = Ci.nsICache.STORE_IN_MEMORY;

  var cache = get_CacheService();
  var session = cache.createSession("PrivateBrowsing", storageFlag, streaming);
  var cacheEntry = session.openCacheEntry(aKey, Ci.nsICache.ACCESS_WRITE, true);

  var oStream = cacheEntry.openOutputStream(0);

  var written = oStream.write(aContent, aContent.length);
  if (written != aContent.length) {
    do_throw("oStream.write has not written all data!\n" +
             "  Expected: " + aContent.length  + "\n" +
             "  Actual: " + written + "\n");
  }
  oStream.close();
  cacheEntry.close();
}

store_in_cache("cache-A", "test content", "memory");
store_in_cache("cache-B", "test content", "disk");
store_in_cache("cache-C", "test content", "offline");

So, all was well when testing this on a debug xpcshell, so I decided to run the code on an optimizied non-debug build of xpcshell as well (like I always do), and boom!  An NS_ERROR_FAILURE was being thrown on the nsICacheSession.openCacheEntry call.  I tested this again and again, but I kept getting the same error in the non-debug build.  After spending some time debugging the error, I found out that the culprit was this code.  What happened was the cache service tried to get the profile directory in order to find a place to store the disk-based cache files (which will of course fail in xpcshell environment where the profile is not accessible), and in debug builds, it reverted to using the current process directory since the profile directory was not accessible.  In non-debug builds, the code simply detected that there’s nowhere to store the cache files, so it reported an error which was thrown as an obscure NS_ERROR_FAILURE to the caller.  Providing the profile directory manually did the trick:

var dirSvc = Cc["@mozilla.org/file/directory_service;1"].
             getService(Ci.nsIProperties);
var provider = {
  getFile: function(prop, persistent) {
    persistent.value = true;
    if (prop == "ProfLD" ||
        prop == "ProfD" ||
        prop == "cachePDir")
      return dirSvc.get("CurProcD", Ci.nsILocalFile);
    throw Cr.NS_ERROR_FAILURE;
  },
  QueryInterface: function(iid) {
    if (iid.equals(Ci.nsIDirectoryProvider) ||
        iid.equals(Ci.nsISupports)) {
      return this;
    }
    throw Cr.NS_ERROR_NO_INTERFACE;
  }
};
dirSvc.QueryInterface(Ci.nsIDirectoryService).registerProvider(provider);

Another oddity I encountered was when trying to read from the cache.  A read from the cache should either lead to success, or failure by throwing NS_ERROR_CACHE_KEY_NOT_FOUND from nsICacheSession.openCacheEntry.  So, I happily wrote the following code, hoping that all would be well.

function retrieve_from_cache(aKey, aWhere) {
  var storageFlag, streaming = true;
  if (aWhere == "disk")
    storageFlag = Ci.nsICache.STORE_ANYWHERE;
  else if (aWhere == "offline")
    storageFlag = Ci.nsICache.STORE_OFFLINE;
  else if (aWhere == "memory")
    storageFlag = Ci.nsICache.STORE_ANYWHERE;

  var cache = get_CacheService();
  var session = cache.createSession("PrivateBrowsing", storageFlag, streaming);
  try {
    var cacheEntry = session.openCacheEntry(aKey, Ci.nsICache.ACCESS_READ, true);
  } catch (e) {
    if (e.result == Cr.NS_ERROR_CACHE_KEY_NOT_FOUND)
      return null;
    else
      do_throw(e);
  }

  var iStream = make_input_stream_scriptable(cacheEntry.openInputStream(0));

  var read = iStream.read(iStream.available());
  iStream.close();
  cacheEntry.close();

  return read;
}

But things were not that easy.  The problem was that inside the private browsing mode (where the disk and offline cache devices are turned off and disabled), trying to read an entry from the disk device lead to the NS_ERROR_CACHE_KEY_NOT_FOUND exception, while trying to read data from the offline device lead to NS_ERROR_FAILURE.  The fix was easy: just handle this error as another possible result value on the exception.

Finally, the whole test looked like this.

const Cc = Components.classes;
const Ci = Components.interfaces;
const Cr = Components.results;

var _PBSvc = null;
function get_PBSvc() {
  if (_PBSvc)
    return _PBSvc;

  try {
    _PBSvc = Cc["@mozilla.org/privatebrowsing;1"].
             getService(Ci.nsIPrivateBrowsingService);
    if (_PBSvc) {
      var observer = {
        QueryInterface: function (iid) {
          const interfaces = [Ci.nsIObserver,
                              Ci.nsISupports];
          if (!interfaces.some(function(v) iid.equals(v)))
            throw Cr.NS_ERROR_NO_INTERFACE;
          return this;
        },
        observe: function (subject, topic, data) {
          subject.QueryInterface(Ci.nsISupportsPRUint32);
          subject.data = 0;
        }
      };
      var os = Cc["@mozilla.org/observer-service;1"].
               getService(Ci.nsIObserverService);
      os.addObserver(observer, "private-browsing-enter", false);
    }
    return _PBSvc;
  } catch (e) {}
  return null;
}

var _CSvc = null;
function get_CacheService() {
  if (_CSvc)
    return _CSvc;

  return _CSvc = Cc["@mozilla.org/network/cache-service;1"].
                 getService(Ci.nsICacheService);
}

var cache_prefs = {
  "browser.cache.disk.enable" : null,
  "browser.cache.offline.enable" : null,
  "browser.cache.memory.enable" : null
};

function set_cache_prefs() {
  var prefs = Cc["@mozilla.org/preferences-service;1"].
              getService(Ci.nsIPrefBranch);
  for (var pref in cache_prefs) {
    cache_prefs[pref] = prefs.getBoolPref(pref);
    prefs.setBoolPref(pref, true);
  }
}

function restore_cache_prefs() {
  var prefs = Cc["@mozilla.org/preferences-service;1"].
              getService(Ci.nsIPrefBranch);
  for (var pref in cache_prefs)
    prefs.setBoolPref(pref, cache_prefs[pref]);
}

function setup_profile_dir() {
  var dirSvc = Cc["@mozilla.org/file/directory_service;1"].
               getService(Ci.nsIProperties);
  var provider = {
    getFile: function(prop, persistent) {
      persistent.value = true;
      if (prop == "ProfLD" ||
          prop == "ProfD" ||
          prop == "cachePDir")
        return dirSvc.get("CurProcD", Ci.nsILocalFile);
      throw Cr.NS_ERROR_FAILURE;
    },
    QueryInterface: function(iid) {
      if (iid.equals(Ci.nsIDirectoryProvider) ||
          iid.equals(Ci.nsISupports)) {
        return this;
      }
      throw Cr.NS_ERROR_NO_INTERFACE;
    }
  };
  dirSvc.QueryInterface(Ci.nsIDirectoryService).registerProvider(provider);
}

function check_devices_available(devices) {
  var cs = get_CacheService();
  var found_devices = [];

  var visitor = {
    QueryInterface: function (iid) {
      const interfaces = [Ci.nsICacheVisitor,
                          Ci.nsISupports];
      if (!interfaces.some(function(v) iid.equals(v)))
        throw Cr.NS_ERROR_NO_INTERFACE;
      return this;
    },
    visitDevice: function (deviceID, deviceInfo) {
      found_devices.push(deviceID);
      return false;
    },
    visitEntry: function (deviceID, entryInfo) {
      do_throw("nsICacheVisitor.visitEntry should not be called " +
        "when checking the availability of devices");
    }
  };

  // get the list of active devices
  cs.visitEntries(visitor);

  // see if any of the required devices was missing
  for (var i = 0; i < devices.length; ++ i) {
    if (!found_devices.some(function(v) v == devices[i])) {
      do_throw("Expected to find the \"" + devices[i] + "\" cache device, " +
        "but that device was not available");
    }
  }

  // see if any extra devices have been found
  if (found_devices.length > devices.length) {
    do_throw("Expected to find these devices: [" + devices.join(", ") +
      "], but instead got: [" + found_devices.join(", ") + "]");
  }
}

function get_device_entry_count(device) {
  var cs = get_CacheService();
  var entry_count = -1;

  var visitor = {
    QueryInterface: function (iid) {
      const interfaces = [Ci.nsICacheVisitor,
                          Ci.nsISupports];
      if (!interfaces.some(function(v) iid.equals(v)))
        throw Cr.NS_ERROR_NO_INTERFACE;
      return this;
    },
    visitDevice: function (deviceID, deviceInfo) {
      if (device == deviceID)
        entry_count = deviceInfo.entryCount;
      return false;
    },
    visitEntry: function (deviceID, entryInfo) {
      do_throw("nsICacheVisitor.visitEntry should not be called " +
        "when checking the availability of devices");
    }
  };

  // get the device entry count
  cs.visitEntries(visitor);

  return entry_count;
}

function store_in_cache(aKey, aContent, aWhere) {
  var storageFlag, streaming = true;
  if (aWhere == "disk")
    storageFlag = Ci.nsICache.STORE_ON_DISK;
  else if (aWhere == "offline")
    storageFlag = Ci.nsICache.STORE_OFFLINE;
  else if (aWhere == "memory")
    storageFlag = Ci.nsICache.STORE_IN_MEMORY;

  var cache = get_CacheService();
  var session = cache.createSession("PrivateBrowsing", storageFlag, streaming);
  var cacheEntry = session.openCacheEntry(aKey, Ci.nsICache.ACCESS_WRITE, true);

  var oStream = cacheEntry.openOutputStream(0);

  var written = oStream.write(aContent, aContent.length);
  if (written != aContent.length) {
    do_throw("oStream.write has not written all data!\n" +
             "  Expected: " + aContent.length  + "\n" +
             "  Actual: " + written + "\n");
  }
  oStream.close();
  cacheEntry.close();
}

function make_input_stream_scriptable(input)
{
    var wrapper = Cc["@mozilla.org/scriptableinputstream;1"].
                  createInstance(Ci.nsIScriptableInputStream);
    wrapper.init(input);
    return wrapper;
}

function retrieve_from_cache(aKey, aWhere) {
  var storageFlag, streaming = true;
  if (aWhere == "disk")
    storageFlag = Ci.nsICache.STORE_ANYWHERE;
  else if (aWhere == "offline")
    storageFlag = Ci.nsICache.STORE_OFFLINE;
  else if (aWhere == "memory")
    storageFlag = Ci.nsICache.STORE_ANYWHERE;

  var cache = get_CacheService();
  var session = cache.createSession("PrivateBrowsing", storageFlag, streaming);
  try {
    var cacheEntry = session.openCacheEntry(aKey, Ci.nsICache.ACCESS_READ, true);
  } catch (e) {
    if (e.result == Cr.NS_ERROR_CACHE_KEY_NOT_FOUND ||
        e.result == Cr.NS_ERROR_FAILURE)
      // a key not found error is expected here, so we will simply return null
      // to let the caller know that no data was retrieved.  We also expect
      // a generic failure error in case of the offline cache.
      return null;
    else
      do_throw(e); // throw the textual error description
  }

  var iStream = make_input_stream_scriptable(cacheEntry.openInputStream(0));

  var read = iStream.read(iStream.available());
  iStream.close();
  cacheEntry.close();

  return read;
}

function run_test() {
  var pb = get_PBSvc();
  if (pb) { // Private Browsing might not be available
    // Simulate a profile dir for xpcshell
    setup_profile_dir();

    // Enable all three cache devices
    set_cache_prefs();

    var cs = get_CacheService();
    do_check_neq(cs, null);

    // Start off with an empty cache
    cs.evictEntries(Ci.nsICache.STORE_ANYWHERE);

    // Store cache-A, cache-B and cache-C
    store_in_cache("cache-A", "test content", "memory");
    store_in_cache("cache-B", "test content", "disk");
    store_in_cache("cache-C", "test content", "offline");

    // Make sure all three cache devices are available initially
    check_devices_available(["memory", "disk", "offline"]);

    // Check if cache-A, cache-B and cache-C are avilable
    do_check_eq(retrieve_from_cache("cache-A", "memory"), "test content");
    do_check_eq(retrieve_from_cache("cache-B", "disk"), "test content");
    do_check_eq(retrieve_from_cache("cache-C", "offline"), "test content");

    // Enter private browsing mode
    pb.privateBrowsing = true;

    // Make sure none of cache-A, cache-B and cache-C are available
    do_check_eq(retrieve_from_cache("cache-A", "memory"), null);
    do_check_eq(retrieve_from_cache("cache-B", "disk"), null);
    do_check_eq(retrieve_from_cache("cache-C", "offline"), null);

    // Make sure only the memory device is available
    check_devices_available(["memory"]);

    // Make sure the memory device is empty
    do_check_eq(get_device_entry_count("memory"), 0);

    // Exit private browsing mode
    pb.privateBrowsing = false;

    // Make sure all three cache devices are available after leaving the private mode
    check_devices_available(["memory", "disk", "offline"]);

    // Check if cache-A is gone, and cache-B and cache-C are still avilable
    do_check_eq(retrieve_from_cache("cache-A", "memory"), null);
    do_check_eq(retrieve_from_cache("cache-B", "disk"), "test content");
    do_check_eq(retrieve_from_cache("cache-C", "offline"), "test content");

    // Restore cache device prefs to their default values
    restore_cache_prefs();
  }
}

The cache service really needs some unit test, and documentation love.  Nice things to see documented would be the lazy initialization of devices, and the fact that the cache service picks up the process directory in debug mode as the cache directory.  I guess all of these can be covered in some sort of tutorial to the cache service APIs.  Anyway, until such a turotial gets written, I hope this post would make the life of people who decide to use the cache service a bit easier.

Posted in Blog Tagged with: ,

Firefox in 2001

Try searching the 2001 index made available by google for Firefox.  What we know as Firefox today is way different than what we used to know as Firefox back then…

Posted in Blog Tagged with: ,

Experimental builds with the latest ACID3 patches

Today I finally made a try server build which includes patches to some bugs, each of which fixing a problem that the ACID3 test is checking for.  The build gives us 97/100 points on the ACID3 test, which is impressive.  Here is the list of the fixes that this build includes:

  • Fix to Bug 174351 – Encoding Errors aren’t treated as fatal XML errors
  • Fix to Bug 178258 – HTML parser ships <script> to implicit <head>, breaking document.forms or document.getElementById
  • Fix to Bug 216462 – (smil) Implement SVG (SMIL) Animation
  • Fix to Bug 302775 – extractContents doesn’t work if start and end node of a Range object is an attribute node [@ nsContentSubtreeIterator::Init]
  • Fix to Bug 454325 – Range.extractContents doesn’t clone partially selected nodes

You can download this build from here for all the three platforms.  Here’s a screenshot of the ACID3 test with a copy of this build under Windows:

The following is a report of the tests which are failing in this build:

Failed 3 tests.
Test 26 passed, but took 258ms (less than 30fps)
Test 69 passed, but took 3 attempts (less than perfect).
Test 77 failed: expected '4776' but got '5560' - getComputedTextLength failed.
Test 78 failed: expected '90' but got '0' - getRotationOfChar(0) failed.
Test 79 failed: expected '34' but got '33' - SVGSVGTextElement.getNumberOfChars() incorrect
Total elapsed time: 3.23s

The ACID3 tracking bug has details on the failing tests, and is also tracking the bugs filed in Bugzilla for fixing the remaining tests.

Oh, and another cool thing about this build (in case it’s not obvious already) is it includes SMIL support!  SMIL is a W3C recommendation for creating declarative animations, which is integrated into SVG.  When you check out this build, make sure to check out some SMIL tests available here.

Oh, and just to mention the obvious, neither of these patches are written by me; I just applied them on my mozilla-central hg clone, and pushed to the try server, with a simple patch I copied from Daniel Holbert’s patch to enable SMIL to build by default (because it doesn’t seem to be possible to pass a customized .mozconfig file to the try server to use in its builds, and SMIL support by default is turned off, unless –enable-smil is used in .mozconfig).

Posted in Blog Tagged with: , ,

Private Browsing progress

My work on the Private Browsing patch is soon going to enter a new stage.  Four of the modules that the patch is touching already have unit tests.  The only part of the patch which is not correctly implemented yet according to the recent changes in the functional spec is the download manager module, which needed a back-end change to support in-memory databases.  I’ve implemented that in another bug I filed to track it, and my patch there is waiting for review.

In a recent discussion with mconnor, we decided that it would be best to split up the patch according to the boundaries of the modules that it’s touching, and ask for review on each part separately.  I’m going to do this today.  There are four modules which already have unit tests and are ready for review:

  • Cookies
  • Content Preferences
  • Passwords Manager
  • Authenticated Sessions

The Places module is also nearly ready for review, thanks to Aaron Train, a Seneca college student who has volunteered to write some unit tests for Private Browsing.

The nifty thing is that once I get the necessary reviews for each module, it would be possible to land it, because the unit tests are designed such that they would pass if the Private Browsing service is not available.  I would still continue to publish monolithic patches in the Private Browsing bug, to make the lives of those who want to try out the full patch easier, so to reduce the amount of confusion, I’m going to mark the "for review" patches by naming them with this pattern: "[for review] module vn", where module is the name of the module and n is the revision of the module specific patch, initially set to 1 and incremented if a reviewer requires changes to the implementation of that module.

Stay tuned for future updates on the Private Browsing progress.  Like always, feedback in form of comments on this blog, bug 248970, or email/IRC notes are welcome.

Posted in Blog Tagged with: , , ,