When the IonMonkey project merged their history to mozilla-central on Tuesday, my scripts responsible for updating the mozilla-central git mirror happily picked up that huge merge of more than 2000 changesets, and started converting them to the git commit. And a few hours later, I was notified that my git mirror is no longer updating. I ssh'ed into the box and tried to do a git push, and saw git complaining about missing commits. And I was sad.
Having never had to fix a corrupt git repository before, I started to run git fsck, and it indeed told me about commits with non-existing parents and blobs. I tried removing those commits and blobs and run hg gexport again to get hg-git to reconvert them, only to find out that now the parent of the commit in question has a missing parent commit. And things would just explode when you got to a merge point where your ancestor line would split into two. I spent a lot of time to try to recreate the missing commits (and also recreate the missing blobs by finding the correct version of the file and then run git hash-object -w on it) manually, and partly by the assistance of a script, but I finally got to a point where hg-git would not be able to recover any more missing commits. And then I was mostly stuck. I suspected either a bug in Mercurial or hg-git and tried to update them both to the latest versions but that didn't help either, so I sent a mail to dev.platform yesterday explaining that I don't have more time to try to fix this problem.
And then, John Schoenick pinged me on IRC and explained to me what could be going on. Mercurial has the notion of multiple heads per branch, and that doesn't really map well to git, where all commits need to be referenced by either a branch or a tag. hg-git would happily convert all of those changesets to git commits, and if something would trigger git gc at some point, it would go ahead and remove all of those commits because they seem like garbage to git. The IonMonkey merge happened to have a number of changesets that were landed on top of such a head, and the commit corresponding to those changesets had been removed by git gc, but it still existed in my git-mapfile, which caused hg-git to assume that the corresponding git commit exists, leading to a broken chain of commits after converting the entire merge. He even had a script which goes through all of the entries in the git-mapfile and would remove them if the corresponding objects did not in fact exist in the git repository, so that hg-git could do the right thing and convert them over again the next time that it runs. I ran that script over night (it take s a long time on a large git-mapfile) and this morning reran the hg-git conversion, and a few hours later, the git repo was in a good shape again!
I have re-enabled the git mirror as of a few hours ago. In order to make sure that this problem never happens again, I set the gc.auto config to 0 to prevent git from running GC on it automatically again.