Wednesday, June 24, 2015

Recovering from a corrupt Git repository

After all these days, Git finally showed me it's nasty side. Amidst the peak hub-bub of activity, a burst of cosmic rays left my central git repository corrupt!

Typical manifestation of a corrupt git repository is getting error messages of the following flavor while committing changes:

error: object file .git/objects/ef/5380fac97d4d077daa2e5b5d15fd1f6842cd98 is empty
Unfortunately not an insightful message. Also equally unfortunate is the fact that restoring the repository to its pre-corrupt state takes some invasive maneuvers.

Fortunately, the situation is salvageable. The following post really helped me get things in order.

http://stackoverflow.com/questions/11706215/how-to-fix-git-error-object-file-is-empty

While going through the experience of restoration, there were a few learnings which I stumbled upon:

1. Once you run git fsck --full, it will show you only the first of the error in the git object space. Don't get disillusioned. It's usually not just a single file which is corrupted.. there are many!

2. Do not go about deleting the empty files one by one.. use the find . -type f -empty -delete command inside the .git folder. It will save you quite a bit of time and frustration.

3. A time will come when running git fsck will throw another cryptic message error: HEAD: invalid sha1 pointer 5380fac97d4d077daa2e5b5d15fd1f6842cd98. This is a good sign. What is simply means is that the head pointer is pointing to an invalid commit reference.

4. Now, switch over to clone repository which is uncorrupted. Inside the .git folder cat the logs/refs/head/master file

This will show you something similar to this:

3862b7d4cd5244e6e3e0957c17619ddbb47732b7 a08b071ef5ec1cb1bf925c1a77d90345e3ebbefb Sandeep <sandeep@deb-op.(none)> 1434793482 +0530 pull: Fast-forward
a08b071ef5ec1cb1bf925c1a77d90345e3ebbefb 3b109e42b7509e750214439ea1a8f576782a61e9 Sandeep <sandeep@deb-op.(none)> 1434824573 +0530 pull: Fast-forward
3b109e42b7509e750214439ea1a8f576782a61e9 f7b7f485adebef47c0138cdca58e53ec185d89ac Sandeep <sandeep@deb-op.(none)> 1434867375 +0530 pull: Fast-forward
f7b7f485adebef47c0138cdca58e53ec185d89ac 883b41751be2224811ca3fc824a7cbefc5e2a10a Sandeep <sandeep@deb-op.(none)> 1434871859 +0530 pull: Fast-forward
883b41751be2224811ca3fc824a7cbefc5e2a10a 19db75e67ef214a91bb41d29f3dfc872aa3be1c3 Sandeep <sandeep@deb-op.(none)> 1434980707 +0530 pull: Fast-forward
19db75e67ef214a91bb41d29f3dfc872aa3be1c3 aa19df8b1da05620edbc817f32ccb25f148b8538 Sandeep <sandeep@deb-op.(none)> 1435028234 +0530 pull: Fast-forward
aa19df8b1da05620edbc817f32ccb25f148b8538 3421481ae75af94dc49bbc992921aaeda0d0d267 Sandeep <sandeep@deb-op.(none)> 1435064357 +0530 pull: Fast-forward
3421481ae75af94dc49bbc992921aaeda0d0d267 41d9b03a5f4aca531a515160d52168936e84f8fd Sandeep <sandeep@deb-op.(none)> 1435135328 +0530 pull: Fast-forward
41d9b03a5f4aca531a515160d52168936e84f8fd 1435a4c9c88c0270a128f3badebf474243bd1e92 Sandeep <sandeep@deb-op.(none)> 1435136176 +0530 pull: Fast-forward
1435a4c9c88c0270a128f3badebf474243bd1e92 12c85c45626a10112527acb11bec37367f88eca4 Sandeep <sandeep@deb-op.(none)> 1435154218 +0530 pull: Fast-forward

This is from one of my clones of the central repository. The important thing to note is the commit reference in the second column.

5. Now keep the above handy and go back to the central repository .git folder. Start issuing 

git show <commit hash> in reverse chronological order till you get a valid output. This is what we will set our head reference to. Let's assume that 19db75e67ef214a91bb41d29f3dfc872aa3be1c3 gives a valid output on git show

6. Issue the git update-ref HEAD 19db75e67ef214a91bb41d29f3dfc872aa3be1c3

If the above command goes through without a scream, you are all set. Run git fsck and see if all is good

7. Do not worry about dangling commits or blobs. They might look frightening but are quite harmless.

The above steps are not supposed to be instructive and they have to be read in conjunction with the stackoverflow URL I have pasted earlier.

Hope this helps someone save a few hours of exquisite frustration.

No comments: