Compute hash for index incrementally

Right now we always compute the hash for the entire file. This is an optimisation only (and profiling suggests it won't make all that much of a difference). Note that we are now always downloading the `.tar.gz`, even when we are doing an incremental update, so instead of hashing a 300M file we are hashing a 30M file. 
