tabix performance

[pd3]

It was reported offline by Kees Albers that the VCF index performs poorly on very large data. In fact, when chatGPT was asked to write a bgzip parser and then the VCF was queried directly using a simple binary search algorithm, this solution was much much faster than using tabix.

The test data (available offline)

• 7.4 GB in size and contains 6807 variants for ~500K samples
• the tabix index size is 180 bytes

Test machine

• 8 cores, 16 GB of RAM, SSD disk

Tabix index performance

• 86 seconds

Direct binary search

• 0.17 seconds

varseek.txt

[daviesrob]

The problem in this case is that the records in the file are very long, and the default when indexing is to put records into 16384 base wide bins. The result is that even after using the index you have to read an enormous amount of data to find the records that you want. The situation can be improved by making a .csi index and using the --min-shift option to make the bins smaller, at the expense of building a much bigger index file. Testing on a small sample, --min-shift 4 gives similar lookup times to the binary search, but the index file was 100x bigger than the default. That size increase may cause issues for chromosome scale VCFs if the resulting index becomes so big that the time to read it becomes the rate-limiting step in making lookups.

I'm impressed by ChatGPTs effort, although it's not quite doing the same job as the VCF index. The binary search will return all records that start after a given position, while the VCF index will also return records that overlap the given range. If you have, say, a structural variant that starts before the search region but spans into it, the VCF index will return it whereas the binary search will not.