Downsampling support

[grendon]

Hi,

The size of the illumina short-reads files continue to grow as the sequencing costs continue to fall.
Running FastQC on the entire input file is compute-intensive.
Performing the downsampling step before running FastQC is also compute-intensive.
It would be great if FastQC could grab a subset of the input by downsampling randomly and then run the analyses on the smaller set rather than on the entire input file.

[s-andrews]

We already do this behind the scenes to some extent. Some of the more intensive modules (GC and Duplication) have optimisations in place to speed up processing. For most of the others the amount of processing for the actual module is trivial. The biggest hit to speed is just the decompression of the reads data. Because gzip isn't a format which allows for decompression starting at arbitrary locations the only way to sample the data is to do a serial decompression and then pick out reads at various points from the stream, but you'd still need to decompress everything.

A simpler, but possibly more biased, approach would be to analyse only the first N reads in a file, which is much quicker but won't tell you the total data size and may not give the correct impression of the file. You can do this at the moment with something like:

zcat myfile.fq.gz | head -4000000 | fastqc stdin:myfile.fq.gz

..which will just analyse the first million reads. This will do very strange things if your file is sorted, and may make the data look worse than it is from large runs where the first reads tend to come from the edges of the flowcell.

There are always ways to potentially improve this though so I'm certainly open to suggestions for ways to do this without biasing the results.