Duplicate % is way higher than what fastqc estimates and what I manually calculate

[sodiumnitrate]

I have a .fastq file with 225358 sequences. When I remove duplicates with a python script, I get 209438. This corresponds to a duplication rate of 7.06%, which is also what fastp gives me. However, fastqc results in ~78% duplicate reads. Does fastqc look at strict string equality or does it consider other things when determining duplication rate?

[s-andrews]

FastQC uses exact string equality from (by default) the first 50bp of the sequence. It also optimises by looking at only the first 100000 different sequences it sees in the file and then tracks their duplication and extrapolates to the rest of the data.

The extrapolation the program does assumes that the sequences arrive in random order, ie similar to what you'd see from an initial fastq file from a sequencing run. If you provide sorted sequences to the program then this will massively over-estimate the duplication level which is likely what you're seeing. If that's not the case then it suggests that you have very repetitive sequences, because 50bp is enough in most cases to provide a unique signature.

If anything, fastqc will normally tend to under-represent duplication since it would consider two sequences differing by a sequencing error to be non-duplicates, whereas programs which use genomic alignments can match more exactly. FastQC also can't use UMI information if you have it, which would be a more reliable way overall of estimating duplication levels (in combination with genomic alignment).