[Query]: P99 Latency for Query frontend vs Querier

[vibgy]

We observed that the P99 for Query frontend for when the stack was under load, is about 30s but at the same time, the P99 latency for Querier was over a minute.

It is not clear why query frontend shows only 30s while the querier is actually taking a lot longer.

To clarify - I used http_request_duration_seconds_bucket metric and histogram_quantile function to plot the above graphs.

Also checked the QPS on guery frontend and querier - they are both roughly the same QPS.