deduplication mechanism

[clalos2592]

Hello,

I would like to ask you the following question.

We are using Thanos to fetch time series metrics from multiple Prometheus instances (replicas). However each Prometheus replica store metrics with different timestamps. Unfortunately, this causing us issues with the deduplication mechanism. More specifically, we have the following test case:
series{foo1="bar1", foo2="bar2", replica="replica-0"}: {1663120560, 127}, {1663120599, 5664}, {1663120658, 5664}, {1663120659, 856}, {1663120718, 856}, {1663120719, 989}, {1663120778, 989}
series{foo1="bar1", foo2="bar2", replica="replica-1"}: {1663120577, 4}, {1663120636, 4}, {1663120637, 50263}, {1663120696, 50263}, {1663120697, 780}, {1663120756, 780}, {1663120757, 0}
series{foo1="bar1", foo2="bar2", replica="replica-2"}: {1663120568, 62}, {1663120627, 62}, {1663120628, 47980}, {1663120687, 47980}, {1663120688, 3033}, {1663120747, 3033}, {1663120749, 0}
series{foo1="bar1", foo2="bar2", replica="replica-3"}: {1663120573, 64}, {1663120632, 64}, {1663120633, 49118}, {1663120692, 49118}, {1663120693, 1385}, {1663120752, 1385}, {1663120753, 72}

and since there is a penalty of 5000ms in the deduplication mechanism, only data from first replica are selected i.e

series{foo1="bar1", foo2="bar2"}: {1663120560, 127}, {1663120599, 5664}, {1663120658, 5664}, {1663120659, 856}, {1663120718, 856}, {1663120719, 989}, {1663120778, 989}

However, this seriously affects the quality of our time series data.

Is there any possible way to tackle this issue ?

Thank you